VFFE — Vision Front-End for Flight Computer (Annita-1)

Every pixel a satellite sends to ground costs power, bandwidth, and ground-station time — resources that are always scarce and often the actual bottleneck of a mission, not the camera itself. Most small satellites still capture first and decide later, on the ground, days after the opportunity to act on the data has passed.

VFFE flips that order. It's an onboard vision front-end that looks at every frame before it's transmitted, scores how valuable it actually is, and decides right there — discard it, compress it, or send it in full — using a Science Value Score (SVS) computed from image quality, a lightweight neural classifier, and transmission cost.

Why phyBOARD-RT1170 (and why not rad-hard)

Rad-hardened flight computers exist for good reason, but they're expensive, slow to iterate on, and usually years behind commercial silicon in raw compute. For CubeSats and many small-sat missions flying in relatively benign orbits with adequate shielding and error-correcting memory, that trade-off is increasingly being questioned — commercial off-the-shelf (COTS) processors are already flying, and the compute they bring is hard to ignore when the mission is compute-bound rather than radiation-bound.

The i.MX RT1176 on the phyBOARD-RT1170 is a strong fit for that argument: a real dual-core crossover MCU (Cortex-M7 + Cortex-M4) with a dedicated hardware image co-processor (PXP), enough headroom to run a quantized neural network on-device, and it's available today, off the shelf, at a fraction of the cost and lead time of a space-qualified part. VFFE is our attempt to show that a serious onboard decision pipeline — not a toy demo — is realistic on hardware like this.

How it works

An important note on the camera: the ESP32-S3 + OV5640 module is a stand-in sensor, not the point of the project. VFFE's actual contribution — the decision pipeline — is sensor-agnostic. That's also why the onboard classifier is trained on real Sentinel-2 satellite imagery (the EuroSAT dataset), not on photos taken by the camera itself. The camera exists to prove the pipeline works against a live, asynchronous, imperfect data source, the way a real payload would.

On the RT1170 itself:

• M7 core runs the full pipeline: TJpgDec JPEG decode, PXP-accelerated rotation, an image quality score (sharpness via Laplacian variance + contrast), classical features (RGB565 histograms, Sobel edge density, ROI blue-dominance), a quantized TinyML CNN (98.44% int8 test accuracy on EuroSAT water-vs-land, TFLite Micro + CMSIS-NN), and the final SVS decision engine.
• M4 core owns UART acquisition from the camera — receiving and validating the framed JPEG protocol (0xAA 0x55 [len] [payload] [checksum]) independently of the M7's compute load, and handing off complete, checksum-verified frames through shared OCRAM.

Pipeline

1. JPEG decode (TJpgDec) and 180 degree rotation on the PXP
2. Image quality score: Laplacian-variance sharpness and contrast
3. Classical features: RGB565 histograms, Sobel edge density, blue dominance in a region of interest
4. Water/land classifier: int8 CNN trained on EuroSAT (Sentinel-2), TFLite Micro with CMSIS-NN
5. Science Value Score (SVS): quality, event likelihood and transmission cost combined into a single decision

The camera is an ESP32-S3 with an OV5640. It is a stand-in sensor; nothing in the pipeline depends on it.

Camera link

The ESP32 sends a JPEG roughly every 2 s over UART, 115200 8N1:

0xAA 0x55 [len_lo] [len_hi] [JPEG bytes] [XOR of the JPEG bytes]

The M4 parses the stream with a small state machine and only publishes a frame to shared memory once the checksum matches:

See Code #1

Starting the M4 from the M7

The M4 image is built separately, embedded in the M7 app as a C array, copied to RAM at boot and released from reset:

See Code #2



Things that cost time and are worth knowing if you do the same on this board:

• The M4 .bin is built with CONFIG_ROM_START_OFFSET=0x2000, so the vector table sits at the load address plus 0x2000, not at the start of the blob. Pointing VTOR at the base gives a core that never runs.
• Flush the M7 data cache after the copy. The M4 reads RAM directly and can otherwise see stale bytes.
• The reset write to SRC->CTRL_M4CORE before BT_RELEASE_M4 mirrors what Zephyr's own second_core_boot() does. Without it the M4 booted unreliably.
• A byte array holding the TFLite model needs alignas(16). Unaligned, AllocateTensors() hangs with an unaligned-access fault.
• CONFIG_HEAP_MEM_POOL_SIZE defaults to 0 here, which makes ipc_service_open_instance() fail with -ENOMEM.
• With the shared-memory hand-off, invalidate the cache on the M7 side before reading what the M4 wrote.

Build

The apps were built inside a PHYTEC zephyr-phytec-application west workspace, each one under my_apps/. Board targets are phyboard_atlas/mimxrt1176/cm4 and phyboard_atlas/mimxrt1176/cm7.

See Code#3

You can see a real, second-by-second replay of the 415-frame stability run — including live quality scores, classifier confidence, and the SVS decision — here: https://alejandroxconti.github.io/annita1/console/ Every number in it is pulled from the actual captured log, not staged.

Engineering notes

Two things are worth stating plainly rather than glossing over:

• The M4 sometimes loses ~20–25% of frames under sustained load. Root cause: interrupt-driven UART reception caused the M4 to lock up entirely on its first kernel sleep call (a real, reproducible ARM Cortex-M scheduler/PendSV interaction we traced with GDB down to the exact instruction). We shipped a stable pure-polling receiver instead, which trades a modest frame-loss rate for zero lockups — a deliberate reliability-over-completeness call under contest time pressure.
• Full end-to-end integration (M4 acquisition feeding the M7's decision pipeline in one continuous run) is implemented but not yet stable as of this submission — the M4 runs, the M7 runs, but wiring them into a single live pipeline surfaced a UART reception issue we could not root-cause in the time available (ruled out: wiring, a real pin-mux conflict we did find and fix, boot timing, and Kconfig differences). It's documented as a known open item rather than hidden.

COTS-based onboard AI: pros and cons

Pros

• Order-of-magnitude more compute and memory than typical rad-hard flight processors, at a fraction of the cost and lead time.
• A real hardware image co-processor (PXP) and enough headroom for on-device ML inference — both out of reach on many legacy space-grade MCUs.
• Fast iteration: the entire pipeline above was built, measured, and revised on a single dev kit in real dev cycles, not multi-month radiation-qualification cycles.

Cons

• No inherent radiation tolerance — viable mainly for shorter missions, benign orbits, or designs that accept redundancy/voting at the system level instead of at the silicon level.
• Long-term flight heritage and qualification data are far thinner than for rad-hard parts, which matters for mission insurance and review boards.
• Dual-core coordination (as this project found firsthand) adds real firmware complexity that a single-core rad-hard design doesn't have to deal with.

What's next

• Finish the M4↔M7 full-pipeline integration and close the remaining frame-loss gap on the M4's UART receiver.
• Add a 4th SVS factor using mission-state variables (battery margin, remaining ground-contact window) — a natural extension once the M4 is fully free to own real-time telemetry.
• Extend the TinyML target beyond water/land (cloud detection has real precedent in CubeSat literature and was deprioritized only for time, not feasibility).

REPO :

https://github.com/AlejandroxConti/annita1.git

Code #1

Code #2

Code #3