Introduction
There are thousands of satellites in space collecting data, but the challenge is, as with any data, how do you process it and harness it to make it actionable? In space, low bandwidth and limited windows of downlink connectivity make it impossible to transmit every byte of data back to Earth, which means that many data-driven decisions are being made with partial, out-of-date, or inaccurate information, leading to critical gaps in understanding for decision makers.
Wallaroo.AI recently partnered with the United States Space Force (USSF) and Loft Orbital to deploy an AI-powered solution in orbit to help tackle this problem. The mission: analyze satellite imagery to identify and track global naval ship movements using image data in near real-time.
The goal of this joint project with Loft Orbital is to harness rich image data and run Edge AI on orbit to filter the collected data down into only the most important key insights. These targeted pieces of information can then be downlinked to arm the USSF with key insights as quickly as possible, allowing for fast and accurate decision-making without the luxury of full bandwidth or persistent connectivity.
“AI on-orbit” Solution Architecture
The solution architecture begins with the Wallaroo Operation Center, which automates model packaging and publication to a container registry for deployment. Model container images are uplinked to the orbiting satellite, which hosts a custom application that utilizes the deployed models at the edge. Key insights, full-size images, and model diagnostics are transmitted to the ground based on available bandwidth for quick analysis.
An RGB image of the Port of San Diego illustrates the pre-processed images expected by the models. RGB is created by Wallaroo’s pre-processing algorithm.
An ensemble of two models identifies ships within the image in yellow boxes. Key insights, such as ship count and detection image chips, are rapidly downlinked. This facilitates on-edge processing of raw images for rapid insight extraction, eliminating the need to downlink large, high-resolution images before analysis.
Implementation Approach
AI Model Development and On-The-Ground Testing
The first stage of the approach was to work with Loft Orbital and USSF to set up and configure a lab environment that reflected the exact hardware and architecture configuration that would be experienced in space.
Wallaroo and Loft Orbital simulated orbit conditions in a lab to test their edge AI solution on radiation-hardened Nvidia Jetson hardware with limited connectivity and bandwidth, using on-orbit cameras. This ensured the entire process, from image capture to downlink insights, functioned correctly before satellite deployment.
The lab environment aided in fine-tuning the model and deploying it for testing in the on-the-ground instance. This allowed for running a series of test inferences in the lab environment, ensuring the model performed as expected, identifying any potential issues before broader deployment to orbit. This helped focus on the goal of getting insights back to the ground in a timely fashion so that the USSF team could take immediate action based on the insights from the latest images captured.
AI Inference Microservice Development and Packaging
Through the lab testing, an ensemble of two models, YOLO and SSDLite, was identified to produce the best results. These models are both saved in ONNX format and are uploaded for Wallaroo’s Auto-Packaging capability, which ensures all model artifacts are bundled into a distributable, OCI-compliant package for inference microservice deployment on the target infrastructure and environment. The Auto-Packaging capability within Wallaroo dynamically converts models from any common or custom model framework, ensuring consistency, portability, scalability, and reproducibility, which is crucial for optimal performance with finite target edge hardware resources.
yolo_model = wl.upload_model('yolov', "./runs/detect/train49/weights/best_quantized.onnx", framework=Framework.ONNX)
ssd_model = wl.upload_model('ssdlite', "./SSDlite with MobileNet/tacfi_ssd_mobilenet_v3.onnx", framework=Framework.ONNX)
To ensure flexible and efficient deployments, a deployment configuration specifying target resource allocation and hardware profiles is defined as part of the inference microservices deployments with the two CV models that have been identified. This initially serves as a test deployment on a representative hardware profile before publishing the inference microservice to run at the edge.
Publishing of Models for Edge Deployment
Finally, for deployment to the Loft satellite hardware at the edge, a deployment configuration is defined to specifically target ARM architecture and Jetson GPU acceleration to target the Jetson board on the Loft satellite. ARM architecture and Jetson acceleration are easily specified using two variables in the deployment configuration. The pipeline is then published with the edge deployment configuration to package it as a standalone Docker container for edge deployment.
The output includes container metadata and a Docker run command for execution on the edge device. Regardless of whether deployment is in the cloud, at the edge, or on premise, there is a great deal of flexibility and portability through providing the ability to deploy inference microservices targeted to specific configurations and hardware resources such as memory, CPUs/GPUs/TPUs.
publish_config = DeploymentConfigBuilder() \
.replica_count(1) \
.cpus(2) \
.memory('1Gi') \
.arch(Architecture.ARM) \
.accel(Acceleration.Jetson) \
.build()
yolo_pipeline.publish(deployment_config=publish_config)
Integrated Bench Testing
The application layer driving the end-to-end workflow first leverages Loft’s SDK to acquire images from the onboard imager. The resulting hyperspectral images undergo pre-processing for alignment of three hyperspectral bands into an RGB image through Wallaroo’s pre-processing algorithm. Finally, inference is called on each deployed inference microservice (powered by a Wallaroo pipeline), and the outputs are ensembled for the final result. Model predictions and insights are then prepared for downlink.
# Process raw data into images
images = processing_loft.raw_to_img(band_list, input_path, img_type)
# Align the images
merged_image = processing_loft.align_images(images)
# Run inference on both models
predictions_yolo = inferring.infer_yolo(yolo_eng, merged_image, brightness_factor=2, conf_threshold=0.1)
logger.info(f"Inferred using YOLO.")
predictions_ssdlite = inferring.infer_ssd(mobilenet_eng, merged_image, brightness_factor=2, conf_threshold=0.19)
logger.info(f"Inferred using SSDLite.")
# Ensemble the predictions
filtered_boxes = ensembling.filter_overlapping(predictions_yolo, predictions_ssdlite, iou_threshold=0.05)
logger.info(f"Successfully ensembled models.")
# Export found ships
amount_found = ensembling.export_bboxes(merged_image, filtered_boxes, output_dir / base_name)
logger.info(f"{amount_found} ships found.")
Inference requests are handled by the Wallaroo AI Inference Engine, a high-performance RUST-based engine optimized for speed and supporting concurrent and dynamic batching. Based on lab testing, a lighter-weight version of the Wallaroo Inference Engine was developed for on-orbit inference, enabling performant AI through efficient processing and hardware acceleration on lower-power ARM and NVIDIA Jetson GPUs.
The bench testing process was then performed using Loft’s Ultimate Edge platform to validate that the application layer and deployed models work well on replicated satellite hardware and the full software stack present on orbit.
Examples of image chips returned via low-latency inter-satellite link for rapid insights, such as the count of ships and coordinates:
Uplink to Orbit
Now that this has been validated on the ground, the next step is to uplink the full software system to orbit – effectively uploading it and initializing it on the satellite itself. Using Loft’s network of ground stations, all software elements are scheduled for uplink at the next window of availability based on the satellite’s orbit. Once these materials are successfully uplinked and present on the satellite itself, Loft’s Cockpit platform allows for the scheduling of specific image captures and the execution of the application code as a final set of on-orbit validations. Once these final checks are complete, our ML-based application and deployed models are officially in production on-orbit and ready to power our workflow.
Loft Orbital – Cockpit – Capture Constraints
Loft Orbital – Cockpit – Collection Scenarios
Loft Orbital – Cockpit – Point Request
Loft Orbital – Cockpit – Request Complete
“Wallaroo’s edge AI expertise and their ability to extract critical insights from rich image data at the edge has the ability to transform how operators react and respond in dynamic situations. Their customer-first mindset and collaborative approach align perfectly with Loft’s goal of making space simple. We’re proud to support their work delivering timely, actionable intelligence, even in bandwidth-constrained environments, to enable faster, more informed decision-making.”
– Dr. Sarafina El-Badry Nance, Virtual Mission Manager, Loft Orbital
Results
Insights
The inference logs created throughout this are downlinked for centralized management from the Wallaroo Operation Center where the UI displays the two pipelines comprising the workflow. Digging into this reveals pipeline metadata, including version numbers and edge deployments. Pipeline inference logs are also accessible, as seen below, which provide a central view of inferences across the edge deployments. These logs can be examined individually for each edge device.
Wallaroo’s automated observability enables easy setup of validation checks and real-time alerts using Wallaroo Assays, which use these logs to automatically monitor pipeline inputs and outputs for drift. In the examples below, assays are set up for average input pixel intensity and average output confidence, creating a feedback loop to detect if on-orbit observations deviate from the training data. This drift triggers an alert. Using these logs and assays, Wallaroo enables performance monitoring of edge-deployed models, facilitating refinement and updates.
Upcoming Revolutions On Orbit
Over the next few weeks, rigorous on-orbit testing will be conducted. Beyond this, the next step is further extending this pattern to larger-scale workflows; this includes deploying multiple models that work in tandem and replacing them efficiently using the Wallaroo In-Line Model Updates capability. With this, we are able to implement a full ML model lifecycle between a ground-based management platform and on-orbit edge deployment by automating model tuning and updates resulting from any detected drift or anomalies.
“The mission of the United States Space Force (USSF) is to organize, train, and equip Guardians to conduct global space operations that enhance the way our joint and coalition forces fight, while also offering decision makers military options to achieve national objectives. To do this effectively, we must invest in Al and ML capabilities that can be deployed in the cloud and at the edge. Wallaroo has demonstrated their Al/ML Enterprise Platform, and I believe this platform – with its uniquely modern, interoperable, and integrated architecture – is positioned exceptionally well to deliver game-changing capabilities to the USSF.”
– Dr. Joel Mozer – Director Science, Technology & Research SPACEWERX
Conclusion
While ship detection presents a compelling initial application, the underlying infrastructure and methodologies established through the project extend far beyond a single use case. The foundational work on this project paves the way for a diverse range of Edge AI-powered applications to transition rapidly and efficiently from lab environments to tangible, real-world execution in orbit. This acceleration from lab experimentation to on-orbit operations substantially advances our ability to deploy and leverage AI in space, opening up possibilities to, paraphrasing Captain James T. Kirk, boldly go where no Edge AI has gone before.
Throughout the project, the Wallaroo Al Universal Inference Platform delivered exceptional capabilities in overcoming the unique challenges of deploying EdgeAI on orbit. The Wallaroo AI Control Plane’s auto-packaging capability enabled easy and quick model deployment to resource-constrained hardware, while the optimized lightweight inference engine ensured high-performance inference and rapid insights despite the connectivity limitations.
Advanced, automated observability combined with a staged approach to data downlinking provided resilience against intermittent connectivity and low bandwidth conditions. These strengths delivered rapid insights and centralized lifecycle model management in the demanding environment of space, marking a significant leap for Edge AI capabilities.
Additional Resources
