Introduction
Wallaroo.AI, in a groundbreaking partnership with the United States Space Force (USSF) and Loft Orbital, has taken on a new challenge by deploying an AI-powered solution in space. The innovative approach leverages Edge AI in orbit to analyze satellite imagery in near real-time, identifying and tracking global naval ship movements. The Wallaroo solution integrates with the radiation-hardened hardware acceleration on lower-power ARM and NVIDIA Jetson GPUs ideal for the harsh environment of space, to capture and filter vast amounts of image data down to only the most critical insights. These targeted pieces of information can then be rapidly downlinked, empowering the USSF with timely and accurate decision-making capabilities, even in bandwidth-constrained environments.
“On-Orbit AI” Solution
Architecture Overview
The core of this solution lies in the Wallaroo AI Control Plane, which streamlines the process of packaging and publishing AI models to a container registry for deployment. These model container images are then uplinked to the orbiting satellite. The satellite, in turn, hosts a custom application that utilizes these deployed models at the edge. Data, including key insights, full-size images, and model diagnostics, is transmitted back to the ground for rapid analysis. This transmission is prioritized according to available bandwidth.
Imagine an RGB image of the Port of San Diego, pre-processed by Wallaroo’s algorithms.
An ensemble of two models is generated identifying ships within the image with yellow bounding boxes.
Key insights—such as the number of ships detected and small “image chips” of the detections—are rapidly downlinked. This on-edge processing eliminates the need to transmit large, high-resolution raw images before analysis, significantly reducing bandwidth requirements and accelerating insight extraction.
Examples of image chips returned via low-latency inter-satellite link for rapid insights, such as the count of ships and coordinates:
Implementation Approach: From Lab to Orbit
AI Model Development and On-The-Ground Testing
The journey to orbital deployment began with meticulous preparation. Wallaroo and Loft Orbital collaborated to establish a lab environment that precisely mirrored the in-space hardware and architectural configuration. This included using radiation-hardened NVIDIA Jetson hardware and on-orbit cameras, simulating limited connectivity and bandwidth to test the edge AI solution thoroughly. This rigorous pre-flight validation ensured that the entire process, from image capture to downlink insights, functioned flawlessly before deployment to the on orbit satellite.
The lab environment proved invaluable for fine-tuning the models and testing performance in a representative ground instance. A series of test inferences were conducted, ensuring the models performed as expected and identifying any potential issues early on. This focused approach guaranteed that the USSF team would receive timely insights from the latest captured images, enabling high confidence and timely actions.
AI Inference Microservice Development and Packaging
Through the extensive lab testing, an ensemble of two highly effective models, YOLO and SSDLite, was identified. These models, saved in ONNX format, were then uploaded to Wallaroo’s Auto-Packaging capability. Wallaroo’s Auto-Packaging dynamically converts models from any common or custom framework automatically bundling all necessary model artifacts into a distributable OCI-compliant package. This ensures consistency, portability, scalability, and reproducibility – all vital for optimal performance on resource-constrained edge hardware. This package is then ready for inference microservice deployment on the target infrastructure, regardless of its specific environment.
yolo_model = wl.upload_model('yolov', "./runs/detect/train49/weights/best_quantized.onnx",framework=Framework.ONNX)
ssd_model = wl.upload_model('ssdlite',"./SSDlitewithMobileNet/tacfi_ssd_mobilenet_v3.onnx", framework=Framework.ONNX)
Publishing of Models for Edge Deployment: The Power of Arm and Jetson
For deployment to the Loft satellite hardware at the edge, an important step was defining a deployment configuration specifically targeting ARM architecture and Jetson GPU acceleration. This directly addressed the hardware capabilities of the Jetson board on the Loft satellite. Wallaroo simplifies this by allowing users to easily specify ARM architecture and Jetson acceleration using just two variables within the deployment configuration. The pipeline is then published with this edge deployment configuration, packaging it as a standalone Docker container ready for edge deployment.
publish_config = DeploymentConfigBuilder() \
.replica_count(1) \
.cpus(2) \
.memory('1Gi') \
.arch(Architecture.ARM) \
.accel(Acceleration.Jetson) \
.build()
yolo_pipeline.publish(deployment_config=publish_config)
This innovative flexibility and portability means that regardless of whether the deployment is in the cloud, at the edge, or on-premise, Wallaroo provides the ability to deploy inference microservices specifically tailored to configurations and hardware resources such as memory and compute. This is a game-changer for Edge AI, particularly in environments with unique hardware constraints such as satellites.
Integrated Bench Testing
The application layer driving the end-to-end workflow on the satellite first utilizes Loft’s SDK to acquire images from the onboard imager. These hyperspectral images then undergo pre-processing, with Wallaroo’s pre-processing algorithm aligning three hyperspectral bands into a standard RGB image. Finally, inference is called on each deployed microservice (powered by a Wallaroo pipeline), and the outputs are ensembled for the final result. Model predictions and insights are then prepared for downlink.
# Process raw data into images
images = processing_loft.raw_to_img(band_list, input_path, img_type)
# Align the images
merged_image = processing_loft.align_images(images)
# Run inference on both models
predictions_yolo = inferring.infer_yolo(yolo_eng, merged_image, brightness_factor=2, conf_threshold=0.1)
logger.info(f"Inferred using YOLO.")
predictions_ssdlite = inferring.infer_ssd(mobilenet_eng, merged_image, brightness_factor=2, conf_threshold=0.19)
logger.info(f"Inferred using SSDLite.")
# Ensemble the predictions
filtered_boxes = ensembling.filter_overlapping(predictions_yolo, predictions_ssdlite, iou_threshold=0.05)
logger.info(f"Successfully ensembled models.")
# Export found ships
amount_found = ensembling.export_bboxes(merged_image, filtered_boxes, output_dir / base_name)
logger.info(f"{amount_found} ships found.")Inference requests are handled by the Wallaroo AI Inference Engine, a high-performance RUST-based engine optimized for speed and supporting concurrent and dynamic batching. Based on lab testing, a lighter-weight version of the Wallaroo Inference Engine was developed specifically for on-orbit inference. This optimized inference engine enables performant AI through efficient processing and hardware acceleration on lower-power Arm and NVIDIA Jetson GPUs, demonstrating Wallaroo’s commitment to supporting diverse edge hardware.
Uplink to Orbit: Deploying AI to Space
With thorough end-to-end validation on the ground complete, the next step uplinked the entire software system to orbit. This involved uploading and initializing the software directly onto the satellite while in orbit. Utilizing Loft’s extensive network of ground stations, all software elements were scheduled for uplink at the next available window, and synchronized with the satellite’s orbit. Once successfully uplinked and present on the satellite, Loft’s Cockpit platform allowed for the scheduling of specific image captures and the execution of the application code, enabling a final set of on-orbit validations. Upon completion of these final checks, the ML-based application and deployed models are officially in production on-orbit, ready to power the mission’s workflow.
Loft Orbital – Cockpit – Capture Constraints
Loft Orbital – Cockpit – Collection Scenarios
Loft Orbital – Cockpit – Request Complete
“Wallaroo’s edge AI expertise and their ability to extract critical insights from rich image data at the edge has the ability to transform how operators react and respond in dynamic situations. Their customer-first mindset and collaborative approach align perfectly with Loft’s goal of making space simple. We’re proud to support their work delivering timely, actionable intelligence, even in bandwidth-constrained environments, to enable faster, more informed decision-making.”
– Dr. Sarafina El-Badry Nance, Virtual Mission Manager, Loft Orbital
Results: Actionable Insights from Space
The inference logs generated throughout this process are downlinked for centralized management from the Wallaroo AI Control Plane. The Wallaroo UI provides a clear overview of the two pipelines comprising the workflow, including pipeline metadata such as version numbers and edge deployments. Pipeline inference logs offer a central view of inferences across all edge deployments. These logs can be individually examined for each edge device, providing granular insights into model performance.
Wallaroo’s automated observability capabilities enable easy setup of validation checks and real-time alerts using Wallaroo Assays. These assays leverage the inference logs to automatically monitor pipeline inputs and outputs for drift. For instance, assays can be configured to monitor average input pixel intensity and average output confidence. If on-orbit observations deviate from the training data, this drift automatically triggers an alert. As a result, Wallaroo facilitates continuous performance monitoring of edge-deployed models, enabling timely refinement and updates.
Upcoming Revolutions On Orbit
The upcoming steps involve rigorous on-orbit testing. Beyond this, the next exciting step is to extend this pattern to larger-scale workflows, deploying multiple models that work in tandem and efficiently replacing them using Wallaroo’s In-Line Model Updates capability. This will enable a full ML model lifecycle between a ground-based management platform and on-orbit edge deployment, automating model tuning and updates based on detected drift or anomalies.
“The mission of the United States Space Force (USSF) is to organize, train, and equip Guardians to conduct global space operations that enhance the way our joint and coalition forces fight, while also offering decision makers military options to achieve national objectives. To do this effectively, we must invest in Al and ML capabilities that can be deployed in the cloud and at the edge. Wallaroo has demonstrated their Al/ML Enterprise Platform, and I believe this platform – with its uniquely modern, interoperable, and integrated architecture – is positioned exceptionally well to deliver game-changing capabilities to the USSF.”
– Dr. Joel Mozer – Director Science, Technology & Research SPACEWERX
Conclusion
While ship detection serves as a compelling initial application, the underlying infrastructure and methodologies established through this project have a far broader impact for Edge AI in space. This foundational work paves the way for a diverse range of Edge AI-powered applications to transition rapidly and efficiently from development to tangible, real-world execution in orbit, significantly advancing our ability to deploy and leverage AI in space operations.
The Wallaroo AI Universal Inference Platform demonstrated exceptional capabilities in overcoming the unique challenges of deploying Edge AI on orbit by auto-packaging models for resource-constrained hardware and using a lightweight inference engine for high performance despite limited connectivity. Seamless support for Arm and NVIDIA Jetson hardware, automated observability, and staged data downlinking ensured resilience, enabled rapid insights and centralized model management, advancing future Edge AI projects in space.
Additional Resources
