How Wallaroo Solves for Edge Machine Learning

April 14, 2023

Wallaroo’s highly efficient inference server makes it possible to run complex ML models in constrained environments, while our Model Operations Center provides a centralized hub for managing models to dozens, hundreds, or even thousands of localized (e.g. factory floor) or decentralized (e.g. 5G network) endpoints.

How it works: Full production functionality to deploy and manage edge ML

Edge ML is more than just sticking some code in a device. MLOps teams need to consider how four different environments interact with each other:

This is an example of the optimal edge deployment architecture that can be adopted fully or partially depending on the ecosystem.

1. Model training/retraining:

Wallaroo is agnostic to where or what you use to develop your models. Wallaroo takes nearly any model artifact resulting from model training and converts it to an optimized format to run as efficiently as possible inside our lightweight server built in Rust.

2. Model registry & operations:

The Wallaroo Model Operations Center keeps track of every model, every version, and manages pushing out the latest ML model and pipeline to each endpoint. Developers can choose to use either the Wallaroo UI as the central hub and dashboard for automation and managing the models, pipelines, and logs, or they can use Wallaroo’s SDK to manage through their portal of choice. What’s more Wallaroo can integrate with ML registries and tooling that teams may use for model training and experimentation, such as MLFlow.

3. Wallaroo Edge:

You can deploy our edge ML solution via connection or air gapped. Simply load software on a USB stick or other separate drive and send a technician to set up the Wallaroo Edge node on-site. From there the edge device takes in sensor data from IoT device, runs ML inference efficiently, and then feeds results to the application (e.g., results of computer vision to the robot arm on the assembly line to let it know something coming off the line does not meet QA).

4. Aggregated Analytics and Observability

The Wallaroo observability framework offers the ability to monitor and observe model performance globally in the Wallaroo model operations center, or locally for a specific edge cluster. Key features include:

Latency and throughput monitoring reports across all pipelines and edge deployments
Analytics for data and model anomalies
Data and concept drift detection reports across all pipelines and edge deployments

Ready to run your models in any environment?

Check out this webinar featuring Wallaroo.AI’s Director of Architecture as he provides an overview of the unique challenges to managing Edge ML and discusses how Wallaroo’s unique platform has helped customers in government, telecommunications, manufacturing, and more to run scaleable AI programs at the edge.

You can also drop us a line so we can demonstrate to you how much more efficiently and effectively we can run you models in any edge environment.