Introduction
Artificial intelligence use cases are exploding, with the potential to reshape entire industries – including how we live, work, and play. Ultimately, the AI models that power these use cases need computational power to run – be it in the cloud, on-premise, or on-location where the data itself is (“edge”). In the popular media, this computational power is often referred to as “AI chips” and strongly associated with NVIDIA GPUs.
Yet, as businesses increasingly deploy AI solutions at scale, the reality is that compute environments are heterogeneous. In particular Arm-based chips are becoming increasingly important to AI, as perfectly captured by the CEO of Arm in their recent earnings call: “AI is changing everything, and you can’t run AI without Arm.”
Like almost every other form of modern technology, AI chips need software for the hardware to be usable. When the AI infrastructure is combined with specialized inference software it enables robust and efficient access to computational resources that can be utilized by AI developers, by hyperscalers and AI cloud providers, and by enterprises and ISVs, i.e. it facilitates a software-enabled ecosystem.
The ability to seamlessly transition AI models from development to real-world application, manage their lifecycle, and scale their impact hinges on establishing a strong foundation of software tools. Without this crucial layer, the transformative potential of AI risks being constrained, highlighting the urgent need to prioritize and invest in the infrastructure and software that will power the next generation of AI innovation. Nvidia has recognized this from the beginning, investing heavily in software starting with low-level CUDA libraries, its Triton inference server, and more recently filling out its offerings with acquisitions such as Run:ai.
In this blog we will discuss how AI infrastructure powered by Arm, combined with Wallaroo’s Software Platform, delivers a high-performance, comprehensive production AI inference solution in all market verticals to easily deploy, monitor and manage efficient and scalable AI workloads – whether in the cloud, on-premises, or at the edge. This solution provides a compelling alternative to a fully Nvidia-based solution, and will help drive success for AI application developers as well as the consumption of Arm-based silicon.
We will cover the Wallaroo AI Inference Engine and how it drives high efficiency AI inference microservices on Arm, coupled with real-time insights and AI feedback loops for lower cost of ownership and faster ROI.
We will also address how the Wallaroo AI Control Plane, helps enterprise AI teams collaborate effectively, and efficiently with seamless integration into existing tools and ecosystems for deploying AI models faster to production, scaling deployment, automating model lifecycle management with built-in observability to deliver effective AI applications in any environment across Cloud, Edge, and On-Prem, leading to tangible AI outcomes.
The Wallaroo AI Inference Engine Optimized on Arm
The power unit of the Wallaroo platform is the Wallaroo AI Inference Engine. Built for turnkey optimized inference that is agnostic to model types and hardware, it can quickly support newer models and hardware acceleration updates rapidly and efficiently.
The Wallaroo AI Inference Engine delivers high-performance for both batch and real-time inference. Its architecture features an optimized, multi-threaded network server and orchestration layer that manages incoming requests, prepares data, and executes models. Wallaroo offers flexible deployment, easily scaling and transitioning from x86, and GPU architectures to Arm through simple configuration.
Additionally, the Wallaroo AI inference engine comes with built-in Autoscaling capabilities that adjust resources utilization using pre-configured triggers based on real-time demand, ensuring optimal inference latency and throughput.
The Wallaroo AI Inference Engine features optimizations both for data-processing and inference serving aspects. Thus enabling optimal AI inference latency and throughput. Written in Rust Language, a modern, high-performance language that enables detailed control over aspects such as memory management to reduce data copies and provides strong concurrency support to take advantage of modern, multi-threaded processors.
Wallaroo offers superior inference performance for critical AI use cases leveraging Computer Vision and Language Models on Arm-based hardware, both in the cloud and at the edge. Compared to bespoke or non-specialized AI inference solution implementations, Wallaroo significantly reduces unnecessary AI inference overhead as seen in the benchmarks below.
The Wallaroo AI Operations Center (Control Plane)
The Wallaroo AI Operations Center (Control Plane), helps AI teams collaborate effectively, and efficiently with seamless integration into their existing AI tools and ecosystems for launching their models faster to production, scaling deployments, model management, and observability to deliver effective AI leading to tangible ROI.
Taking a closer look we will see how the Wallaroo AI Operations Center is designed to simplify and automate the entire AI production lifecycle. Streamlining everything from automated model packaging and scaling, while enabling continuous model delivery for reducing manual intervention. With centralized model management through secure workspaces and robust automated observability, ensuring AI deployments are healthy, perform optimally, and meet defined throughput and latency requirements, empowering data scientists and ML engineers.
Automating Model Packaging & Deployment
The Wallaroo self-service toolkit simplifies and automates the process of deploying models to production, from packaging and containerization to scaling and serving.
AI Model deployment is simplified through Auto-packaging by bundling all necessary components into a single distributable package. Models are converted from any common or custom framework into an open format that Wallaroo uses to run the models. This can be done via the UI, SDK, and API depending on where the model artifacts are being managed after training. Auto-packaging helps ensure consistency, portability, scalability, and reproducibility across platforms and environments.
Continuous Model Delivery
Through centralized AI Workflow Automation, teams can ensure that their AI models are always up-to-date and performing optimally. The Wallaroo Workload Orchestration capability reduces the risk of errors and delays associated with manual deployment processes, and frees up valuable time and resources for other prioritized tasks.
The workload orchestration capability empowers teams to effortlessly construct automated analytics, MLOps/LLMops, agentic AI and model retraining/fine-tuning workflows that continuously operate their inference pipelines. The scale and flexibility that this capability delivers means that workflows can be scheduled, or run as a continuous service until terminated. The benefit is that AI teams can rapidly prototype agentic workflows and then scale both the workload and pipeline resources to manage demanding production tasks.
Unified Model Management
Centralized model management in Wallaroo is anchored in the concept of workspaces that helps Data Scientists and ML engineers break down operating silos and collaborate securely and effectively on AI deployments. Wallaroo Workspaces are a supercharged version of the typical model registry offered by other deployment solutions helping MLOps/LLMOps teams as they scale their deployments by adding more context and security around their model rollouts.
The workspaces concept is used by enterprises to organize their rollouts by market, region, location, or even by team depending on the use case. As part of managing their models in production at scale, Data Scientists can try different rollout strategies such as A/B testing, Canary deploys, Shadow deploys, and Blue/Green strategies to identify the best performing models to run. Once that decision happens, replacing an existing model with a better performing model can take place in seconds through in-line updates and without any inference interruption.
Automated Observability
Model observability is critical to successful AI production lifecycle and through this Wallaroo enables model monitoring Data Scientists ensure that deployment pipelines are healthy and performing as planned with the appropriate throughput and latency requirements that have been originally defined. Having centralized access to these metrics provides insights into any potential model or infrastructure actions to take.
At the same time, Data Scientists can easily and quickly configure a set of validation checks and automate alerts using the Wallaroo Assays capability, enabling real-time model monitoring and data drift detection. Additionally, data scientists can generate local and global explainability reports against deployed models to get a comprehensive understanding of attributes that contributed to a certain prediction or set of predictions that inform the AI team how the models are performing, helping to ensure that they are effective in production and generating value.
“The beauty of leveraging Wallaroo for the production of data science models is the ease of transitioning a local prototype model/code base into its production environment. I’ve worked at companies where the production framework involves months of refactoring your code into abstract classes.
With Wallaroo, you have two classes – one of which is simply your code base. This greatly reduces the time to production and empowers data scientists to transition their code into production frameworks earlier in the development process so they can leverage all of the monitoring and additional capabilities that Wallaroo offers.
As a result, your model is better tested and understood well before it officially enters production for your company.”
Lead Data Scientist – RealPage
Conclusion
The Wallaroo Universal AI Inference Platform, powered by an ultra fast high-performance inference engine and optimized for Arm infrastructure, provides an end-to-end solution for organizations to grasp the nettle of AI in the cloud, on-prem and at the edge.
The Wallaroo Al Universal Inference Platform combined with Arm enables full end to end automation of the AI lifecycle equipping enterprise Al teams with the means to quickly and efficiently operationalize Al production by the Wallaroo AI Control Plane scaling model deployment, governance and observability, leveraging real-time data insights for faster decision making driving unparalleled scale, agility and efficiency for enterprise strategic AI initiatives while realizing optimal performance.
By industrializing AI at scale, the Wallaroo Al Universal Inference Platform allows Al teams to focus on high-value tasks with maximum business impact and ROI.
Next steps
Request a demo today and discover how Wallaroo can transform your AI DevOps processes, accelerate your AI initiatives, and drive business success. Visit our website to learn more about Wallaroo and the benefits of our AI DevOps platform. Don’t miss out on this opportunity to revolutionize your AI capabilities and unlock the full potential of your AI investments.
Additional Resources
