Introduction
As a new generation of AI chips from companies like AMD, ARM/Ampere, Qualcomm and various startups emerges, the AI landscape is diversifying beyond Nvidia and large cloud providers.
This market presents a significant opportunity, especially in AI inference. In 2024, data center sales for inference hardware topped $60 billion, with an expected fourfold increase to over $240 billion by 2027. Even if Nvidia’s market share, which stood over 70% last year, only drops to 50% by 2027, there is still a substantial $120 billion AI inference market for other hardware manufacturers within two to three years.
A major hurdle for developers and enterprises in leveraging new chips is the inflexibility of current AI deployment tools which are often bound to specific hardware or cloud environments, such as Nvidia’s software for its GPUs. Nvidia aims to sustain its market share through a robust combination of hardware and software, including its AI libraries and its inference serving and orchestration software – much of which has been enhanced through recent acquisitions.
The resulting lock-in greatly complicates the migration of AI workloads between different hardware types, for example from Nvidia to AMD GPUs. On top of that, the need to shift workloads between cloud providers (e.g., from a hyperscaler to a neocloud) to leverage available infrastructure and its benefits presents another challenge. Both factors severely hinder the adoption of new infrastructure, and put the market opportunity at risk for the chip makers aside from Nvidia.
Properly addressing these challenges also requires considering the details of inference workloads. Specifically, inference microservices increasingly involve multiple steps or components. The way these steps are deployed and orchestrated onto the hardware influences both performance and cost. Examples of multi-step inference services are:
- Pre-processing, vision model, and post-processing
- RAG LLM
- Multi-model chain, e.g. speech-to-text, followed by text summarization
- Agentic AI workflow with a combination of models, many with 10B or fewer parameters
Finally, for LLMs, splitting steps like prefill and decode across multiple chips can improve latency and/or throughput. While Nvidia provides a solution for its own GPUs, the evolving chip landscape presents opportunities to further enhance inference throughput, latency, and cost by leveraging the most suitable hardware for each step in an inference service. However, manually implementing this process is complex and challenging for most developers.
For these reasons, for 2025, Nvidia’s inference market share percentage will stay steady even as the market itself grows quickly. However, if other chip makers can put a solid hardware and software foundation in place in 2025, they are poised to reap the benefits in 2026 and beyond – and this is exactly where Wallaroo’s Universal AI Platform can help break apart infrastructure lock-in and drive universal, accelerated AI inference.
Wallaroo Software for Universal, Accelerated AI
Wallaroo solves the migration and orchestration challenges with a single software platform that makes the new generation of silicon, including AMD’s chips, much more accessible and appealing to AI developers for running inference workloads. Wallaroo enables:
- Easy deployment and migration of AI inference across various hardware, including AMD GPUs, and from hyperscalers to neoclouds, on-prem, and edge environments.
- Automated packaging and orchestration of AI deployment & inference tasks to accelerate time to production, and maximize performance for AI applications & agents
- Optimal infrastructure utilization – through a combination of smart orchestration utilizing the right hardware for the different tasks on an inference pipeline, a highly efficient data plane, continuous batching, and a purpose-built Rust-based inference engine
- Centralized and unified control of deployed inference services, from cloud to edge
This helps AI developers to easily make the best use of available infrastructure, but also benefits silicon providers such as AMD by making it easier and faster for their hardware to be integrated into AI workflows, without requiring developers to learn new tools or manually implement complex infrastructure orchestrations or migrations.
Wallaroo effectively eliminates infrastructure silos, accelerating chip makers’ go-to-market for new silicon by providing a new level of automation, orchestration, and control for AI inference. This bridges the gap between AI developers and diverse infrastructure, optimally leveraging and supporting current technologies.
So how does Wallaroo do this?
First, Wallaroo provides software (“inference engine”) to enable optimized, scalable and industrial-grade inference microservices that complements and integrates with software from chip companies (e.g., AMD ROCm) and available AI runtimes (vLLM, ONNX, PyTorch…).
Second, Wallaroo provides a unified control pane (“control center”) that allows for the automated and fast packaging of models and inference steps into an inference microservice, migration of workloads, and the ability to update, scale, monitor and manage the deployed services.
Going one level deeper, we can lay out the key components that Wallaroo provides across hardware and deployment environments.
Our blog, Orchestrating Hybrid Agentic AI with Wallaroo is a great read to delve into more technical details. Here, we want to highlight some key elements to Wallaroo’s automation and orchestration.
Auto-packaging significantly streamlines AI inference deployment by consolidating all essential elements—code, models, and libraries—into a single distributable package. Wallaroo facilitates the conversion of models from various common or custom frameworks into an open format, enabling their portable execution. This process can be managed through Wallaroo’s UI, SDK, or API, depending on where model artifacts are maintained post-training. By ensuring consistency, portability, scalability, and reproducibility across diverse platforms and environments, auto-packaging eliminates substantial manual effort and troubleshooting, saving hours or even days.
Wallaroo’s orchestration works at various levels:
- Hardware orchestration – working with underlying Kubernetes in diverse environments to deploy and scale compute resources for the inference engine.
- Inference pipeline orchestration – orchestrating and coordinating between different steps in an inference service, such as pre- and post-processing, or multiple models working together.
- Task orchestration at the hardware level is achieved by offering sophisticated granular and dynamic allocation of compute resources (both CPU and GPU) based on tasks within an inference pipeline.
Putting the automation, orchestration and the unified control all together enables Wallaroo’s magic. For instance, an AI developer could use Wallaroo’s developer toolkit to move an agentic AI workload from AWS Sagemaker on NVIDIA to AMD hardware in a different cloud (e.g. Digital Ocean), all within minutes.
Conclusion
In the fast evolving landscape of AI, true value isn’t found in the models or hardware, but in the ability to deploy and manage them without limitations. Hybrid AI paves the way for this future, with the Wallaroo Universal AI Inference Platform serving as the essential tool to achieve it.
By breaking down the silos of infrastructure, Wallaroo enables enterprises to achieve unprecedented speed, flexibility, and control over their AI operations – no matter where the data lives or what hardware is available.
Contact Us to discover how Wallaroo.AI can help you accelerate your hybrid AI journey with advanced automation, inference orchestration and unified control.
