Introduction
AMD has an opportunity to significantly increase its market share in the GPU inference market, which will triple to $150B by 2027. AMD has an excellent product in its latest GPUs, which already outperform Nvidia in raw AI compute (validated in our own benchmarks), but lag behind Nvidia in sales due to gaps in its software.
These gaps are higher in the stack then AMD’s ROCm, where AMD has significant investment in kernel optimizations and inference layers such vLLM and SGLang, yet there is still substantial friction for adoption. While Superlabs like OpenAI will integrate HW into their own software stack, Enterprise Customers and (and the clouds they run on) will need much easier AI deployment of Enterprise-grade inference microservices, fast workload migration, advanced inference optimization, flexible deployment options, and enterprise-grade control & operations. Beyond raw performance metrics, new AI silicon adoption will hinge mostly upon the software layer that enables it.
The critical criteria are: Enterprise AI developer cycle time in deploying & maintaining inference applications, the end-to-end request latency and throughput under production loads, the ability for enterprise-grade control of deployments and the quantifiable effectiveness (e.g. generation accuracy, infrastructure cost savings per inference, reliability/error rates ) of the resulting AI application.
Wallaroo can help AMD close these software gaps, driving an additional $12 billion in GPU sales in 2027 (equivalent to 8% market share) and adding to the momentum from its OpenAI deal. Instead of asking if that’s too much share growth, the questions are: what needs to be put in place in 2026 to get there in 2027 & why wouldn’t AMD’s share grow with enterprises if its chips perform competitively and are easy to use? The assumption that hyperscalers, neoclouds, and enterprises will each build their own custom software to fill these gaps is misguided as it would lead to interoperability issues, and be hindered by a lack of necessary skills in enterprises and also by hyperscalers prioritizing their in-house chips while having to support multiple chip suppliers.
A Wallaroo + AMD solution will deliver a uniform best-in-class inference stack for all of AMD’s end-users so they can easily experience the power of AMD hardware for their applications and migrate workloads from Nvidia in minutes.
- Ship AI fast via low-friction deployment + migration of AI workloads from Nvidia to AMD.
- Run AI fast via superior overall performance for complex and agentic workflows, combining AMD’s power with Wallaroo’s efficient Rust-based software and features including orchestration, smart routing, disaggregated serving, and smart batching.
- Enterprise-grade, production ready microservices with authentication, encryption, advanced orchestration including for prefill/decode production clusters, observability, and lifecycle management.
- Out-of-the-box support for on-prem/edge and private AI deployments needed by regulated industries and governments.
In anticipation of the rapid evolution of AI infrastructure, Wallaroo.AI‘s software was built with a highly extensible architecture to easily integrate new AI runtimes, acceleration libraries and HW. This flexibility enables seamless support for specialized GPUs like the prefill-optimized Rubin CPX. Wallaroo easily extended its enterprise-grade capabilities by leveraging its built-in support for CUDA libraries and compatible runtimes like vLLM to run and orchestrate the latest LLMs with MOEs and/or long context processing. This same flexibility applies to AMD’s MI355X and upcoming MI400x chips. By supporting ROCm and integrating vLLM and SGLang, Wallaroo.AI empowers users to migrate AI inference workloads to AMD GPUs, capitalizing on AMD’s substantial AI acceleration investments.
This paper outlines how strategic software can drive AMD’s GPU market share, countering Nvidia’s dominance. It covers AMD’s opportunity, the cost to AMD from Nvidia lock-in, end-user needs, and gaps in AMD’s inference software. We discuss how Wallaroo’s technology provides a robust solution unlocking $12B in AMD GPU sales. The appendix compares AMD MI325X and NVIDIA H100 benchmarks, showing that AMD can deliver competitive performance with 70% lower cost.
Strategic Software To Drive Significant AMD Share Growth
In talking to neoclouds we often get the question: “Are you seeing enterprise demand for AMD GPUs?” Our answer is: If you make it easy for users to experience AMD’s performance for their applications, easy to migrate from their legacy infrastructure (e.g. NVIDIA) to AMD, and give them a production-ready experience, demand for AMD will explode.
In fact, AMD can accelerate its GPU sales from an estimated $7B in 2024 (compared to Nvidia’s $80B) to over $40B in 2027. Around $12B of this growth hinges on a strategic focus to enhance software for driving consumption across hyperscalers, neoclouds (and other infrastructure providers), and on-prem Enterprise and Government deployments. To match or surpass Nvidia’s software offerings, AMD needs to address key gaps, including:
- Ease of Deployment and Migration: simplifying the deployment process to AMD GPUs in general and also migration of workloads from Nvidia to AMD GPUs.
- Advanced Inference Optimization: Providing advanced and optimized inference capabilities for complex and agentic AI workloads.
- Production-Ready Inferencing: Ensuring standardized, production-ready inferencing with autoscaling, load balancing, access control, logging.
- Flexible Deployment Options: Facilitating easy remote and private on-prem deployments.
- Enterprise-Grade Operations & Orchestration: Offering enterprise-grade operations, control, and orchestration.
The gaps are higher in the stack then AMD’s ROCm, vLLM and other kernel optimizations and create friction for adoption. Wallaroo can close these gaps through tight integration with AMD’s foundational layer and our complementary inference software platform. This collaboration can create a groundbreaking solution to drive an incremental $12B in AMD sales by 2027, significantly exceeding the sales AMD would achieve by not having a full software stack to compete with Nvidia.
The presumption that all the hyperscalers, neoclouds and enterprises themselves will build custom software to fill any gaps is misguided for key reasons:
- If in fact they all did that, then there would be an almost limitless number of software stacks that would not be interoperable; i.e. there would be a lot of friction in moving between clouds or from an enterprise on-prem deployment to a neocloud. This is one of the reasons that Nvidia offers a uniform stack
- 95% of enterprises don’t have the skills to build what’s needed, and neoclouds don’t have the large scale experience working with Enterprises that hyperscalers have
- Hyperscalers have prioritized growing workloads on to their own in-house chips, and already have a lot of silicon suppliers, so it’s unlikely that having a best-in-class AMD experience will be their priority #1
Wallaroo + AMD unlocks value by providing a best in class inference solution:
- Best overall inference performance for complex & agentic workflows by combining AMD’s raw power with Wallaroo’s extremely efficient Rust-based software with smart model routing, disaggregated serving, smart batching and production-ready inference
- Most advanced set of features needed by Enterprises, including authentication, encryption, observability, and lifecycle management
- Out-of-the box support for on-prem/edge and private AI deployments needed by regulated enterprises, governments and in verticals such as manufacturing, life sciences or retail
- Fast, low-friction migration of workloads from Nvidia to AMD in minutes
AMD Market Opportunity & The High Cost of Nvidia Lock-in
First, let’s talk about the opportunity which is very large. The market for high-end GPUs for inference is expected to grow 3X from $50B in 2024 to $150B in 2027. Maintaining its current market share, and layering in the OpenAI deal, we estimate $30B in 2027 sales for AMD. However, if AMD could add another 8% market share from faster Enterprise adoption, it would mean a total of $42B in such sales for AMD in 2027!
- Nvidia stated (2024) that 40% of its DC sales were for AI inference, which means about $45B in 2024 sales for inference GPUs (over 90% of all inference GPU sales)
- Nvidia’s 2025 sales thus far imply at least $80B in 2025 GPU sales for inference, which is only increasing
- AMD did $5B in high-end GPU sales in 2024, and is forecast to do $7B in 2025, so about 8% of the relevant market
Nvidia’s Moat and End-User Needs
NVIDIA maintains its dominant market share through a powerful combination of hardware and software. The company achieves this through internal development and recently by investing $1.5 billion in AI software acquisitions, including Run:AI, DECI, and OctoML.
For AMD to gain market share it has to pierce NVIDIA’s hardware + software moat. Superior hardware (raw performance, or better latency/cost) is not sufficient. For AMD, we know this to already be the case because from our own testing, AMD’s Instinct GPUs are highly competitive with NVIDIA (eg H100), delivering 3-4X better results for throughput vs. cost on a variety of generative AI models while not growing market share from a year ago (see Appendix).
Let’s look at the end-user needs to drive hardware consumption:
What AMD needs to Boost Enterprise Market Share
We estimate that the total high end GPU market for inference will grow from ~$50B (2024) to $150B (2027). Looking at publicly available information, and market forecasts around growth in neoclouds and Enterprise/Government on-prem, we did our best to estimate sales in 3 buckets.
We then looked at AMDs potential sales in 2027 based on adding the OpenAI impact on top of its current market share vs. adding an additional 8% share from faster Enterprise adoption. Here is a summary of our estimated GPU sales for inference:
In order for AMD to achieve these results, it has to gain $3B-5B in each of three segments vs. its baseline scenario, i.e. AMD needs to gain share in hyperscalars, neoclouds/alternative AI infrastructure providers, and on-prem enterprise/government sales. Superlabs like OpenAI will integrate AMD into their own software stacks, but the vast majority of enterprises will look for software that already gets them there.
How can AMD do this, i.e. what will drive market share gains in each segment? We believe the answer is to provide a superior experience for enterprise AI on top of the excellent performance in the specific ways listed here:
We can remap and categorize these items into Day 1 (ship), Day 2 (run) and Day 3 (Operate):
Gaps in AMD’s Inference Software Stack to Boost Market Share
First, let’s categorize the components of a robust deployment and high-performance inference stack into five main functions and see where AMD has gaps.
Breaking it down in more detail, we can compare AMD’s full offering vs. Nvidia’s. The logos are placed to indicate which software functionality either Nvidia or AMD natively support.
While AMD has done an excellent job with the foundational layer (our benchmarks show compelling raw performance vs. Nvidia GPUs – see Appendix), we can see here the various software gaps.
Wallaroo’s complementary Software to boost AMD market share
What is Wallaroo?
Wallaroo is an AI inference software software solution that helps enterprise AI teams to easily deploy AI fast AI inference to any hardware, from cloud to edge, and then operate and orchestrate via a unified AI hub. It also helps infrastructure providers compete with Nvidia and hyperscalar offerings, while simplifying the migration of workloads from one architecture/environment (e.g. Nvidia on AWS) to another (e.g. AMD on Digital Ocean).
Wallaroo is in production usage at some of the world’s biggest brands, and has over 100 person-years of product and engineering development. Wallaroo is comprised of 3 broad software components:
- A “Wallaroo Engine” that provides ultrafast, production ready inference microservices
- A “Wallaroo AI Operations Hub” that provides centralized operations, orchestration, and lifecycle management of the deployed AI
- A “Wallaroo Integrations Toolkit” for flexible integration into almost any data science or data environment, application, or workflow.
At a high level:
Going one level deeper, we can draw out the components of Wallaroo here, with an indication where AMD or Nvidia have an software capability that is similar:
A Wallaroo + AMD integrated solution provides a complete, best-in-class inference solution:
Finally, we want to mention the fully integrated orchestration and routing stack within Wallaroo to deliver outstanding performance at scale. While Disaggregated Serving, discussed next, is an essential part of supporting high performance LLM inference, in Wallaroo it is combined with overall orchestration, load balancing, model routing in a modular architecture to deliver THE state-of-the-art serving engine available.
This means for example, in the case of high-volume agentic inferencing:
- Orchestration ensures the best available HW is stood up for the microservice
- Work is spread across autoscaled compute nodes
- Different models can be routed to different HW for processing
- Individual LLMs are split and optimized at their prefill/decode level
- Batching as needed ensures the underlying HW is maximally utilized
Wallaroo’s Disaggregated Serving Technology
As AMD and other silicon providers are aware, prefill-decode disaggregation is now a foundational distributed approach for LLM inference. Its importance is rapidly growing as its independent execution is the key to unlocking performance for long-context workloads that dominate high-value markets like AI-assisted coding. With the industry landscape evolving and the potential for future competition from custom-designed clusters, the ability to deliver a flexible and highly-orchestrated disaggregated solution is a critical differentiator.
This is where the synergy between Wallaroo and AMD creates a powerful market advantage. While runtimes like SGLang enable disaggregation on AMD hardware, Wallaroo provides the comprehensive orchestration layer required to operationalize it efficiently and at scale. Our platform intelligently directs the compute-heavy pre-fill stages to throughput-optimized AMD Instinct™ GPUs while assigning the memory-bound decode stages to the most suitable nodes.
The rise of Mixture-of-Experts (MoE) models like Kimi K2 and DeepSeek-R1 also amplifies the challenge as expert sharding creates operational complexity in deployment, routing, and failure recovery.. Wallaroo’s Another key architectural choice is a centralized Prefix/KV caching layer for better hit rates and efficiency through sharing as shown in the figure. Prefix/KV centralized caching is also popular in small-context applications such as voice chat.
Wallaroo’s modular approach can support these architectures as a natural extension to its ability to orchestrate complex, distributed workloads. Wallaroo can de-risk these next-generation AMD systems by providing the essential management layer for handling distributed expert placement, resilient routing, and sophisticated cache management. This provides the foundation for enterprises to reliably scale their most demanding AI applications on the joint AMD-Wallaroo platform.
Wallaroo’s Technology – Recap
While AMD offers compelling performance and cost benefits, there are key user roadblocks that generally hinder the adoption of new silicon in high-volume AI production environments. These roadblocks fall into Ship (Day 1), Run (Day 2), Operate (Day 3) categories, but also general flexibility to work in and across diverse infrastructure environments.
Wallaroo serves as a key enabler for AMD by simplifying the complexities of integrating, deploying, and managing AI models on AMD GPUs in cloud and edge environments, helping AMD overcome customer adoption challenges and fully realize the benefits of its hardware.
| Challenge Area | Challenge Description | Wallaroo’s solution |
| Day 0 – Built-in Integrations & Optimizations | AMD-based runtimes for LLMs require a significant amount of development time and configuration to be able to achieve optimal performance numbers | Out-of-the-box integration with AMD HW and Software, as well as vLLM and SGLang. To ensure AI developers are able to deliver high-performance in a repeatable manner when working with AMD, Wallaroo’s built-in integration of AMD-based runtimes will guarantee best-of-breed performance across the widest range of models including custom LLMs and agents, with full abstraction of complexity in manipulating low level libraries. Wallaroo.AI has a proven track record and key partnerships & integrations with silicon providers such as Ampere, IBM and Qualcomm enabling AI developers to realize the full potential of these AI silicon offerings. With Agentic AI workloads, AI developers can load pre-trained foundation model artifacts for small and large contexts (e.g llama 8B, llama 70B, qwen 32B etc.) from any public or private model registry onto Wallaroo. Wallaroo’s built-in vLLM & SGLang integrations and optimizations across AMD, NVIDIA, Qualcomm, IBM, TPUs and other emerging silicon will be invoked without any low level user manipulation of acceleration libraries to convert, compile and prepare the models for deployment onto the target accelerator. |
| Day 1 -Easy Deployment & Migration | Once models are optimized for AMD-based runtimes, there needs to be a path to easily deploy them on AMD and similarly migrate workloads away from NVIDIA and Emerging silicon (e.g TPUs, Qualcomm, Groq) while maintaining underlying model fidelity. | Easy Autopackaging of Microservices and Fast Migration from Nvidia and other silicon With Wallaroo’s autopackaging capabilities, AI developers have a path to easily migrate AI models and agents across hardware architectures and accelerations. During this process Wallaroo handles the necessary conversion, compilation and optimization of models for the target runtime and underlying hardware. This video showcases how Wallaroo enables AI developers to migrate AI models from NVIDIA to AMD in minutes. The example shows how in less than 10 minutes a developer can use the Wallaroo SDK in their current environment (e.g Azure) to load and migrate a large context LLM (llama 3.1 70B) from NVIDIA to AMD GPUs running in the same cloud (e.g Azure) or in a different cloud (e.g Digital Ocean), while preserving application fidelity using Wallaroo’s built-in OpenAI compatibility. |
| Day 2 – Automation, Orchestration & Scalability | As AI inference microservices are deployed to production, AI developers will need to ensure that those services are able to support the following production requirements:High availability & Load balancingAutoscaling Efficient Request HandlingOrchestration of AI inference steps (e.g prefill & decode, embedding generation & RAG) across GPU and CPU nodes | Out-of-the-box Production-Ready & Fast MicroservicesThe Wallaroo Rust-based AI Inference Engine uses an optimized, multi-threaded approach to handle incoming requests, prepare data, and run models efficiently.The inference engine’s autoscaling and smart batching (such as dynamic and continuous batching) features allow it to adjust hardware resources and maximize utilization, which helps ensure low latency and high throughput for AI applications.For complex AI systems like Agentic AI and large language models (LLMs), Wallaroo uses advanced techniques, including multi-stage inference serving and the transfer of KV Cache across GPU nodes, to optimize performance during the prefill and decode steps.The example below shows how the Wallaroo SDK is used to deploy a production-grade AI inference microservice (known as Wallaroo pipeline) on AMD with autoscaling requirements, while Wallaroo handles the full orchestration of underlying GPU nodes, in addition to standing up the necessary load balancing and request handling for production load requirements. Learn more here: https://wallaroo.ai/orchestrating-hybrid-agentic-ai-with-wallaroo/ |
| Day 3 – Orchestration and Control | To ensure proactive management of AI inference microservices in production at scale, AI developers and enterprise need to ensure they have the necessary tools & capabilities for:AI inference performance management AI model & agent evaluationOngoing Model iterations and updates | Unified AI Hub Wallaroo’s AI Hub is the AI control pane with centralized monitoring and orchestration of AI model & agent deployments across cloud, on-prem and edge targets. AI developers and teams can leverage the Wallaroo AI control plane to: Package AI inference microservice for deploymentAnalyze AI inference performance (throughput, latency, scaling, GPU consumption)Evaluate AI models and agents for anomalies & unwanted behaviors (e.g LLM as a judge, hallucinations and toxic/harmful language detection, drift detection etc.)Testing (A/B testing, canary and shadow testing) & Automated redeployment of models and agents with zero-downtime in production. Learn more: https://wallaroo.ai/orchestrating-hybrid-agentic-ai-with-wallaroo/ |
Summary – The Opportunities that Wallaroo Unlocks
Wallaroo can unlock an incremental $12B in AMD sales in 2027 across hyperscalers, neoclouds, and enterprise/governments by making it very easy for end-users to experience the power of AMD for their applications and helping them migrate workloads from Nvidia and other infrastructure. In particular, Agentic AI, On-prem and Private AI will drive much of the consumption and Wallaroo + AMD can offer a much better user outcome.
Appendix – Benchmarks (Wallaroo + AMD vs. Nvidia Stack)
For 8B – 70B parameter models we see a very significant cost reduction from 70% to 85% on a per token basis. In the case of 70B models, we compared two of Nvidia’s H100 GPUs to one AMD MI325x.
Llama 3.1 8B
1000 Concurrent Connections, Max 1024 Tokens, Runtime: vLLM
Llama 3.1 70B
1000 Concurrent Connections, Max 1024 Tokens, Runtime: vLLM
Blog Collaborators: Younes Amar, and Bharadwaj Pudipeddi.
Additional Resources:
