GenAI / Agentic AI

Go, Llama. Go! Accelerate AI Inference on GPUs, No Compromise.

8 min read
January 31, 2025
By Vid Jain

Enterprises are exploring LLMs with customizations for a wide range of use cases – from healthcare, financial services to marketing – to be deployed securely in their cloud or on-prem environments. These customizations are important to meet the business needs and can include RAG deployments, or even runtime optimizations.

The opportunity is immense, but deploying custom LLM inference microservices can be slow, complex, and expensive. The engineering effort of packaging, deploying and updating custom inference services combined with the hardware cost of running these in production and monitoring them will severely limit the enterprise adoption of agentic AI, multi-model AI, and the general application of Generative AI beyond select, high-ROI use cases.

Packaging, configuring and standing up a microservice can take hours to over a day of an ML engineer’s time for each new iteration of a model.
Inference hardware can be costly – $200/day (over $50K/year) for access to a single H100 GPU in Azure for example to run “smaller” LLMs such as Llama 8B, and double that for Llama 70B.
Enterprise features, such as encryption, audit logs, and model observability data typically require extra engineering effort and reduce performance.

That means the deployment and inference cost of a single model at high volume or several models in a single combined AI workflow could easily top $1M/year (e.g. 7 or more deployed LLMs similar to Llama 70B), and if your model needs the latest chips just securing enough HW can itself be a challenge. What’s more, as available models multiply rapidly (DeepSeek’s low-cost training and open-sourcing means there will be a lot more of these), infrastructure options increase, and the need for more efficient deployments becomes critical, Enterprise AI teams will be hard pressed to keep up.

At Wallaroo.AI, we’ve built an inference platform for turnkey optimized inference that is agnostic to model types and hardware, and can quickly support newer and newer models (e.g. the next DeepSeek) with efficiency. Our platform, Wallaroo, can reduce deployment time from a day to 5 minutes and increase throughput by up to 4X, while not compromising on enterprise features, accuracy or flexibility. In addition, for latency-insensitive use cases (e.g. content summarization in batch), enterprises do not often need the latest, most expensive hardware to meet SLAs when using Wallaroo. For example, for Llama 8B and 70B models, inference on Wallaroo + A100 has over 60% higher throughput than on Nvidia Triton + H100 (tok/seconds).

Wallaroo supports optimized ML, AI, and GenAI inference on x86, ARM, GPUs, IBM Power (PPC) – but here we will focus on the most common HW for LLM inference – Nvidia GPUs – and the benefits Wallaroo brings to inference on these. We will also focus here on a common model framework that we see enterprises working with – Meta’s Llama (3.x series) – other model types and HW will be discussed in future posts.

NVIDIA’s AI Inference Hardware and Software Offerings.

Nvidia, as the category leader, has shaped the market’s understanding of inference and performance. Nvidia provides a range of hardware and software offerings. On the hardware side, A100s and H100s are the most commonly used and while A100’s are easily available in the major clouds, the H100s are supply constrained and can take time to get access to.

Nvidia’s software strategy in a nutshell is:

CUDA is their AI software library that is needed to run workloads on their HW
Triton Inference Server is used to support the inference microservices
TRT-LLM runtime is an optimization library designed to increase inference performance on NVIDIA HW – in our testing, we’ve seen up to 2X throughput and latency improvements when using it with Wallaroo.AI

These components combine in a layered stack to deliver the inference microservice:

Finally, there is NIMS (NVIDIA Inference Microservices) that are pre-built images for deploying inference microservices for a variety of models. If NIMs supports the model and hardware that you want to deploy, you can easily deploy the inference microservice and start using them quickly without any packaging effort, but with out-of-the-box performance and with additional effort still needed for enterprise features such as access control, encryption, auto-scaling, monitoring, etc.

The Wallaroo AI Inference Platform.

Our mission is to make enterprise-grade AI inference dramatically easier to deploy, more efficient to operate and scale – with full flexibility across models, use cases & infrastructure.

Wallaroo provides the three fundamental capabilities that are essential for enterprises to deploy and operate AI inference workloads for real-world use cases, including cloud, on-prem, and edge deployments. These workloads can be as simple as single ML models to complex multi-modal or agentic workflows that involve multiple models and steps.

Our solution includes:

A high performance Inference engine
AI Operations center with Enterprise AI capabilities
An Integration Toolkit for support and integration in your enterprise environment

Wallaroo delivers lower cost, higher performance inference and easier and faster AI workflow deployment, optimized for both MLOps and LLMOps across a wide range of infrastructure. It is market validated with enterprise customers and infrastructure partners such as Oracle, IBM, Intel, & Microsoft Azure.

Wallaroo’s inference engine serves a similar function as NVIDIA’s Triton on not only NVIDIA HW but across a diverse set of HW and associated AI runtimes and libraries, many (but not all) that are shown here:

Wallaroo.AI’s Approach to Accelerating LLM Inference

We aim to be the “Switzerland” of AI inference – any model, any hardware, anywhere.

First, Wallaroo eliminates the guesswork and engineering overhead of packaging and configuring all the artifacts, acceleration and runtimes that can be stood up as a custom microservice, in a couple of lines of python via the Wallaroo SDK/API:.

Wallaroo completely abstracts away underlying infrastructure with its model autopackaging & customization capabilities regardless of the use case (e.g RAG, Agentic AI) or runtimes.
Wallaroo allows AI teams to flexibly specify the deployment resources and scaling options while automating the mechanics of deploying highly-scalable inference microservices.
Single models, multi-step inference pipelines, and AB testing are all supported.

Second, our Rust-based inference engine is based on years of experience in very-high performance applications, such as high-frequency trading, and offers very low latency response in real-time use cases and high throughput in batch use cases.

All deployments are production-ready: connections are encrypted and authenticated, all requests are logged for audit and observability purposes, and load-based auto-scaling (including scale-to-zero) is built-in.
Built-in dynamic and concurrent batching capabilities help maximize HW utilization and performance in real-time concurrent user request scenarios while also minimizing latency.
Extensible LLM optimization framework that allows AI developers to choose from multiple runtimes/acceleration libraries, including TRT-LLM, vLLM, Llama.cpp, as well as HW optimized ONNX, PyTorch and others.

Other Wallaroo capabilities and related topics of interest for enterprise AI are discussed here.

Wallaroo + NVIDIA: Accelerated Inference Benchmarks

Now to the meat of it! We’re going to present benchmarks for Llama 3.1 8B on Nvidia H100s. We’ve benchmarked on A100s as well as other model types; the results here are representative.

Inference Microservice Set up Time.

The persona here is a Principal Machine Learning Engineer

	Wallaroo	NVIDIA Triton	Observations and learnings
LLM packaging & deployment time of llama 3.1 8B on TRT and Triton	1 hour	2 days	Identifying the correct implementation of Triton and TRT with batching configurations requires a learning curve on the AI development side. Wallaroo offers simplified batching configurations out of the box via a simplified SDK.
Accelerator configuration changes (A100 to H100)	< 5 min	1 day	NVIDIA Triton required full compilation and redeployment of the service vs. Wallaroo that handles the hardware change with a deployment configuration label.
LLM scaling & orchestration setup time	< 1 min	2 hours	Setting up load balancing and additional scaling configurations with Triton require advanced K8s manipulation. Wallaroo offers autoscaling out of the box.

Inference Latency and Throughput

Note that for the Llama 8B model tested above, average throughput with the default Wallaroo deployment on A100 HW was benchmarked at 1070 tok/s, with average latency of 2.7 seconds,

Wallaroo + NVIDIA: Happy developers, Lower Cost Per Token, No Compromises

Supporting NVIDIA hardware and software libraries is core to delivering on our mission and will help drive adoption, accelerate enterprise AI inference on Nvidia’s silicon. Wallaroo:

Significantly reduces the time to stand-up custom inference microservices and simplifies iterating through settings and hardware options in cloud, edge or on-prem.
- AI developers, including data scientists, get more done, faster.
- Reduction for Triton H100s and A100s: from a day to 5 minutes.
- For the Llama 8B model benchmarked as above, the average throughput is 60% higher with Wallaroo on A100 vs. Triton on H100. For latency-insensitive use cases, A100s vs. H100s becomes a good deployment option.
Wallaroo enables teams to use the most advanced model runtimes (vLLM, TRT, llama.cpp, etc) through an extensible model framework, providing a unified developer experience and best-of-breed AI inference performance.
Wallaroo’s built-in LLM continuous batching and concurrent connection management capabilities offer comparable latency in addition to throughput improvements in certain scenarios VS. Triton, while adding enterprise-required features such as encryption, authentication, auto-scaling, observability and portability across hybrid infrastructure environments.
- Get the most out of Nvidia’s hardware with minimal engineering effort – up to 2.4X higher throughput benchmarked with Llama models.
Extends and integrates Nvidia’s existing software investments to provide a full-featured enterprise AI inference-stack with out-of-the box model monitoring, lifecycle management, workflow automation and more.
Provides a flexible and easy way to integrate and support an ever-increasing set of AI software runtimes, model types and hardware architectures in a fast-evolving AI landscape.

If you’re involved in Enterprise AI, these capabilities let you iterate and validate your use cases faster whether in pre-production testing or production. You don’t need to compromise either by quantizing or simplifying your models and potentially reducing accuracy, nor by giving up enterprise features like encryption, auditability or model observability. Of course, if quantizing models gives you better performance with acceptable accuracy, we support that as well and will bring the same speed-of-deployment and efficient inference no matter the underlying runtimes and hardware.

We would really love to hear from you on your AI use cases and challenges and we are happy to delve further into what’s been presented here. Please reach out to us at info@wallaroo.ai.