Introduction
In order to gain a competitive business advantage in today’s world, modernizing enterprise and data center infrastructure must leverage the rapidly growing space of AI for producing performant and efficient AI applications while retaining absolute security, compliance, and control over their data.
Concerns over security, privacy, and compliance are leading to a growing consensus among public sector and enterprises that turning to On-Premises and Virtual Private Cloud infrastructure air-gapped environments is the solution for complete sovereignty over their AI production environment, achieving an optimal balance of agility, scalability, and automation.
What is an Air Gap environment? An air-gapped deployments isolate a system from external connections, providing benefits of enhanced data security, regulatory compliance, operational control, and IP protection to name a few. This is especially important to organizations in regulated industries such as defense, healthcare, finance.
The Wallaroo AirGap™ Universal AI Inference Platform
Wallaroo.AI, a leader in AI inference software for LLMOps and MLOps in the cloud, on-premises, and at the edge, is at the forefront of the AI revolution, delivering the critical requirements for air-gapped AI environments built directly into the platform.
In this blog, we will address how organizations can jumpstart their IT infrastructure modernization, enabling high-volume token throughput on finite infrastructure across diverse hardware architectures and use cases through air-gapped, high-speed AI inference with Wallaroo. By enabling a consistent installation, deployment, and management experience for AI models in air-gapped environments, either On-Premises or in a Virtual Private Cloud, Wallaroo helps to meet security, privacy, and compliance goals and establish resilient AI operations while ensuring AI production remains robust and efficient.
Installation of Wallaroo in an Air Gap Environment
Installing Wallaroo in an air-gapped cluster or server is designed to maintain isolation while ensuring full platform functionality for AI inference, microservice packaging, deployment and lifecycle management.
In non-air-gapped environments, installing Wallaroo has access to external networks to pull artifacts required as part of the installation from an internet connected source.
In an Air Gap environment, the process of pulling the necessary container images from an internet connected source is obviously not available. As such, a private container registry is used that is accessible from the air-gapped cluster where images are pushed and pulled for the installation.
For ease-of-use, Wallaroo provides the necessary automation to load the necessary images to a private image repository of the user’s choosing. When complete, the script also produces a yaml file which is used during installation to retrieve the images from their respective locations in the repository.
Installation Preparation and Verification
Before installing, Wallaroo ensures the environment (kubernetes, openshift, single node server etc.) meets the installation requirements by running preflight checks to validate cluster health, resource availability, and configurations, preventing later complications.
This results in the following preflight checks report
Wallaroo Installation Methods: Helm and KOTS
Air-gapped installations & upgrades with Wallaroo leverage Helm and KOTS. While Wallaroo AirGapTM installations offer a consistent experience with the standard Helm and KOTS installations of Wallaroo available public cloud environments or cloud marketplaces, Wallaroo AirGapTM installations feature a private & secure installation package, known as a Wallaroo AirGapTM bundle, that can be imported and managed into any enterprise private artifact registry.
Wallaroo AirGapTM Helm installation
For an air-gapped setup, configurable Helm settings help to define network connections as internal to ensure Wallaroo components will communicate within the isolated environment and no unauthorized external connections are allowed. This setting is fundamental to maintaining the integrity of the air-gapped environment.
Installing Wallaroo comes with several default settings, however organizations have flexibility to customize these by using a local values file to overwrite the defaults. The following example represents a Wallaroo installation with standard configurations for a helm installation.
Once pre-flight checks pass and settings are configured, Wallaroo can be installed via Helm.
helm install wallaroo
oci://[Registry Path]/wallaroo/<release_number>/wallaroo
--version <wallaroo_release_version_number>
--values values.yaml
--values image-values.yamlWhere
- values.yaml: The .yaml file containing the local values settings specific to the user and their environment.
- image-values.yaml: File generated in the Wallaroo Image Retrieval and Installation Preparation steps.
Wallaroo AirGapTM KOTS installation
kubectl kots install $RELEASE
--namespace wallaroo
--shared-password <password>
--airgap-bundle wallaroo.airgap
--license-file Customer_License.yaml
--kotsadm-registry User_Registry_Path
--registry-username Generic_Username
--registry-password <password>Where
- RELEASE: The release applicable to the customer’s Wallaroo license
- Wallaroo: The namespace to install Wallaroo into
- wallaroo.airgap: Wallaroo AirGapTM bundle that contains all necessary wallaroo files to install in the airgapped environment
- Customer_License.yaml: Wallaroo activation key/license file
- User_Registry_Path: Where all the wallaroo installation files are stored.
- Generic_Username: Registry Login Username
- <password>: Registry Login Password
Wallaroo AirGapTM Private python package repository configuration
Python libraries required by the included Python script are specified in the requirements.txt file and included in the .zip file. For air-gapped environments, administrators configure a private python repository that is referenced when uploading files, and can be configured to only use python packages found in the repository, ensuring only vetted packages are available in Wallaroo.
Learn more about Wallaroo Air-Gapped Installation Guide
Wallaroo AI Control Plane (Model Ops Center)
Post Wallaroo AirGapTM installation often presents unique challenges for comprehensive AI model lifecycle management across deployment to ongoing monitoring and updates due to the isolated nature of the environment.
The Wallaroo.AI Universal AI Inference Platform enables efficient AI model operations in air-gapped environments, meeting strict security, privacy, and compliance. The Wallaroo AI Control Plane, a self-service toolkit, delivers this capability through centralized automation, scaling, model management, observability, and optimization. It simplifies and automates model deployment, from packaging to serving, maximizing performance and efficiency while maintaining tight security and user management crucial for production-grade air-gapped AI.
Learn more about factory scale AI inference operations with Wallaroo.
Automated Model Packaging and Deployment
In Wallaroo, AI model deployment is simplified through Auto-packaging by bundling all necessary components into a single distributable package. Wallaroo provides powerful flexible model conversion from any common model framework wherever it may be on the spectrum from pre-trained to fine-tuned and heavily customized. Using Wallaroo’s SDK or API to upload and package models provides a fast and easy way to ensure consistency, portability, scalability, and reproducibility across platforms and environments.
For air-gapped environments, the benefit of Auto-packaging is that the resulting package contains all the components needed to deploy the model without the need for calls being made to external sources for supporting artifacts.
For example Custom Models, also known as Bring Your Own Predict (BYOP) in Wallaroo allows for custom model inference methods with supporting scripts and artifacts. These are used with pre-trained models and runtimes (PyTorch, Tensorflow, vLLM, etc) along with their supporting artifacts & dependencies.
Learn more:
- Custom Models (BYOP) in Wallaroo
- How RealPage is Streamlining Production MLOps/LLMOps with Wallaroo
- The Universal AI Inference Platform Technical Whitepaper
Secure Unified Model Management
Model deployment does not signify the end of the journey. In air-gapped AI solutions, effective collaboration and control are vital for AI teams due to the isolated nature of the environment. Wallaroo helps AI teams to collaborate securely through centralized model management and a secure model registry.
Built on the concept of workspaces, the Wallaroo AI Control Plane enables secure collaboration and full AI lifecycle control allowing flexible AI rollouts organized by use case, market, region, location, or team, supporting scalable model management and advanced strategies like A/B testing and Canary deploys. Closing the loop, in-line updates can replace models in seconds, without inference interruption, ensuring operational continuity within the air-gapped perimeter.
Learn more about Secure model management in Wallaroo:
Automated Observability
Beyond model management, Wallaroo provides automated, centralized observability for AI solutions, crucial for air-gapped environments. This capability delivers key metrics for proactive model health and performance, meeting throughput and latency needs. Wallaroo AssaysTM enables data scientists to configure validation checks, automate alerts for real-time model monitoring and data drift detection, generating explainability reports that offer a comprehensive understanding of model behavior and predictions, ensuring models remain effective in isolated air-gapped setups.
Learn more about Automated Observability in Wallaroo
Continuous Model Delivery
Automated Observability identifies model performance issues, while continuous model improvement helps achieve optimal AI model performance. Wallaroo’s centralized AI Workflow Automation reduces manual deployment risks and delays, saving time and resources. This extends to air-gapped environments, where administrators can configure a private Python package repository for vetted packages, enhancing security and compliance. Wallaroo’s workload orchestration enables automated analytics, MLOps/LLMOps, Agentic AI, and model retraining/fine-tuning for continuous air-gapped inference pipelines. Workflows can be scheduled or run continuously, offering flexibility for AI operations in both connected and secure air-gapped environments.
Learn more about Wallaroo’s centralized AI Workflow Automation
Wallaroo AI Inference Engine
The power unit of the Wallaroo platform is the Wallaroo AI Inference Engine built for turnkey optimized high-performance inference that is agnostic to model types and hardware configurations. The inference engine supports both batch and real-time inference through an optimized, multi-threaded architecture.
This unique ability to optimize efficient high-speed inference across a diverse spectrum of hardware, allows organizations to ease past pitfalls of Air Gap projects such as costly hardware acquisitions or compromising performance on suboptimal hardware resulting in unrealized operational success.
The Wallaroo Inference Engine architecture also leverages these heterogeneous hardware environments with an optimized, multi-threaded network server and orchestration layer that manages incoming requests, prepares data, and executes models. In addition built-in Autoscaling capabilities adjust resource utilization using pre-configured triggers based on real-time demand, ensuring optimal inference latency and throughput.
Learn more about the power of the Wallaroo Inference Engine performance on diverse hardware:
- Wallaroo.AI Software on Arm®: A Best-of-Breed AI Solution
- Optimizing AI Inference and Governance with Wallaroo on IBM® Power®
- Go, Llama. Go! Accelerate AI Inference on GPUs, No Compromise.
Conclusion
Organizations prioritizing security, compliance, and control in their AI production air-gapped environments, the Wallaroo Universal AI Inference Platform enables full end-to-end AI lifecycle automation regardless of hardware environment. Equipping enterprise Al teams with the means to quickly and efficiently operationalize Al production on diverse infrastructures and architectures supporting a full spectrum of use cases.
By enabling a consistent, simplified experience for installation, deployment, and management, coupled with capabilities like automated model packaging for air-gapped environments, unified model management, intelligent autoscaling, automated observability, and continuous model delivery, Wallaroo empowers enterprises to achieve efficient, production-grade AI at scale for air-gapped environments whether on-premises or within a Virtual Private Cloud without compromising on security, privacy, or compliance goals, ultimately driving significant business competitive advantage.
Additional Resources
