Private OpenAI Compatible API Gateway on Kubernetes with vLLM

Private OpenAI Compatible API Gateway on Kubernetes with vLLM

Building a Private OpenAI-Compatible API Gateway on Kubernetes with vLLM

TL;DR: For accuracy pick Qdrant, for scale pick Milvus, for simplicity pick pgvector. — the table below saves you hours, then we unpack each option.

For enterprise software engineering organizations, relying on public AI API providers exposes infrastructure to severe data privacy risks, compliance violations (SOC2, HIPAA, GDPR), unpredictable API rate limits (HTTP 429 throttling), and escalating per-token billing expenditures. Self-hosting open-source Large Language Models (LLMs) inside a private cloud Kubernetes (K8s) cluster provides total data sovereignty, zero external data egress, deterministic latency SLAs, and massive cost savings at scale.

By pairing vLLM (which natively exposes an OpenAI-compatible REST server) with Kubernetes, NVIDIA GPU Operator, Istio Ingress Gateway, and KEDA (Kubernetes Event-driven Autoscaling), platform engineering teams can deploy a self-healing, multi-tenant private AI gateway. This gateway scales GPU pod replicas dynamically based on pending request queues and KV cache memory metrics, providing seamless compatibility with existing codebases built on standard OpenAI client SDKs.

This technical guide provides a cloud-native architectural blueprint, complete production Kubernetes YAML manifests (Deployment, PVC, Istio VirtualService, and KEDA ScaledObject), an asynchronous Python load testing suite, production failure modes, and an operational zero-downtime deployment playbook.

Blueprint: Cloud-Native Private AI Gateway

The private AI gateway decouples client microservices from underlying GPU hardware management. Microservices send standard OpenAI-formatted API requests (/v1/chat/completions) to an Istio Ingress Gateway, which terminates TLS and routes traffic across a dynamic pool of vLLM pod replicas running on GPU-accelerated Kubernetes worker nodes.


Client Microservices (Python, Go, Node.js, Rust)
       │ (Standard OpenAI API REST / SSE Request)
       ▼
┌────────────────────────────────────────────────────────────────────────┐
│                      Istio Ingress Gateway / Envoy                     │
│  - SSL/TLS Termination & JWT Authentication Header Parsing             │
│  - Long-Lived Server-Sent Events (SSE) Streaming Timeout Configuration  │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                   vLLM Kubernetes Service Cluster IP                   │
└───────┬────────────────────────────────────────────────────────┬───────┘
        │                                                        │
        ▼                                                        ▼
┌──────────────────────────────┐                         ┌──────────────────────────────┐
│  vLLM GPU Pod 1              │                         │  vLLM GPU Pod 2              │
│  - Llama 3.1 70B (AWQ 4-bit) │                         │  - Llama 3.1 70B (AWQ 4-bit) │
│  - AsyncLLMEngine (PagedAttn)│  ◄── Shared PVC ──────►  │  - AsyncLLMEngine (PagedAttn)│
│  - Prometheus Exporter :8000 │      (NFS/EFS Volume)   │  - Prometheus Exporter :8000 │
└──────────────┬───────────────┘                         └──────────────┬───────────────┘
               │                                                        │
               └───────────────────────────┬────────────────────────────┘
                                           ▼
┌────────────────────────────────────────────────────────────────────────┐
│         KEDA Autoscaler (Kubernetes Event-driven Autoscaling)          │
│  - Queries `vllm:num_requests_waiting` from Prometheus Engine          │
│  - Dynamically Scales GPU Pod Replicas (1 ──► N Active Pods)           │
└────────────────────────────────────────────────────────────────────────┘

Core Platform Components

  • NVIDIA GPU Operator: Automatically manages GPU driver installation, CUDA runtime libraries, NVIDIA Container Toolkit, and DCGM (Data Center GPU Manager) metric exporters across Kubernetes worker nodes.
  • vLLM OpenAI API Server: Exposes an OpenAI-compatible HTTP interface backed by PagedAttention and continuous batching, serving requests at maximum memory bandwidth.
  • KEDA Metric Scaler: Scales vLLM pod count based on custom Prometheus metrics (e.g., pending request queue depth `vllm_num_requests_waiting` and KV cache utilization `vllm_gpu_cache_usage_perc`) rather than generic CPU/RAM metrics.
  • Istio Service Mesh: Enforces client rate limiting, provides mutual TLS (mTLS) between microservices, handles SSE streaming timeout configuration, and orchestrates zero-downtime canary traffic splits.

Cloud AI Deployment Comparison

Evaluating self-hosted Kubernetes LLM infrastructure against cloud provider APIs involves comparing data sovereignty, cost scaling models, latency predictability, and deployment flexibility:

Evaluation Vector Self-Hosted K8s + vLLM AWS Bedrock Azure OpenAI Service Anyscale / Modal Cloud
Data Sovereignty & Egress 100% Private (Zero External Egress) AWS VPC (Cloud Managed) Azure VPC (Cloud Managed) Data egress to vendor cloud
Cost Model at High Volume Fixed GPU Node Hourly Rate Per-1M Token Consumption Rate Per-1M Token Consumption Rate Per-GPU Second Billing
OpenAI SDK Compatibility 100% Native Compatibility Requires AWS Boto3 SDK Native Compatibility Native Compatibility
KEDA GPU Queue Autoscaling Native Metric-Driven Queue Scaling Managed Auto-scaling Provisioned Throughput Units Serverless Auto-scaling
Multi-LoRA Adapter Swapping Native (Dynamic Multi-LoRA in vLLM) Requires Custom Import Step Requires Custom Deployment Supported
Zero-Downtime Rollouts Istio Canary / Blue-Green Splits AWS Traffic Shift Azure Deployment Slots Version Aliases
Custom PagedAttention Tuning Full Control (Block Size, APC) No (Black Box) No (Black Box) Limited

Enterprise Kubernetes Manifest Architecture

Below are production Kubernetes manifests defining GPU node scheduling, vLLM pod deployments, shared PVC model caching, Istio routing, and KEDA metric autoscaling.

Step A: vLLM Deployment & PVC Manifest (`vllm-deployment.yaml`)


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hf-model-cache-pvc
  namespace: ai-gateway
spec:
  accessModes:
    - ReadWriteMany # Allows multiple GPU pods to read cached model weights simultaneously
  resources:
    requests:
      storage: 200Gi
  storageClassName: efs-sc # Cloud Shared NFS/EFS Storage Class
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-70b
  namespace: ai-gateway
  labels:
    app: vllm-llama3-70b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama3-70b
  template:
    metadata:
      labels:
        app: vllm-llama3-70b
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      # Ensure pod schedules ONLY on GPU-equipped worker nodes
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      nodeSelector:
        accelerator: nvidia-h100 # Target NVIDIA H100/A100 GPU nodes
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.5.4
          imagePullPolicy: IfNotPresent
          command:
            - "python3"
            - "-m"
            - "vllm.entrypoints.openai.api_server"
            - "--model=meta-llama/Meta-Llama-3.1-70B-Instruct"
            - "--quantization=awq"
            - "--tensor-parallel-size=4" # Split across 4 GPUs inside pod
            - "--gpu-memory-utilization=0.92"
            - "--max-model-len=16384"
            - "--enable-chunked-prefill"
            - "--port=8000"
          env:
            - name: HUGGINGFACE_HUB_CACHE
              value: "/root/.cache/huggingface"
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: token
          ports:
            - containerPort: 8000
              name: http
          resources:
            limits:
              nvidia.com/gpu: "4" # Request 4 NVIDIA GPUs
              memory: 128Gi
              cpu: "16"
            requests:
              nvidia.com/gpu: "4"
              memory: 64Gi
              cpu: "8"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120 # Model load delay allowance
            periodSeconds: 10
          volumeMounts:
            - mountPath: /root/.cache/huggingface
              name: model-cache-volume
      volumes:
        - name: model-cache-volume
          persistentVolumeClaim:
            claimName: hf-model-cache-pvc

Step B: KEDA Metric Scaler Manifest (`keda-autoscaler.yaml`)


apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-gpu-autoscaler
  namespace: ai-gateway
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-70b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300 # Wait 5 mins before scaling down GPU nodes to prevent thrashing
  pollingInterval: 15
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        # Scale UP if average pending request queue depth exceeds 5 requests
        query: sum(vllm:num_requests_waiting{app="vllm-llama3-70b"})
        threshold: '5'

Step C: Istio VirtualService & Streaming Timeout Manifest (`istio-routing.yaml`)


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ai-gateway-ingress
  namespace: ai-gateway
spec:
  hosts:
    - "ai-gateway.internal.domain"
  gateways:
    - istio-system/ingress-gateway
  http:
    - match:
        - uri:
            prefix: /v1
      timeout: 300s # Enable 5-minute long timeout for SSE streaming responses
      route:
        - destination:
            host: vllm-llama3-70b.ai-gateway.svc.cluster.local
            port:
              number: 8000
          weight: 100

Hands-On: End-to-End Kubernetes Async Load Test

Below is a production Python script that executes asynchronous concurrent requests against the Kubernetes vLLM Ingress API, measures response time percentiles (p50, p95, p99), tracks throughput (tokens/sec), and handles HTTP status errors.


import os
import sys
import time
import asyncio
import statistics
import logging
from typing import List, Dict, Any

try:
    from openai import AsyncOpenAI
except ImportError:
    print("OpenAI SDK required. Install via: pip install openai")

# ---------------------------------------------------------------------------
# 1. Logging & Client Setup
# ---------------------------------------------------------------------------
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("k8s_load_test")

GATEWAY_URL = os.getenv("GATEWAY_URL", "http://ai-gateway.internal.domain/v1")
API_KEY = os.getenv("API_KEY", "sk-k8s-private-gateway-key")
MODEL_NAME = os.getenv("MODEL_NAME", "meta-llama/Meta-Llama-3.1-70B-Instruct")

CONCURRENT_REQUESTS = 20  # Number of simultaneous client connections to simulate

client = AsyncOpenAI(
    api_key=API_KEY,
    base_url=GATEWAY_URL
)

# ---------------------------------------------------------------------------
# 2. Worker Task: Transmit Request & Measure Latency
# ---------------------------------------------------------------------------
async def send_load_test_request(task_id: int) -> Dict[str, Any]:
    start_time = time.time()
    try:
        response = await client.chat.completions.create(
            model=MODEL_NAME,
            messages=[
                {"role": "system", "content": "You are a Kubernetes cloud infrastructure auditor."},
                {"role": "user", "content": f"Task #{task_id}: Explain how KEDA scales vLLM pods based on Prometheus metrics."}
            ],
            max_tokens=250,
            temperature=0.3,
        )
        latency = time.time() - start_time
        tokens = response.usage.completion_tokens if response.usage else 0
        
        logger.info(f"  [Request #{task_id:02d}] Success in {latency:.2f}s | Returned Tokens: {tokens}")
        return {"success": True, "latency": latency, "tokens": tokens}
        
    except Exception as err:
        latency = time.time() - start_time
        logger.error(f"  [Request #{task_id:02d}] ❌ Error after {latency:.2f}s: {err}")
        return {"success": False, "latency": latency, "tokens": 0}

# ---------------------------------------------------------------------------
# 3. Main Benchmark Orchestrator
# ---------------------------------------------------------------------------
async def run_k8s_load_test():
    logger.info(f"🚀 Launching Kubernetes Private AI Gateway Load Test...")
    logger.info(f"Target Gateway: {GATEWAY_URL}")
    logger.info(f"Target Model: {MODEL_NAME}")
    logger.info(f"Concurrent Worker Tasks: {CONCURRENT_REQUESTS}\n" + "-"*60)
    
    start_suite = time.time()
    
    # Launch concurrent worker tasks simultaneously
    tasks = [send_load_test_request(i) for i in range(1, CONCURRENT_REQUESTS + 1)]
    results = await asyncio.gather(*tasks)
    
    total_time = time.time() - start_suite
    successful = [r for r in results if r["success"]]
    latencies = [r["latency"] for r in successful]
    total_tokens = sum(r["tokens"] for r in successful)
    
    if latencies:
        latencies.sort()
        p50 = statistics.median(latencies)
        p95 = latencies[int(len(latencies) * 0.95) - 1]
        p99 = latencies[-1]
        
        logger.info("\n" + "="*60)
        logger.info("      KUBERNETES AI GATEWAY LOAD TEST RESULTS")
        logger.info("="*60)
        logger.info(f"Total Execution Time: {total_time:.2f} seconds")
        logger.info(f"Successful Requests: {len(successful)} / {CONCURRENT_REQUESTS}")
        logger.info(f"Total Tokens Generated: {total_tokens}")
        logger.info(f"Average Token Speed: {total_tokens / total_time:.2f} tokens/second")
        logger.info(f"Average Request Latency: {sum(latencies)/len(latencies):.2f} seconds")
        logger.info(f"p50 Latency (Median): {p50:.2f} seconds")
        logger.info(f"p95 Latency: {p95:.2f} seconds")
        logger.info(f"p99 Latency: {p99:.2f} seconds")
        logger.info(f"Effective Request Throughput: {len(successful) / total_time:.2f} req/sec")
    else:
        logger.error("❌ All load test requests failed. Inspect Kubernetes Ingress logs.")

if __name__ == "__main__":
    asyncio.run(run_k8s_load_test())

Production Playbook: Zero-Downtime Rollouts & Storage

Operating a Kubernetes AI gateway requires strict adherence to cloud-native operational best practices:

  • Persistent Volume Shared Cache (ReadWriteMany): Store Hugging Face model weights on a shared ReadWriteMany PVC (such as AWS EFS, GCP Filestore, or Ceph FS). When KEDA triggers a scale-up event from 1 to 4 pods, new GPU pods mount cached model weights instantly from the shared volume, reducing pod boot cold-start time from 15 minutes down to under 60 seconds.
  • Istio Canary Model Swaps: When updating model versions (e.g., upgrading from Llama 3 to Llama 3.1), deploy the new vLLM version as a separate Kubernetes Deployment. Use an Istio VirtualService weight split (e.g., 90% traffic to old deployment, 10% to new deployment) to validate model performance without risking downtime.
  • Taints & Tolerations for GPU Isolation: Add dedicated taints to GPU worker nodes (nvidia.com/gpu=present:NoSchedule). Configure vLLM deployments with matching tolerations to prevent lightweight web applications or ingress controllers from occupying GPU-enabled nodes.

Production Failure Modes & Edge Cases

Managing GPU workloads in Kubernetes introduces infrastructure-level failure modes:

A. Failure Mode 1: Pod Scale-Up Cold Start Delay

Symptom: When KEDA triggers a scale-up event during traffic spikes, pending user requests time out with HTTP 504 Gateway Timeout while waiting for new pods.
Root Cause: Loading a 70B parameter model from cloud storage into 4x H100 GPU VRAM takes 60 to 120 seconds. Readiness probes fail until model loading completes.
Mitigation: Maintain a minimum active replica pool (`minReplicaCount: 2` in KEDA), enforce shared PVC model caching via EFS, and set Istio request timeouts to 300 seconds.

B. Failure Mode 2: Prometheus Scrape Metric Lag

Symptom: KEDA fails to scale up pods until 2 minutes after a traffic surge has already degraded latency.
Root Cause: Prometheus scrape intervals configured to default 60-second periods introduce significant telemetry delay.
Mitigation: Configure Prometheus scrape intervals for vLLM pods to 5 seconds (`prometheus.io/scrape_interval: "5s"`) and lower KEDA's `pollingInterval` to 10 seconds.

Last updated: September 1, 2026 -- reviewed for technical accuracy. Some benchmarks and API details evolve quickly; verify against the official docs linked below before production use.

Your turn: Which pattern matched your stack? Drop a comment or try the related guides below.

Sources & Further Reading

Related on AI SaaS Edu

Questions We Get Asked

Why is GPU queue length a better metric than CPU/RAM utilization for KEDA autoscaling of LLM workloads?

CPU and RAM utilization are terrible indicators for LLM scaling because a vLLM pod reserves ~90% of GPU VRAM up front upon initialization for weights and KV cache. As request load increases, GPU memory usage remains flat, and CPU utilization stays low because CUDA warps handle computation. Monitoring vllm_num_requests_waiting (the pending request queue depth) directly measures user demand, triggering scale-up events before latency degrades.

How does the NVIDIA GPU Operator expose VRAM metrics to Prometheus?

The NVIDIA GPU Operator deploys a DCGM-Exporter daemonset on every GPU worker node. DCGM-Exporter queries GPU telemetry via low-level NVML drivers, exposing metrics such as DCGM_FI_DEV_GPU_UTIL (GPU compute utilization) and DCGM_FI_DEV_FB_USED (VRAM allocation) to Prometheus scrapers on port 9400.

How can I mount fine-tuned model weights efficiently without downloading checkpoints on every pod restart?

Mount a shared ReadWriteMany Persistent Volume Claim (PVC) backed by an enterprise NFS, AWS EFS, or Azure Files storage class to /root/.cache/huggingface. When a pod initializes, vLLM checks the local mount path first. If model weights exist, initialization bypasses external network downloads, eliminating Hugging Face API rate limits and reducing cold-start latency.

What ingress controller configuration is required to support long-lived SSE streaming connections?

Standard ingress controllers enforce default request timeouts (e.g., 60 seconds). For streaming responses or long-context LLM requests, configure explicit timeout annotations in your Ingress or Istio VirtualService manifest (e.g., proxy-connect-timeout: "300s" and proxy-read-timeout: "300s"). Additionally, disable response buffering in Envoy to allow server-sent events (SSE) to stream to clients immediately.

How does Istio VirtualService manage zero-downtime model swaps during canary deployments?

An Istio VirtualService routes incoming HTTP traffic across multiple Kubernetes Service subsets based on explicit weight percentages. During a canary deployment, platform engineers deploy a new model version alongside the active deployment and set weight routes to 90% primary / 10% canary. Once telemetry verifies zero 5xx errors, traffic is shifted to 100% canary before tearing down the old deployment.

What node taints and tolerations should be configured to isolate GPU workloads?

Taint GPU worker nodes with key=nvidia.com/gpu, value=present, effect=NoSchedule. In your vLLM pod spec, specify matching tolerations alongside a nodeSelector (e.g., accelerator: nvidia-h100). This ensures non-GPU pods can't schedule on expensive GPU nodes while guaranteeing that vLLM pods land exclusively on high-performance GPU hardware.

You may like these posts

Previous Post Next Post

Contact Form