Running AI/ML Workloads on Kubernetes

Assumes you're comfortable with scheduling and resource requests from Part 2, autoscaling from Part 12, and multi-tenancy from Part 13 — this chapter applies all three to the specific, GPU-shaped constraints AI/ML workloads add on top of everything else in this series.

Table of Contents#

1. Why This Part Exists
2. GPUs as a Schedulable Resource — Device Plugins
3. GPU Sharing Strategies: Time-Slicing, MIG, and MPS
4. Dynamic Resource Allocation — the Emerging Model
5. Gang Scheduling for Distributed Training
6. Kubeflow and the MLOps Lifecycle on Kubernetes
7. KServe Deep Dive — the InferenceService Architecture
8. LLM-Specific Serving: vLLM, KV Cache, and Continuous Batching
9. Autoscaling GPU and LLM Workloads
10. Multi-Tenancy for Shared GPU Clusters
11. Batch Inference vs. Real-Time Serving — Choosing the Right Pattern
12. Storage for AI/ML: Model Weights and Datasets
13. Observability for GPU Workloads
14. Model Versioning and Canary Rollouts for Inference
15. Cost Optimization for GPU Workloads
16. Data Scientist Self-Service — Notebooks Within Tenant Guardrails
17. A Full Worked Scenario: Deploying an LLM Inference Service
18. Part 14 CLI Cheat Sheet
19. Quick Reference: Every Tool in This Chapter, What It Actually Does
20. A Note on Reviewing GPU Configuration Changes
21. Common Mistakes and Interview Traps
22. Worked Practice Problems
23. Summary and What's Next

Why This Part Exists#

Per the industry research behind this series' chapter list, AI/ML workloads are now the dominant driver of new Kubernetes adoption — a fundamentally different capacity-planning problem from the CPU/memory-shaped workloads every other part of this series has assumed. A GPU is not just "a bigger CPU request" — it's an indivisible-by-default, extremely expensive, often supply-constrained resource that idles at real dollar cost the moment it isn't doing useful work, which reshapes scheduling, autoscaling, and multi-tenancy decisions in ways this chapter covers on top of everything Parts 2, 12, and 13 already established.

The throughline system gains one more component here: a recommendations team's model-serving deployment, recommendation-model, running inference on GPU nodes alongside the CPU-only checkout/catalog services from earlier parts. This is also the same recommendations team introduced in Part 13's multi-tenancy worked scenario — several of this chapter's worked examples build directly on the tenant boundary already established there, rather than treating GPU capacity as an isolated, separately-governed resource.

GPUs as a Schedulable Resource — Device Plugins#

Kubernetes has no native concept of a GPU — the Scheduler only understands CPU and memory as first-class resource types out of the box. GPUs (and other accelerators) become schedulable entirely through the Device Plugin framework, a well-defined extension point rather than a core Kubernetes feature.

apiVersion: v1
kind: Pod
metadata:
  name: recommendation-model-trainer
  namespace: recommendations
spec:
  containers:
    - name: trainer
      image: registry.internal/recommendation-trainer:2.0.0
      resources:
        limits:
          nvidia.com/gpu: 1   # GPUs are ALWAYS specified in limits, never requests

The GPU Operator (NVIDIA's, for the NVIDIA case specifically) automates the entire stack this depends on — the driver installation, the container toolkit, and the device plugin itself — as a set of Kubernetes-native operators, rather than requiring each of those to be manually installed and version-matched on every GPU node by hand, which was the norm before the Operator pattern (Part 4) was applied to this specific, previously painful bootstrapping problem. A new GPU node joining the cluster is bootstrapped automatically the same way any other Operator-managed dependency reconciles itself, rather than requiring a manual per-node setup script every time the node pool scales.

GPU Sharing Strategies: Time-Slicing, MIG, and MPS#

Without any sharing strategy, nvidia.com/gpu: 1 means exactly one Pod occupies one entire physical GPU for its whole lifetime — for many inference workloads that don't need a full GPU's throughput, this is significant waste on an extremely expensive resource.

# GPU Operator ConfigMap enabling time-slicing — this specific GPU
# now advertises 4 schedulable slices instead of 1 whole device
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  a100-4-way: |
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Dynamic Resource Allocation — the Emerging Model#

Dynamic Resource Allocation (DRA) is a newer Kubernetes-native mechanism for expressing accelerator requirements far more richly than the Device Plugin framework's simple integer-count model allows — ResourceClaim objects let a Pod ask for a GPU matching specific attributes (a minimum memory size, a specific MIG profile, a NVLink topology requirement) rather than just "one unit of nvidia.com/gpu."

apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaim
metadata:
  name: training-gpu-claim
  namespace: recommendations
spec:
  devices:
    requests:
      - name: gpu
        deviceClassName: gpu.nvidia.com
        selectors:
          - cel:
              expression: 'device.attributes["gpu.nvidia.com"].memory >= 40000'

DRA moves resource-matching logic out of the Scheduler's built-in, hardcoded extended-resource counting and into a structured, extensible claims model that a vendor's own driver can populate with rich device attributes — the same underlying idea as CSI did for storage (Part 3) and CNI did for networking (Part 3), applied to accelerators. This is a genuinely active area of the project: confirm DRA's current graduation stage (alpha/beta/stable) and API version against your specific cluster's Kubernetes release before relying on it for production scheduling decisions, since a fast-moving beta feature's exact CRD shape and default enablement can change between minor versions in ways the Device Plugin framework's long-stable API does not.

Gang Scheduling for Distributed Training#

A distributed training job spanning 8 GPU pods across multiple nodes has a requirement the default Scheduler (Part 2) doesn't natively satisfy: all 8 pods must be scheduled together, or the job should wait entirely — starting 5 of 8 workers and leaving 3 Pending wastes the 5 already-running GPUs on a job that can't make progress without every worker present.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: recommendation-model-training
  namespace: recommendations
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: registry.internal/recommendation-trainer:2.0.0
              resources: { limits: { nvidia.com/gpu: 1 } }
    Worker:
      replicas: 7
      template:
        spec:
          containers:
            - name: pytorch
              image: registry.internal/recommendation-trainer:2.0.0
              resources: { limits: { nvidia.com/gpu: 1 } }

Kubeflow and the MLOps Lifecycle on Kubernetes#

Kubeflow is a collection of Kubernetes-native components covering the full ML lifecycle, not one single tool — understanding which component solves which stage avoids reaching for the wrong piece.

Katib deserves one concrete example, since "automated hyperparameter search" is easy to gloss over as an abstraction: a Katib Experiment defines a search space (e.g. learning rate between 0.0001 and 0.1) and an objective metric to optimize, then launches many PyTorchJob trial runs (the previous section's CRD) automatically, each with a different hyperparameter combination, converging toward the best-performing configuration without a human manually launching and comparing each run by hand — the same "encode the operational pattern once as a controller, not as a runbook a human follows" principle from Part 4's Operator pattern, applied to the ML experimentation loop specifically.

The 2026 framing worth internalizing: per the earlier research, the industry-wide challenge has shifted from "can we train a model" to "can we operate it reliably at scale" — Kubeflow's components map directly onto that shift, with Pipelines and Katib addressing the training side and KServe addressing the operational, always-on serving side that increasingly dominates real infrastructure cost and complexity.

KServe Deep Dive — the InferenceService Architecture#

KServe's InferenceService custom resource abstracts model serving behind a consistent interface across frameworks (TensorFlow, PyTorch, scikit-learn, and LLM-specific runtimes), building on Knative Serving for request-driven autoscaling including scale-to-zero.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: recommendation-model
  namespace: recommendations
spec:
  predictor:
    model:
      modelFormat: { name: sklearn }
      storageUri: "s3://ml-models/recommendation-model/v3/"
      resources:
        limits: { cpu: "2", memory: 4Gi }
    minReplicas: 1
    maxReplicas: 10

InferenceService deliberately hides the Knative/Istio machinery underneath a single, framework-agnostic spec — a data scientist promoting a newly trained model to production interacts with storageUri and modelFormat, not with Knative Revisions or Istio VirtualServices directly, the same abstraction-hiding principle the Operator pattern (Part 4) applies generally: encode the operational complexity once, expose a simple domain-specific interface to everyone else.

LLM-Specific Serving: vLLM, KV Cache, and Continuous Batching#

Serving an LLM has a resource-usage shape fundamentally different from a traditional stateless HTTP service, and generic HPA/CPU-based thinking (Part 12) doesn't transfer cleanly — vLLM (the dominant open-source LLM inference engine) exposes the metrics that actually matter for this workload type.

Continuous batching is the single biggest throughput lever specific to LLM serving — unlike a traditional request-response service where each request is processed independently, vLLM dynamically interleaves multiple requests' token-generation steps into shared GPU batches, letting a new request join an already-running batch rather than waiting for the current batch to fully complete, which is what makes GPU utilization for LLM serving meaningfully higher than naive one-request-per-GPU-call serving would achieve.

Autoscaling GPU and LLM Workloads#

A naive requests-per-second HPA metric (Part 12) has no way to express the batching-capacity distinction from the previous section — two requests generating very different output lengths consume very different amounts of GPU capacity for the same "one request" unit, which is exactly why the metric choice below matters more here than for a typical stateless HTTP service.

Part 12 covered HPA/VPA/KEDA in general — GPU and LLM workloads specifically need KEDA's external-metric capability, since the signal that actually matters (queue depth, KV cache fullness) lives in the application/GPU layer, invisible to a Resource-type HPA metric entirely.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: recommendation-model-vllm
  namespace: recommendations
spec:
  scaleTargetRef: { name: recommendation-model-predictor }
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.observability.svc:9090
        query: vllm:num_requests_waiting
        threshold: "10"

This is a direct application of Part 12's Prometheus-adapter/KEDA pattern, with one GPU-specific wrinkle worth calling out: a scale-up decision here doesn't just create a new pod — it likely triggers a new GPU node provisioning event (Part 7's Karpenter, extended to GPU node pools) if no existing node has spare GPU capacity, and GPU node boot time (including driver/toolkit initialization via the GPU Operator) is typically markedly slower than a standard CPU node's boot time, extending exactly the scale-up latency window Part 12's worked scenario walked through — worth budgeting for explicitly in any LLM-serving SLO.

Multi-Tenancy for Shared GPU Clusters#

Part 13's multi-tenancy models apply directly to GPU capacity, with the economics sharpened by how expensive an idle GPU actually is compared to an idle CPU core.

Batch Inference vs. Real-Time Serving — Choosing the Right Pattern#

Not every inference workload should be a long-running InferenceService — a meaningful fraction of real ML workloads are batch: scoring a large dataset overnight, not answering individual live requests, and forcing that shape into an always-on serving pattern wastes GPU cost on idle time between batch runs.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-catalog-rescore
  namespace: recommendations
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: batch-scorer
              image: registry.internal/recommendation-batch-scorer:1.0.0
              resources: { limits: { nvidia.com/gpu: 2 } }
          restartPolicy: Never

Getting this choice wrong in the expensive direction — running a genuinely batch workload as an always-on InferenceService "because that's the pattern the team already knows" — is a common and costly default. The decision tree above is worth applying explicitly rather than defaulting to whichever pattern a team happens to be more familiar with: a nightly catalog re-scoring job run as a CronJob might occupy 2 GPUs for 45 minutes a night, while the same workload misconfigured as an always-on InferenceService with minReplicas: 1 would occupy that GPU capacity for the full 24 hours, a roughly 30x cost difference for identical actual compute work.

Storage for AI/ML: Model Weights and Datasets#

Part 3 covered CSI and PersistentVolumes generally — AI/ML workloads add two access patterns that stress that model differently: very large model weight files that need fast, repeated read access across many serving replicas, and very large training datasets that need high aggregate throughput across many concurrent workers.

Observability for GPU Workloads#

A GPU's own health and utilization are entirely invisible to the standard Kubernetes metrics pipeline (kubectl top, Part 10) — metrics-server only ever reports CPU and memory, never GPU utilization, temperature, or memory, which requires a GPU-specific exporter.

Model Versioning and Canary Rollouts for Inference#

Promoting a newly retrained model to production has the same "don't flip everyone over at once" risk profile as any application deployment (Part 2's rolling update mechanics) — KServe's InferenceService supports the same canary pattern natively, splitting traffic between model versions rather than requiring every request to hit whichever version was deployed most recently.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: recommendation-model
  namespace: recommendations
spec:
  predictor:
    canaryTrafficPercent: 10   # 10% of traffic to the new version below
    model:
      modelFormat: { name: sklearn }
      storageUri: "s3://ml-models/recommendation-model/v4/"
  # the previous revision (v3) continues serving the remaining 90%
  # until canaryTrafficPercent is raised or the rollout is rolled back

The genuinely ML-specific wrinkle here, beyond a standard application canary: what "healthy" means for a canary model version isn't just latency/error-rate (the standard application signals) — it's model quality, which typically requires comparing business/accuracy metrics (click-through rate, prediction accuracy against ground truth collected after the fact) that arrive on a much longer feedback loop than a standard HTTP canary's latency/error signals do. A model canary can look perfectly healthy on infrastructure metrics for days while quietly serving worse recommendations, which is why ML platforms typically pair infrastructure-level canary rollout with a separate, slower-feedback model-quality evaluation pipeline (often a Kubeflow Pipeline, from earlier in this chapter) rather than relying on infrastructure health checks alone to gate the rollout.

Cost Optimization for GPU Workloads#

GPU-hour cost is typically the single largest line item on an ML platform's cloud bill — several levers specific to this workload type go well beyond the general autoscaling cost guidance in Part 12.

# Checkpointing pattern: training periodically writes progress to
# durable storage, so a spot-instance reclamation loses at most
# one checkpoint interval of work, not the entire run
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: recommendation-model-training-spot
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 7
      template:
        spec:
          nodeSelector:
            node.kubernetes.io/lifecycle: spot   # scheduled onto spot GPU capacity
          containers:
            - name: pytorch
              image: registry.internal/recommendation-trainer:2.0.0
              args: ["--checkpoint-interval=300", "--checkpoint-path=s3://ml-checkpoints/rec-model/"]
              resources: { limits: { nvidia.com/gpu: 1 } }

Working through the batch-vs-realtime cost gap from earlier in this chapter with real numbers makes the scale of the mistake concrete. A single high-end GPU instance at a representative on-demand rate of roughly $3.50/hour, run as an always-on InferenceService with minReplicas: 1, costs approximately $2,520/month regardless of actual traffic. The same GPU, used only for a nightly 45-minute batch job via a CronJob, costs roughly $79/month for identical total compute work — the always-on pattern isn't a modest overhead here, it's over 30x more expensive for a workload that never needed to be continuously available in the first place.

Data Scientist Self-Service — Notebooks Within Tenant Guardrails#

Kubeflow Notebooks gives data scientists self-service, resource-quota-aware Jupyter environments — the same self-service-within-guardrails pattern Part 13 established generally, applied to the specific, GPU-shaped risk of a data scientist's ad-hoc experiment silently consuming an entire shared GPU pool.

apiVersion: kubeflow.org/v1
kind: Notebook
metadata:
  name: recs-experiment-jsmith
  namespace: recommendations
spec:
  template:
    spec:
      containers:
        - name: notebook
          image: kubeflownotebookswg/jupyter-pytorch-cuda:v1.9.0
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 16Gi

Because a Notebook is just another namespaced object, every Part 13 guardrail already applies to it automatically without any Notebook-specific configuration: the recommendations namespace's ResourceQuota (including requests.nvidia.com/gpu) caps how many notebook GPUs can run concurrently alongside production inference workloads, RBAC scopes who can create notebooks in that namespace at all, and a PriorityClass (Part 13) can deliberately rank ad-hoc notebook workloads below production serving, so a genuinely resource-constrained cluster preempts an idle-but-still-running experiment notebook before it ever preempts live user-facing inference traffic.

A Full Worked Scenario: Deploying an LLM Inference Service#

Bringing this chapter's pieces together for recommendation-model's move from a traditional sklearn model to an LLM-based recommendation re-ranker:

Every step above is a direct application of an earlier part or section in this series, not a new primitive — this is a deliberate closing illustration that AI/ML workloads don't require an entirely separate operational model from everything else in this series; they require the same primitives (scheduling, autoscaling, multi-tenancy, storage), applied with GPU-aware sizing and metrics instead of the generic CPU/memory assumptions those primitives default to. A platform team walking through this checklist for a genuinely new model deployment should expect each step to surface a real decision, not a rubber-stamp — the isolation tier in step 2, the metric choice in step 4, and the quota ceiling in step 5 all warrant the same deliberate sizing discipline this series has applied to every other resource type.

Part 14 CLI Cheat Sheet#

Quick Reference: Every Tool in This Chapter, What It Actually Does#

A single table worth keeping close, since this chapter introduced more distinct tools than most others in this series.

A Note on Reviewing GPU Configuration Changes#

Everything in this chapter — sharing strategy, autoscaling metric, quota sizing, checkpointing interval — is expensive enough per mistake that a second-reviewer pass on any GPU-workload configuration change is worth the friction it adds, more so than for an equivalent CPU/memory-only change.

This is the same second-reviewer discipline Part 11 applied to security-sensitive changes, applied here because a GPU misconfiguration's blast radius is measured in real dollars per hour, not just correctness — the cost of a five-minute review is negligible against the cost of even one of the mistakes this chapter's worked examples walked through.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team enables 4-way time-slicing on their GPU nodes to run more inference replicas per card. Shortly after, one replica starts crashing intermittently with a CUDA out-of-memory error, even though kubectl top pods shows the pod comfortably under its memory limit. What's the most likely explanation?

Answer: Time-slicing multiplies the count of schedulable GPU units without isolating or partitioning the GPU's actual physical memory — each of the 4 time-sliced replicas can attempt to allocate GPU memory independently, and if their combined demand exceeds the physical card's total memory, one of them will hit a CUDA-level out-of-memory error. This is invisible to kubectl top pods because that reports host CPU/memory usage, not GPU memory usage, and invisible to Kubernetes's own OOMKilled mechanism because Kubernetes never enforces or observes GPU memory limits under time-slicing at all.

Problem 2: An InferenceService scaled via a Resource-type HPA on CPU utilization shows healthy, low CPU usage even while users report the service is slow and its request queue is visibly growing. What's the fundamental mismatch in this setup?

Answer: For an LLM/GPU-bound serving workload, the actual bottleneck is GPU compute and the model's internal request queue, not CPU — the serving pod's CPU usage can stay low even under heavy GPU load, since CPU is mostly just handling HTTP request/response plumbing while the GPU does the real work. An HPA scaling on CPU utilization for this workload shape is scaling on a metric that structurally doesn't reflect real load; the fix is switching to a KEDA ScaledObject driven by a GPU/application-level metric like vLLM's vllm:num_requests_waiting.

Problem 3: A platform team sizes a recommendations namespace's requests.nvidia.com/gpu quota at 20, assuming Karpenter will simply provision more GPU nodes if the team needs more than the cluster currently has. During a demand spike, the team's pods sit Pending for over an hour. What assumption from the CPU/ memory autoscaling model (Part 7, Part 12) failed to hold here, and why?

Answer: The assumption that "unschedulable pods trigger new node provisioning, which resolves the capacity gap within normal boot-time latency" holds well for commodity CPU/memory capacity but not necessarily for GPUs, which are frequently supply-constrained at the cloud provider's own inventory level — Karpenter/Cluster Autoscaler can only provision a new GPU node if the cloud provider actually has GPU capacity of that instance type available in that region/zone at that moment, a constraint that essentially never applies to standard CPU instance types. Sizing a GPU quota assuming infinite on-demand elasticity, the same way a CPU/memory quota reasonably can, is the actual mistake — GPU capacity planning needs to account for real supply constraints, not just a policy ceiling.

Problem 4: A team runs a gang-scheduled 8-GPU training job entirely on spot instances with no checkpointing configured, to save cost. Over a week, the job never completes — it keeps restarting from scratch whenever any single worker's spot instance is reclaimed. What's the actual cost outcome, and what single change would most directly fix it?

Answer: Without checkpointing, every reclamation event forces the entire gang-scheduled job to discard all progress and restart, since gang scheduling requires all 8 workers to be present together — losing one worker effectively loses the whole run's progress up to that point. Over enough reclamation events, the job can consume more cumulative GPU-hours restarting repeatedly than a comparable on-demand run would have cost outright, completely erasing the intended savings. Adding checkpointing at an interval shorter than the spot instance type's typical time-between-reclamations is the single highest-leverage fix — it turns each reclamation into a bounded loss (one checkpoint interval) rather than a total loss of progress.

Summary and What's Next#

AI/ML workloads on Kubernetes don't need a separate operational model — they need the same scheduling, autoscaling, multi-tenancy, and storage primitives covered across this entire series, applied with GPU-aware metrics and sizing instead of the generic CPU/memory defaults those primitives assume. The device plugin framework makes GPUs schedulable at all; time-slicing, MIG, and MPS trade isolation for utilization in different ways; and LLM-specific serving via vLLM/KServe needs application-level metrics (queue depth, KV cache usage) that generic Resource-type autoscaling structurally cannot see.

The recurring theme worth carrying forward is that GPU capacity's economics — expensive, often supply-constrained, idling at real dollar cost — sharpen every general Kubernetes discipline covered earlier in this series rather than requiring a new one: scheduling becomes gang-aware, autoscaling needs application-level metrics instead of generic resource utilization, and multi-tenancy's noisy-neighbor and chargeback concerns matter proportionally more when the shared resource is a GPU instead of a CPU core.

Part 15, the final part of this series, turns to cluster lifecycle management — upgrading a running cluster safely, whether self-managed (building on Part 6's kubeadm bootstrapping) or managed (extending Part 5's provider-specific patterns), including the version-skew policy and rollback strategy that keeps an upgrade from becoming an outage.