# Kubernetes Deep Dive — Part 14: Running AI/ML Workloads on Kubernetes

> **Series:** Kubernetes Deep Dive (14 of 19)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Part 10:** `10-troubleshooting-kubernetes.md` — Troubleshooting Kubernetes, Systematically
> **Part 11:** `11-kubernetes-security-deep-dive.md` — Kubernetes Security Deep Dive (CKS-Aligned)
> **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling
> **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale
> **Part 14:** This file — Running AI/ML Workloads on Kubernetes
> **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management
> **Part 16:** `16-capacity-planning.md` — Capacity Planning for Kubernetes Clusters
> **Part 17:** `17-cilium-ebpf-networking.md` — Cilium & eBPF Networking Deep Dive
> **Part 18:** `18-backup-and-disaster-recovery.md` — Backup & Disaster Recovery for Workloads
> **Part 19:** `19-building-custom-controllers-and-operators.md` — Building Custom Controllers and Operators
> **Questions:** `questions.md`

Assumes you're comfortable with scheduling and resource requests from Part 2, autoscaling from Part 12, and
multi-tenancy from Part 13 — this chapter applies all three to the specific, GPU-shaped constraints AI/ML
workloads add on top of everything else in this series.

## Table of Contents

1. [Why This Part Exists](#why-this-part-exists)
2. [GPUs as a Schedulable Resource — Device Plugins](#gpus-as-a-schedulable-resource--device-plugins)
3. [GPU Sharing Strategies: Time-Slicing, MIG, and MPS](#gpu-sharing-strategies-time-slicing-mig-and-mps)
4. [Dynamic Resource Allocation — the Emerging Model](#dynamic-resource-allocation--the-emerging-model)
5. [Gang Scheduling for Distributed Training](#gang-scheduling-for-distributed-training)
6. [Kubeflow and the MLOps Lifecycle on Kubernetes](#kubeflow-and-the-mlops-lifecycle-on-kubernetes)
7. [KServe Deep Dive — the InferenceService Architecture](#kserve-deep-dive--the-inferenceservice-architecture)
8. [LLM-Specific Serving: vLLM, KV Cache, and Continuous Batching](#llm-specific-serving-vllm-kv-cache-and-continuous-batching)
9. [Autoscaling GPU and LLM Workloads](#autoscaling-gpu-and-llm-workloads)
10. [Multi-Tenancy for Shared GPU Clusters](#multi-tenancy-for-shared-gpu-clusters)
11. [Batch Inference vs. Real-Time Serving — Choosing the Right Pattern](#batch-inference-vs-real-time-serving--choosing-the-right-pattern)
12. [Storage for AI/ML: Model Weights and Datasets](#storage-for-aiml-model-weights-and-datasets)
13. [Observability for GPU Workloads](#observability-for-gpu-workloads)
14. [Model Versioning and Canary Rollouts for Inference](#model-versioning-and-canary-rollouts-for-inference)
15. [Cost Optimization for GPU Workloads](#cost-optimization-for-gpu-workloads)
16. [Data Scientist Self-Service — Notebooks Within Tenant Guardrails](#data-scientist-self-service--notebooks-within-tenant-guardrails)
17. [A Full Worked Scenario: Deploying an LLM Inference Service](#a-full-worked-scenario-deploying-an-llm-inference-service)
18. [Part 14 CLI Cheat Sheet](#part-14-cli-cheat-sheet)
19. [Quick Reference: Every Tool in This Chapter, What It Actually Does](#quick-reference-every-tool-in-this-chapter-what-it-actually-does)
20. [A Note on Reviewing GPU Configuration Changes](#a-note-on-reviewing-gpu-configuration-changes)
21. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
22. [Worked Practice Problems](#worked-practice-problems)
23. [Summary and What's Next](#summary-and-whats-next)

---

## Why This Part Exists

**Per the industry research behind this series' chapter list, AI/ML workloads are now the dominant driver
of new Kubernetes adoption — a fundamentally different capacity-planning problem from the CPU/memory-shaped
workloads every other part of this series has assumed.** A GPU is not just "a bigger CPU request" —
it's an indivisible-by-default, extremely expensive, often supply-constrained resource that idles at real
dollar cost the moment it isn't doing useful work, which reshapes scheduling, autoscaling, and multi-tenancy
decisions in ways this chapter covers on top of everything Parts 2, 12, and 13 already established.

The throughline system gains one more component here: a `recommendations` team's model-serving deployment,
`recommendation-model`, running inference on GPU nodes alongside the CPU-only `checkout`/`catalog` services
from earlier parts. This is also the same `recommendations` team introduced in Part 13's multi-tenancy
worked scenario — several of this chapter's worked examples build directly on the tenant boundary already
established there, rather than treating GPU capacity as an isolated, separately-governed resource.

## GPUs as a Schedulable Resource — Device Plugins

**Kubernetes has no native concept of a GPU — the Scheduler only understands CPU and memory as first-class
resource types out of the box. GPUs (and other accelerators) become schedulable entirely through the
Device Plugin framework, a well-defined extension point rather than a core Kubernetes feature.**

```mermaid
sequenceDiagram
    participant Kubelet
    participant Plugin as NVIDIA Device Plugin<br/>(DaemonSet on GPU nodes)
    participant Driver as NVIDIA driver +<br/>container toolkit
    participant Sched as Scheduler

    Plugin->>Driver: Discover physical GPUs<br/>on this node
    Plugin->>Kubelet: Register nvidia.com/gpu<br/>as an extended resource,<br/>advertise count
    Kubelet->>Sched: Node's Allocatable now<br/>includes nvidia.com/gpu: 8
    Note over Sched: A Pod requesting<br/>nvidia.com/gpu: 1 in its<br/>resources.limits is now<br/>schedulable like any<br/>other resource type
```

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: recommendation-model-trainer
  namespace: recommendations
spec:
  containers:
    - name: trainer
      image: registry.internal/recommendation-trainer:2.0.0
      resources:
        limits:
          nvidia.com/gpu: 1   # GPUs are ALWAYS specified in limits, never requests
```

> [!IMPORTANT]
> Extended resources like `nvidia.com/gpu` **must be specified in `limits` only** — Kubernetes doesn't
> support fractional or over-committed extended resources the way it does CPU/memory, so `requests` and
> `limits` are implicitly equal, and specifying only `requests` for a GPU is rejected by the API server
> entirely. This is a real, common first-time mistake for anyone coming from CPU/memory resource
> configuration, where separate `requests`/`limits` values are the norm. The practical consequence: there
> is no Burstable-QoS equivalent for GPUs (Part 2) — a pod either gets a whole GPU unit reserved for its
> exclusive use, or it doesn't get scheduled at all, absent one of the sharing strategies covered next.

**The GPU Operator (NVIDIA's, for the NVIDIA case specifically) automates the entire stack this depends
on** — the driver installation, the container toolkit, and the device plugin itself — as a set of
Kubernetes-native operators, rather than requiring each of those to be manually installed and version-matched
on every GPU node by hand, which was the norm before the Operator pattern (Part 4) was applied to this
specific, previously painful bootstrapping problem. A new GPU node joining the cluster is bootstrapped
automatically the same way any other Operator-managed dependency reconciles itself, rather than requiring a
manual per-node setup script every time the node pool scales.

## GPU Sharing Strategies: Time-Slicing, MIG, and MPS

**Without any sharing strategy, `nvidia.com/gpu: 1` means exactly one Pod occupies one entire physical GPU
for its whole lifetime — for many inference workloads that don't need a full GPU's throughput, this is
significant waste on an extremely expensive resource.**

```mermaid
flowchart TD
    Choice{"How much isolation<br/>does this workload need<br/>from its GPU neighbors?"}
    Choice -->|"None — best-effort<br/>sharing is fine"| TS["Time-Slicing:<br/>multiple pods share one<br/>GPU, interleaved in time,<br/>no memory isolation"]
    Choice -->|"Some — predictable<br/>memory/fault isolation"| MIG["MIG (Ampere+ only):<br/>hardware-partitions ONE<br/>physical GPU into several<br/>isolated mini-GPUs"]
    Choice -->|"Cooperating processes<br/>from the SAME team/job"| MPS["MPS: concurrent kernel<br/>execution from multiple<br/>processes, no hard<br/>isolation, higher<br/>throughput than time-slicing"]
```

| Strategy | Isolation | Best fit |
|---|---|---|
| Time-Slicing | None — no memory or fault isolation between sharers; a crashing workload can affect others on the same GPU | Low-priority, non-critical inference workloads where occasional interference is acceptable |
| MIG (Multi-Instance GPU) | Hardware-level memory and fault isolation, on Ampere-generation and later NVIDIA GPUs only | Multi-tenant scenarios (Part 13) needing genuine isolation between different teams' workloads sharing one physical card |
| MPS (Multi-Process Service) | None between untrusted parties, but higher throughput than time-slicing for cooperating processes | Multiple processes from the *same* trusted job/team that want to share a GPU efficiently, not different tenants |

```yaml
# GPU Operator ConfigMap enabling time-slicing — this specific GPU
# now advertises 4 schedulable slices instead of 1 whole device
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  a100-4-way: |
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
```

> [!WARNING]
> Time-slicing multiplies the *schedulable count* of a GPU resource without multiplying its actual memory
> or compute capacity — 4 pods each requesting `nvidia.com/gpu: 1` on a time-sliced 4-way GPU can each
> attempt to allocate the GPU's full memory, and the one that doesn't fit will crash with an out-of-memory
> error at the CUDA level, invisible to Kubernetes's own OOMKilled mechanism (Part 10) since it never
> touches host memory limits at all. Size time-sliced workloads' own memory footprint deliberately — this
> isn't enforced by Kubernetes the way CPU/memory limits are.

## Dynamic Resource Allocation — the Emerging Model

**Dynamic Resource Allocation (DRA) is a newer Kubernetes-native mechanism for expressing accelerator
requirements far more richly than the Device Plugin framework's simple integer-count model allows —
`ResourceClaim` objects let a Pod ask for a GPU matching specific attributes (a minimum memory size, a
specific MIG profile, a NVLink topology requirement) rather than just "one unit of `nvidia.com/gpu`."**

```yaml
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaim
metadata:
  name: training-gpu-claim
  namespace: recommendations
spec:
  devices:
    requests:
      - name: gpu
        deviceClassName: gpu.nvidia.com
        selectors:
          - cel:
              expression: 'device.attributes["gpu.nvidia.com"].memory >= 40000'
```

**DRA moves resource-matching logic out of the Scheduler's built-in, hardcoded extended-resource counting
and into a structured, extensible claims model that a vendor's own driver can populate with rich device
attributes** — the same underlying idea as CSI did for storage (Part 3) and CNI did for networking (Part 3),
applied to accelerators. This is a genuinely active area of the project: confirm DRA's current graduation
stage (alpha/beta/stable) and API version against your specific cluster's Kubernetes release before relying
on it for production scheduling decisions, since a fast-moving beta feature's exact CRD shape and default
enablement can change between minor versions in ways the Device Plugin framework's long-stable API does not.

| | Device Plugin framework | Dynamic Resource Allocation |
|---|---|---|
| Resource model | A flat integer count per extended resource name | Rich, structured claims with attribute-based selection |
| Sharing expression | Configured out-of-band (time-slicing ConfigMaps, MIG profiles) | Expressible directly in the claim itself |
| Maturity | Long-stable, the current production default across essentially every cluster running GPUs today | Actively evolving — verify current stage before depending on it |

## Gang Scheduling for Distributed Training

**A distributed training job spanning 8 GPU pods across multiple nodes has a requirement the default
Scheduler (Part 2) doesn't natively satisfy: all 8 pods must be scheduled together, or the job should wait
entirely — starting 5 of 8 workers and leaving 3 `Pending` wastes the 5 already-running GPUs on a job that
can't make progress without every worker present.**

```mermaid
flowchart TD
    Job["Distributed training job:<br/>needs exactly 8 GPU workers"] --> Default["Default Scheduler:<br/>schedules pods one at a time,<br/>independently"]
    Default --> Partial["5 of 8 scheduled,<br/>3 stuck Pending<br/>(insufficient GPU capacity)"]
    Partial --> Waste["5 GPUs sit idle running<br/>a job that can't start<br/>real training work without<br/>all 8 workers present"]

    Job --> Gang["Gang scheduler<br/>(Volcano, Kueue):<br/>all-or-nothing"]
    Gang --> AllOrNothing{"Can all 8 be<br/>scheduled together<br/>right now?"}
    AllOrNothing -->|"No"| Queue["Entire job queues,<br/>ZERO GPUs consumed<br/>until capacity exists"]
    AllOrNothing -->|"Yes"| AllStart["All 8 start<br/>simultaneously"]
```

| Tool | Approach |
|---|---|
| Volcano | A dedicated batch-scheduling system for Kubernetes, replacing the default Scheduler for jobs that opt in, with native gang-scheduling and job-queueing semantics |
| Kueue | A newer, Kubernetes-native job-queueing layer sitting *alongside* the default Scheduler (not replacing it), managing admission and gang-scheduling for batch/ML workloads specifically |
| Kubeflow Training Operator | Provides the `PyTorchJob`/`TFJob` CRDs (the actual training job abstraction most teams interact with) and can integrate with either Volcano or Kueue underneath for the gang-scheduling guarantee itself |

```yaml
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: recommendation-model-training
  namespace: recommendations
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: registry.internal/recommendation-trainer:2.0.0
              resources: { limits: { nvidia.com/gpu: 1 } }
    Worker:
      replicas: 7
      template:
        spec:
          containers:
            - name: pytorch
              image: registry.internal/recommendation-trainer:2.0.0
              resources: { limits: { nvidia.com/gpu: 1 } }
```

## Kubeflow and the MLOps Lifecycle on Kubernetes

**Kubeflow is a collection of Kubernetes-native components covering the full ML lifecycle, not one single
tool — understanding which component solves which stage avoids reaching for the wrong piece.**

```mermaid
mindmap
  root((Kubeflow))
    Pipelines
      Orchestrates multi-step<br/>ML workflows as DAGs
    Training Operator
      PyTorchJob, TFJob CRDs<br/>(previous section)
    Katib
      Hyperparameter tuning<br/>via automated experiments
    KServe
      Model serving<br/>(next section)
    Notebooks
      Managed Jupyter<br/>environments per user
```

| Component | Solves |
|---|---|
| Pipelines | Chaining data prep → training → evaluation → deployment as a reproducible, versioned DAG |
| Training Operator | Running the actual distributed training job (previous section) |
| Katib | Automated hyperparameter search across many training runs, tracking which configuration performs best |
| KServe | Serving a trained model as a production inference endpoint |
| Notebooks | Self-service, resource-quota-aware (Part 13) Jupyter environments for exploratory work |

**Katib deserves one concrete example, since "automated hyperparameter search" is easy to gloss over as an
abstraction**: a `Katib Experiment` defines a search space (e.g. learning rate between 0.0001 and 0.1) and
an objective metric to optimize, then launches many `PyTorchJob` trial runs (the previous section's CRD)
automatically, each with a different hyperparameter combination, converging toward the best-performing
configuration without a human manually launching and comparing each run by hand — the same
"encode the operational pattern once as a controller, not as a runbook a human follows" principle from
Part 4's Operator pattern, applied to the ML experimentation loop specifically.

**The 2026 framing worth internalizing**: per the earlier research, the industry-wide challenge has shifted
from "can we train a model" to "can we operate it reliably at scale" — Kubeflow's components map
directly onto that shift, with Pipelines and Katib addressing the training side and KServe addressing the
operational, always-on serving side that increasingly dominates real infrastructure cost and complexity.

## KServe Deep Dive — the InferenceService Architecture

**KServe's `InferenceService` custom resource abstracts model serving behind a consistent interface across
frameworks (TensorFlow, PyTorch, scikit-learn, and LLM-specific runtimes), building on Knative Serving for
request-driven autoscaling including scale-to-zero.**

```mermaid
flowchart TD
    Client["Client request"] --> Istio["Istio ingress<br/>(traffic routing,<br/>canary rollout)"]
    Istio --> Knative["Knative Serving:<br/>request-driven autoscaling,<br/>scale-to-zero when idle"]
    Knative --> Predictor["Predictor pod:<br/>the actual model server"]
    Predictor -->|"cold start,<br/>if scaled to zero"| Load["Load model weights<br/>from storage"]
```

```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: recommendation-model
  namespace: recommendations
spec:
  predictor:
    model:
      modelFormat: { name: sklearn }
      storageUri: "s3://ml-models/recommendation-model/v3/"
      resources:
        limits: { cpu: "2", memory: 4Gi }
    minReplicas: 1
    maxReplicas: 10
```

**`InferenceService` deliberately hides the Knative/Istio machinery underneath a single, framework-agnostic
spec** — a data scientist promoting a newly trained model to production interacts with `storageUri` and
`modelFormat`, not with Knative Revisions or Istio VirtualServices directly, the same abstraction-hiding
principle the Operator pattern (Part 4) applies generally: encode the operational complexity once, expose a
simple domain-specific interface to everyone else.

> [!NOTE]
> Since v0.16, KServe also ships a dedicated `LLMInferenceService` CRD, distinct from the general
> `InferenceService` above, specifically for LLM-shaped serving concerns (the next section's KV cache and
> continuous batching considerations don't map cleanly onto the general predictive-model interface). Confirm
> which CRD your KServe version and workload type actually calls for before assuming one interface covers
> both cases — a traditional `sklearn`/`xgboost` predictive model and a multi-billion-parameter LLM have
> different enough serving characteristics that treating them identically is a common source of
> underprovisioned or misconfigured deployments.

## LLM-Specific Serving: vLLM, KV Cache, and Continuous Batching

**Serving an LLM has a resource-usage shape fundamentally different from a traditional stateless HTTP
service, and generic HPA/CPU-based thinking (Part 12) doesn't transfer cleanly — vLLM (the dominant
open-source LLM inference engine) exposes the metrics that actually matter for this workload type.**

| vLLM metric | What it reveals | Why CPU/memory can't substitute for it |
|---|---|---|
| `vllm:num_requests_waiting` | Inference queue depth — how many requests are waiting for GPU capacity right now | A request queue can build up while CPU usage on the serving pod stays flat, since the bottleneck is entirely GPU-side |
| `vllm:gpu_cache_usage_perc` | How full the KV cache (the per-request memory holding an LLM's attention context) is | GPU memory pressure specifically from serving concurrent long-context requests, invisible to host-level memory metrics |
| `vllm:time_to_first_token_seconds` | User-perceived latency to the first generated token | The actual SLO metric users experience, several layers removed from any infrastructure-level resource metric |

```mermaid
flowchart LR
    Req1["Request A"] --> Batch["Continuous batching:<br/>new requests join an<br/>in-flight batch as soon<br/>as GPU capacity frees up,<br/>token by token"]
    Req2["Request B"] --> Batch
    Req3["Request C<br/>(arrives later)"] --> Batch
    Batch --> GPU["GPU processes the<br/>whole batch together,<br/>far higher throughput<br/>than one-request-at-a-time"]
```

**Continuous batching is the single biggest throughput lever specific to LLM serving** — unlike a
traditional request-response service where each request is processed independently, vLLM dynamically
interleaves multiple requests' token-generation steps into shared GPU batches, letting a new request join
an already-running batch rather than waiting for the current batch to fully complete, which is what makes
GPU utilization for LLM serving meaningfully higher than naive one-request-per-GPU-call serving would
achieve.

## Autoscaling GPU and LLM Workloads

**A naive requests-per-second HPA metric (Part 12) has no way to express the batching-capacity distinction
from the previous section — two requests generating very different output lengths consume very different
amounts of GPU capacity for the same "one request" unit, which is exactly why the metric choice below
matters more here than for a typical stateless HTTP service.**

**Part 12 covered HPA/VPA/KEDA in general — GPU and LLM workloads specifically need KEDA's external-metric
capability, since the signal that actually matters (queue depth, KV cache fullness) lives in the
application/GPU layer, invisible to a Resource-type HPA metric entirely.**

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: recommendation-model-vllm
  namespace: recommendations
spec:
  scaleTargetRef: { name: recommendation-model-predictor }
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.observability.svc:9090
        query: vllm:num_requests_waiting
        threshold: "10"
```

**This is a direct application of Part 12's Prometheus-adapter/KEDA pattern, with one GPU-specific wrinkle
worth calling out**: a scale-up decision here doesn't just create a new pod — it likely triggers a new GPU
node provisioning event (Part 7's Karpenter, extended to GPU node pools) if no existing node has spare GPU
capacity, and GPU node boot time (including driver/toolkit initialization via the GPU Operator) is
typically markedly slower than a standard CPU node's boot time, extending exactly the scale-up latency
window Part 12's worked scenario walked through — worth budgeting for explicitly in any LLM-serving SLO.

> [!TIP]
> Pre-pulled container images and cached model weights on GPU nodes (rather than downloading a large model's
> weights fresh on every cold start) meaningfully cut the scale-up latency this section just described —
> this is one of the specific "bare-metal control" concerns the research behind this chapter flagged as a
> genuine gap in serverless-GPU offerings that don't expose that level of node-image control.

## Multi-Tenancy for Shared GPU Clusters

**Part 13's multi-tenancy models apply directly to GPU capacity, with the economics sharpened by how
expensive an idle GPU actually is compared to an idle CPU core.**

| Part 13 concept | GPU-specific application |
|---|---|
| ResourceQuota | `requests.nvidia.com/gpu` as an explicit quota line per tenant — without it, one team can consume every GPU in the cluster with no ceiling at all |
| PriorityClass + taints/tolerations | Dedicated GPU node pools per tenant tier, since GPU capacity is usually far scarcer and more expensive than general compute, making noisy-neighbor contention on it a bigger business risk |
| Cost chargeback | GPU-hour cost is typically the dominant line item on an ML platform's cloud bill by a wide margin — chargeback accuracy matters proportionally more here than for general CPU/memory workloads |
| MIG (this chapter) | A genuine hardware-isolation option specific to GPU multi-tenancy, with no equivalent for CPU/memory sharing |

> [!CAUTION]
> A `ResourceQuota` capping `requests.nvidia.com/gpu` does nothing to prevent one tenant from requesting
> every available GPU node's capacity the moment it's provisioned, if quota sizing wasn't coordinated with
> actual GPU node pool capacity — unlike CPU/memory, where the cluster can often absorb an oversized request
> by simply provisioning more of a commodity, cheap resource, GPU capacity is frequently supply-constrained
> at the cloud-provider level itself, meaning "just add more nodes" isn't always an available response to a
> quota being sized too generously relative to real availability.

## Batch Inference vs. Real-Time Serving — Choosing the Right Pattern

**Not every inference workload should be a long-running `InferenceService` — a meaningful fraction of real
ML workloads are batch: scoring a large dataset overnight, not answering individual live requests, and
forcing that shape into an always-on serving pattern wastes GPU cost on idle time between batch runs.**

```mermaid
flowchart TD
    Question{"Does this need a<br/>response within seconds,<br/>to an unpredictable<br/>arrival pattern?"}
    Question -->|"Yes"| Realtime["Real-time serving:<br/>KServe InferenceService<br/>(this chapter)"]
    Question -->|"No — a known dataset,<br/>processed on a schedule<br/>or trigger"| Batch["Batch inference:<br/>a Kubernetes Job or<br/>CronJob (Part 2), or a<br/>KEDA ScaledJob (Part 12)"]
```

| | Real-time serving (KServe) | Batch inference (Job/CronJob) |
|---|---|---|
| GPU occupied | Continuously, or scaled per live request volume | Only for the duration of the batch run |
| Latency expectation | Seconds or sub-second, per request | Minutes to hours for the whole batch, per-item latency irrelevant |
| Cost profile | Pays for standby capacity even between requests, unless scaled to zero | Pays only for actual processing time |
| Typical use | User-facing recommendations, live fraud scoring, chat interfaces | Nightly re-scoring of an entire catalog, bulk embedding generation, offline model evaluation |

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-catalog-rescore
  namespace: recommendations
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: batch-scorer
              image: registry.internal/recommendation-batch-scorer:1.0.0
              resources: { limits: { nvidia.com/gpu: 2 } }
          restartPolicy: Never
```

**Getting this choice wrong in the expensive direction — running a genuinely batch workload as an
always-on `InferenceService` "because that's the pattern the team already knows" — is a common and costly
default.** The decision tree above is worth applying explicitly rather than defaulting to whichever pattern
a team happens to be more familiar with: a nightly catalog re-scoring job run as a CronJob might occupy 2
GPUs for 45 minutes a night, while the same workload misconfigured as an always-on `InferenceService` with
`minReplicas: 1` would occupy that GPU capacity for the full 24 hours, a roughly 30x cost difference for
identical actual compute work.

## Storage for AI/ML: Model Weights and Datasets

**Part 3 covered CSI and PersistentVolumes generally — AI/ML workloads add two access patterns that
stress that model differently: very large model weight files that need fast, repeated read access across
many serving replicas, and very large training datasets that need high aggregate throughput across many
concurrent workers.**

| Need | Typical approach |
|---|---|
| Model weights shared read-only across many inference replicas | An object store (S3/GCS) as the source of truth, with `ReadOnlyMany` PVCs or a node-local cache layer to avoid re-downloading large weights on every pod start |
| High-throughput distributed training data | A parallel filesystem CSI driver (e.g. Lustre/FSx for Lustre, or a cloud-native equivalent) sized for aggregate throughput across all training workers simultaneously, not just single-client IOPS |
| Very large individual model files (tens to hundreds of GB) | Consider whether a full re-download on every cold-start pod is acceptable, or whether a node-local cache/pre-pulled volume is needed to keep scale-up latency (previous section) within SLO |

## Observability for GPU Workloads

**A GPU's own health and utilization are entirely invisible to the standard Kubernetes metrics pipeline
(`kubectl top`, Part 10) — `metrics-server` only ever reports CPU and memory, never GPU utilization,
temperature, or memory, which requires a GPU-specific exporter.**

```mermaid
flowchart LR
    GPU["Physical GPU"] --> DCGM["NVIDIA DCGM Exporter<br/>(DaemonSet on GPU nodes)"] --> Prom["Prometheus"] --> Grafana["GPU utilization/<br/>temperature/memory<br/>dashboards"]
```

| Metric | What it reveals |
|---|---|
| `DCGM_FI_DEV_GPU_UTIL` | Real compute utilization — the single most important sanity check that expensive GPU capacity isn't sitting idle |
| `DCGM_FI_DEV_FB_USED` | GPU memory (framebuffer) actually in use — catches the time-slicing memory-pressure risk flagged earlier in this chapter before it becomes a crash |
| `DCGM_FI_DEV_GPU_TEMP` | Thermal health — sustained high temperature under load is an early signal of a cooling or node-placement problem, not just an inference workload concern |
| `DCGM_FI_DEV_XID_ERRORS` | Hardware-level GPU errors (Xid codes) — the GPU equivalent of a kernel panic, often the first signal of failing hardware before a node goes fully `NotReady` |

> [!TIP]
> `DCGM_FI_DEV_GPU_UTIL` sitting persistently low across a fleet of GPU nodes is one of the highest-value
> FinOps signals a platform team can alert on — given GPU cost dominance (covered later in this chapter),
> a fleet of GPUs averaging 20% utilization is a direct, quantifiable cost-optimization opportunity
> (consolidate onto fewer nodes, adopt time-slicing/MIG, or right-size instance types) in a way that's much
> harder to see from cost dashboards alone, which show spend but not the utilization behind it.

## Model Versioning and Canary Rollouts for Inference

**Promoting a newly retrained model to production has the same "don't flip everyone over at once" risk
profile as any application deployment (Part 2's rolling update mechanics) — KServe's `InferenceService`
supports the same canary pattern natively, splitting traffic between model versions rather than requiring
every request to hit whichever version was deployed most recently.**

```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: recommendation-model
  namespace: recommendations
spec:
  predictor:
    canaryTrafficPercent: 10   # 10% of traffic to the new version below
    model:
      modelFormat: { name: sklearn }
      storageUri: "s3://ml-models/recommendation-model/v4/"
  # the previous revision (v3) continues serving the remaining 90%
  # until canaryTrafficPercent is raised or the rollout is rolled back
```

```mermaid
flowchart LR
    Traffic["Incoming requests"] --> Split{"canaryTrafficPercent: 10"}
    Split -->|"90%"| V3["v3 (stable,<br/>already validated)"]
    Split -->|"10%"| V4["v4 (canary,<br/>newly trained)"]
    V4 --> Monitor["Compare v4's real-world<br/>accuracy/latency against v3<br/>before raising traffic %"]
```

**The genuinely ML-specific wrinkle here, beyond a standard application canary**: what "healthy" means for
a canary model version isn't just latency/error-rate (the standard application signals) — it's model
*quality*, which typically requires comparing business/accuracy metrics (click-through rate, prediction
accuracy against ground truth collected after the fact) that arrive on a much longer feedback loop than a
standard HTTP canary's latency/error signals do. A model canary can look perfectly healthy on
infrastructure metrics for days while quietly serving worse recommendations, which is why ML platforms
typically pair infrastructure-level canary rollout with a separate, slower-feedback model-quality
evaluation pipeline (often a Kubeflow Pipeline, from earlier in this chapter) rather than relying on
infrastructure health checks alone to gate the rollout.

## Cost Optimization for GPU Workloads

**GPU-hour cost is typically the single largest line item on an ML platform's cloud bill — several
levers specific to this workload type go well beyond the general autoscaling cost guidance in Part 12.**

| Lever | Mechanism | Tradeoff |
|---|---|---|
| Spot/preemptible GPU instances | Significantly cheaper than on-demand, in exchange for the cloud provider being able to reclaim the instance with short notice | Requires checkpointing (below) to avoid losing significant training progress on reclamation |
| Right-sizing GPU type per workload | A large model requiring 80GB of GPU memory genuinely needs a high-end card; a small model doesn't — matching instance type to actual model size avoids paying for unused headroom | Requires knowing the model's real memory footprint, not just defaulting to "the biggest GPU available" |
| Time-slicing/MIG for inference (this chapter) | Multiple smaller inference workloads share one physical GPU instead of each reserving a whole card | Isolation tradeoffs already covered earlier in this chapter |
| Scale-to-zero for bursty/non-production inference (Part 12's KEDA) | No GPU cost at all during genuinely idle periods | Cold-start latency, worse for workloads needing to be always-instantly-available |

```yaml
# Checkpointing pattern: training periodically writes progress to
# durable storage, so a spot-instance reclamation loses at most
# one checkpoint interval of work, not the entire run
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: recommendation-model-training-spot
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 7
      template:
        spec:
          nodeSelector:
            node.kubernetes.io/lifecycle: spot   # scheduled onto spot GPU capacity
          containers:
            - name: pytorch
              image: registry.internal/recommendation-trainer:2.0.0
              args: ["--checkpoint-interval=300", "--checkpoint-path=s3://ml-checkpoints/rec-model/"]
              resources: { limits: { nvidia.com/gpu: 1 } }
```

**Working through the batch-vs-realtime cost gap from earlier in this chapter with real numbers makes the
scale of the mistake concrete.** A single high-end GPU instance at a representative on-demand rate of
roughly $3.50/hour, run as an always-on `InferenceService` with `minReplicas: 1`, costs approximately
$2,520/month regardless of actual traffic. The same GPU, used only for a nightly 45-minute batch job via a
CronJob, costs roughly $79/month for identical total compute work — the always-on pattern isn't a modest
overhead here, it's over 30x more expensive for a workload that never needed to be continuously available
in the first place.

> [!WARNING]
> Deploying a distributed, gang-scheduled training job (this chapter's earlier section) onto spot GPU
> capacity without checkpointing means a single worker's reclamation can force the *entire* gang-scheduled
> job to restart from scratch, not just that one worker — the cost savings from spot pricing can be
> completely erased by repeatedly losing hours of multi-GPU training progress. Checkpointing frequency
> should be tuned against both the spot instance type's typical reclamation notice period and the real cost
> of re-running lost work, not left at a framework default chosen without either number in mind.

## Data Scientist Self-Service — Notebooks Within Tenant Guardrails

**Kubeflow Notebooks gives data scientists self-service, resource-quota-aware Jupyter environments — the
same self-service-within-guardrails pattern Part 13 established generally, applied to the specific,
GPU-shaped risk of a data scientist's ad-hoc experiment silently consuming an entire shared GPU pool.**

```yaml
apiVersion: kubeflow.org/v1
kind: Notebook
metadata:
  name: recs-experiment-jsmith
  namespace: recommendations
spec:
  template:
    spec:
      containers:
        - name: notebook
          image: kubeflownotebookswg/jupyter-pytorch-cuda:v1.9.0
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 16Gi
```

**Because a `Notebook` is just another namespaced object, every Part 13 guardrail already applies to it
automatically without any Notebook-specific configuration**: the `recommendations` namespace's
`ResourceQuota` (including `requests.nvidia.com/gpu`) caps how many notebook GPUs can run concurrently
alongside production inference workloads, RBAC scopes who can create notebooks in that namespace at all,
and a `PriorityClass` (Part 13) can deliberately rank ad-hoc notebook workloads *below* production serving,
so a genuinely resource-constrained cluster preempts an idle-but-still-running experiment notebook before
it ever preempts live user-facing inference traffic.

> [!TIP]
> Setting an aggressive `culling` policy (Kubeflow Notebooks supports automatically stopping an idle
> notebook after a configurable period of no kernel activity) is one of the highest-value, lowest-effort
> GPU cost controls on a shared ML platform — a data scientist stepping away from their desk with a GPU
> notebook still attached is a surprisingly common source of the low-utilization pattern the DCGM
> `DCGM_FI_DEV_GPU_UTIL` alert from earlier in this chapter is specifically designed to catch.

## A Full Worked Scenario: Deploying an LLM Inference Service

**Bringing this chapter's pieces together for `recommendation-model`'s move from a traditional sklearn model
to an LLM-based recommendation re-ranker:**

```mermaid
flowchart TD
    S1["1. GPU node pool provisioned<br/>with GPU Operator installed<br/>(driver, toolkit, device plugin)"] --> S2["2. MIG enabled for<br/>hardware isolation —<br/>this is a shared,<br/>multi-tenant cluster (Part 13)"]
    S2 --> S3["3. LLMInferenceService<br/>deployed via KServe,<br/>backed by vLLM"]
    S3 --> S4["4. KEDA ScaledObject<br/>scaling on vllm:num_requests_waiting,<br/>not CPU"]
    S4 --> S5["5. ResourceQuota for<br/>recommendations namespace<br/>caps requests.nvidia.com/gpu"]
    S5 --> S6["6. Cost chargeback<br/>attributes GPU-hours<br/>to the recommendations<br/>team's budget"]
```

**Every step above is a direct application of an earlier part or section in this series, not a new
primitive** — this is a deliberate closing illustration that AI/ML workloads don't require an entirely
separate operational model from everything else in this series; they require the same primitives
(scheduling, autoscaling, multi-tenancy, storage), applied with GPU-aware sizing and metrics instead of the
generic CPU/memory assumptions those primitives default to. A platform team walking through this checklist
for a genuinely new model deployment should expect each step to surface a real decision, not a rubber-stamp
— the isolation tier in step 2, the metric choice in step 4, and the quota ceiling in step 5 all warrant the
same deliberate sizing discipline this series has applied to every other resource type.

## Part 14 CLI Cheat Sheet

| Command | Purpose |
|---|---|
| `kubectl describe node <gpu-node> \| grep -A5 Allocatable` | Confirm `nvidia.com/gpu` is actually advertised as schedulable on a node |
| `kubectl get pods -n kube-system -l app=nvidia-device-plugin-daemonset` | Confirm the device plugin itself is running and healthy |
| `kubectl logs -n gpu-operator -l app=nvidia-dcgm-exporter` | Check the GPU metrics exporter feeding Prometheus |
| `kubectl get inferenceservice -n <ns>` | KServe's own resources, independent of the Knative/Istio machinery underneath |
| `kubectl get pytorchjob -n <ns>` / `kubectl get tfjob -n <ns>` | Kubeflow Training Operator's distributed training job status |
| `kubectl get scaledobject -n <ns> -o yaml \| grep query` | Confirm which vLLM/Prometheus metric a KEDA `ScaledObject` is actually scaling on |
| `kubectl exec -it <pod> -- nvidia-smi` | Ground-truth GPU utilization/memory from inside a running pod, independent of any exporter pipeline |

## Quick Reference: Every Tool in This Chapter, What It Actually Does

**A single table worth keeping close, since this chapter introduced more distinct tools than most others
in this series.**

| Tool | Layer | What it actually does |
|---|---|---|
| NVIDIA GPU Operator / Device Plugin | Node | Makes GPUs schedulable as `nvidia.com/gpu`, automates driver/toolkit installation |
| DRA (`ResourceClaim`) | Node/Scheduler | Emerging, richer alternative to the flat device-plugin count model |
| DCGM Exporter | Observability | Exposes real GPU utilization/memory/temperature to Prometheus |
| Volcano / Kueue | Scheduler | Gang-scheduling and batch job queueing for distributed training |
| Kubeflow Training Operator | Workload | `PyTorchJob`/`TFJob` CRDs — the actual distributed training job abstraction |
| Kubeflow Pipelines / Katib | MLOps | Orchestrating multi-step workflows / automated hyperparameter search |
| Kubeflow Notebooks | Self-service | Namespaced, quota-aware Jupyter environments for data scientists |
| KServe | Serving | `InferenceService`/`LLMInferenceService` — the model-serving abstraction, built on Knative + Istio |
| vLLM | Serving (LLM-specific) | The inference engine actually running the model, exposing queue-depth/KV-cache metrics |
| KEDA (Part 12) | Autoscaling | Scales serving replicas on vLLM's own metrics, not generic CPU |

## A Note on Reviewing GPU Configuration Changes

**Everything in this chapter — sharing strategy, autoscaling metric, quota sizing, checkpointing interval
— is expensive enough per mistake that a second-reviewer pass on any GPU-workload configuration change is
worth the friction it adds, more so than for an equivalent CPU/memory-only change.**

| Change | What a reviewer should specifically check |
|---|---|
| A new time-sliced GPU node pool | Does any workload assume full-card memory, given the memory-isolation gap this chapter covered? |
| A new KEDA `ScaledObject` targeting a GPU workload | Is the trigger a real GPU-aware metric, not a leftover CPU-based default? |
| A new `ScaledJob` for batch inference | Is `maxReplicaCount` sized against the downstream dependency's real concurrency limit, not just GPU node capacity? |
| Enabling spot capacity for a training job | Is checkpointing actually configured, not just planned for "later"? |
| A new `InferenceService` replacing a batch Job | Was the batch-vs-realtime decision (earlier in this chapter) made deliberately, or defaulted to whichever pattern the team already knew? |
| A namespace's `requests.nvidia.com/gpu` quota increase | Was it checked against confirmed, real node-pool GPU availability, or just approved as a policy number? |

This is the same second-reviewer discipline Part 11 applied to security-sensitive changes, applied here
because a GPU misconfiguration's blast radius is measured in real dollars per hour, not just correctness —
the cost of a five-minute review is negligible against the cost of even one of the mistakes this chapter's
worked examples walked through.

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Setting `requests.nvidia.com/gpu` without `limits`, or vice versa with different values | Extended resources like GPUs don't support requests≠limits — the API server rejects a mismatch | Always set GPU count in `limits`; Kubernetes implicitly treats it as the request too |
| Treating time-sliced GPU replicas as memory-isolated | Time-slicing shares physical GPU memory with zero enforcement — a CUDA-level OOM is invisible to Kubernetes | Use MIG when genuine memory isolation between sharers is required |
| Autoscaling an LLM-serving Deployment on CPU utilization | The GPU is the actual bottleneck; CPU usage on the serving pod can stay flat while the inference queue backs up badly | Scale on vLLM's own queue-depth/KV-cache metrics via KEDA, per this chapter |
| Assuming DRA is universally available and stable across any recent cluster | It's an actively evolving feature — exact API version and graduation stage vary by Kubernetes release | Confirm DRA's current stage on your specific cluster version before depending on it for production scheduling |
| Sizing a GPU `ResourceQuota` without checking real node-pool GPU availability | Unlike CPU/memory, GPU capacity is often supply-constrained — "just provision more nodes" may not be available | Coordinate quota sizing with actual, confirmed GPU node pool capacity, not just a policy number |
| Ignoring model weight download time in cold-start/scale-up latency budgets | A multi-gigabyte model re-downloaded on every fresh pod start can dominate total scale-up time, far more than typical pod-image pull time | Cache weights node-locally or pre-pull them; budget scale-up SLOs around real cold-start measurements |
| Running a nightly batch-scoring job as an always-on `InferenceService` | Pays for 24 hours of GPU standby to do 45 minutes of real work | Use a Job/CronJob (or KEDA `ScaledJob`, Part 12) for genuinely batch, non-live-request workloads |
| Running gang-scheduled distributed training on spot capacity with no checkpointing | A single worker's reclamation forces the entire job to restart from zero, potentially costing more in wasted GPU-hours than on-demand pricing would have | Always pair spot GPU training with checkpointing at an interval shorter than typical reclamation notice |
| Leaving data-scientist notebooks with GPUs attached and no idle-culling policy | An idle, forgotten notebook occupies real GPU cost indefinitely with zero production value | Configure Kubeflow Notebooks' culling policy to auto-stop on extended kernel inactivity |

## Worked Practice Problems

**Problem 1**: A team enables 4-way time-slicing on their GPU nodes to run more inference replicas per
card. Shortly after, one replica starts crashing intermittently with a CUDA out-of-memory error, even
though `kubectl top pods` shows the pod comfortably under its memory limit. What's the most likely
explanation?

*Answer*: Time-slicing multiplies the *count* of schedulable GPU units without isolating or partitioning
the GPU's actual physical memory — each of the 4 time-sliced replicas can attempt to allocate GPU memory
independently, and if their combined demand exceeds the physical card's total memory, one of them will hit
a CUDA-level out-of-memory error. This is invisible to `kubectl top pods` because that reports host
CPU/memory usage, not GPU memory usage, and invisible to Kubernetes's own OOMKilled mechanism because
Kubernetes never enforces or observes GPU memory limits under time-slicing at all.

**Problem 2**: An `InferenceService` scaled via a Resource-type HPA on CPU utilization shows healthy,
low CPU usage even while users report the service is slow and its request queue is visibly growing. What's
the fundamental mismatch in this setup?

*Answer*: For an LLM/GPU-bound serving workload, the actual bottleneck is GPU compute and the model's
internal request queue, not CPU — the serving pod's CPU usage can stay low even under heavy GPU load, since
CPU is mostly just handling HTTP request/response plumbing while the GPU does the real work. An HPA scaling
on CPU utilization for this workload shape is scaling on a metric that structurally doesn't reflect real
load; the fix is switching to a KEDA `ScaledObject` driven by a GPU/application-level metric like vLLM's
`vllm:num_requests_waiting`.

**Problem 3**: A platform team sizes a `recommendations` namespace's `requests.nvidia.com/gpu` quota at 20,
assuming Karpenter will simply provision more GPU nodes if the team needs more than the cluster currently
has. During a demand spike, the team's pods sit `Pending` for over an hour. What assumption from the CPU/
memory autoscaling model (Part 7, Part 12) failed to hold here, and why?

*Answer*: The assumption that "unschedulable pods trigger new node provisioning, which resolves the
capacity gap within normal boot-time latency" holds well for commodity CPU/memory capacity but not
necessarily for GPUs, which are frequently supply-constrained at the cloud provider's own inventory level —
Karpenter/Cluster Autoscaler can only provision a new GPU node if the cloud provider actually has GPU
capacity of that instance type available in that region/zone at that moment, a constraint that essentially
never applies to standard CPU instance types. Sizing a GPU quota assuming infinite on-demand elasticity,
the same way a CPU/memory quota reasonably can, is the actual mistake — GPU capacity planning needs to
account for real supply constraints, not just a policy ceiling.

**Problem 4**: A team runs a gang-scheduled 8-GPU training job entirely on spot instances with no
checkpointing configured, to save cost. Over a week, the job never completes — it keeps restarting from
scratch whenever any single worker's spot instance is reclaimed. What's the actual cost outcome, and what
single change would most directly fix it?

*Answer*: Without checkpointing, every reclamation event forces the entire gang-scheduled job to discard
all progress and restart, since gang scheduling requires all 8 workers to be present together — losing one
worker effectively loses the whole run's progress up to that point. Over enough reclamation events, the
job can consume more cumulative GPU-hours restarting repeatedly than a comparable on-demand run would have
cost outright, completely erasing the intended savings. Adding checkpointing at an interval shorter than
the spot instance type's typical time-between-reclamations is the single highest-leverage fix — it turns
each reclamation into a bounded loss (one checkpoint interval) rather than a total loss of progress.

## Summary and What's Next

AI/ML workloads on Kubernetes don't need a separate operational model — they need the same scheduling,
autoscaling, multi-tenancy, and storage primitives covered across this entire series, applied with
GPU-aware metrics and sizing instead of the generic CPU/memory defaults those primitives assume. The device
plugin framework makes GPUs schedulable at all; time-slicing, MIG, and MPS trade isolation for
utilization in different ways; and LLM-specific serving via vLLM/KServe needs application-level metrics
(queue depth, KV cache usage) that generic Resource-type autoscaling structurally cannot see.

The recurring theme worth carrying forward is that GPU capacity's economics — expensive, often
supply-constrained, idling at real dollar cost — sharpen every general Kubernetes discipline covered
earlier in this series rather than requiring a new one: scheduling becomes gang-aware, autoscaling needs
application-level metrics instead of generic resource utilization, and multi-tenancy's noisy-neighbor and
chargeback concerns matter proportionally more when the shared resource is a GPU instead of a CPU core.

Part 15, the final part of this series, turns to cluster lifecycle management — upgrading a running cluster
safely, whether self-managed (building on Part 6's kubeadm bootstrapping) or managed (extending Part 5's
provider-specific patterns), including the version-skew policy and rollback strategy that keeps an upgrade
from becoming an outage.