Scheduling & Workload Objects

Table of Contents#

1. Recap: What the Scheduler's Job Actually Is
2. The Two-Phase Scheduling Process
3. Filtering — Ruling Out Impossible Nodes
4. Scoring — Ranking the Possible Nodes
5. The Scheduler Framework — Extension Points
6. Resource Requests and Limits — Why They Drive Scheduling
7. ResourceQuota and LimitRange — Namespace-Level Resource Governance
8. Taints and Tolerations
9. Node Affinity and Anti-Affinity
10. Pod Affinity and Anti-Affinity
11. Topology Spread Constraints
12. Priority and Preemption
13. Preemption Mechanics and PodDisruptionBudgets
14. Deployments — The Workhorse
15. How a Rolling Update Actually Works
16. StatefulSets — For Things That Need an Identity
17. StatefulSet Update Strategies and Partitions
18. DaemonSets — One Per Node
19. Jobs and CronJobs
20. Job Completion Modes, Backoff, and Parallelism
21. CronJob Concurrency Policy and Missed Schedules
22. Init Containers and Native Sidecar Containers
23. Choosing the Right Workload Object
24. Part 2 CLI Cheat Sheet
25. Common Mistakes
26. Worked Practice Problems
27. Summary and What's Next

Recap: What the Scheduler's Job Actually Is#

Part 1 introduced the Scheduler at a high level: when a new Pod exists with no node assigned, the Scheduler decides which node it should run on. This part goes deep into exactly how that decision gets made, and then covers the higher-level objects (Deployments, StatefulSets, and more) that actually create pods in the first place.

The Two-Phase Scheduling Process#

Every scheduling decision happens in two distinct phases, and knowing both by name is a very commonly expected piece of Kubernetes interview knowledge.

Simple analogy: think of hiring for a job. Filtering is the initial resume screen — ruling out anyone who genuinely doesn't meet the minimum requirements (no relevant degree, wrong location). Scoring is the interview process that ranks the remaining, qualified candidates against each other to find the best fit. A candidate has to survive filtering before scoring is even relevant.

Worth stating precisely, since it's a common point of confusion: this two-phase process runs independently for EVERY unscheduled pod, one at a time — the scheduler doesn't batch-optimize placement across multiple pending pods simultaneously in its default behavior. This matters concretely for understanding scheduling throughput: a burst of many simultaneously-pending pods (a sudden scale-up event) is processed as a queue, each pod going through its own full filter-then-score cycle, which is part of why genuinely large, high-churn clusters care about scheduler performance and queue depth as real, monitorable metrics, not just an abstract implementation detail.

Filtering — Ruling Out Impossible Nodes#

Filtering asks, for every node in the cluster: "could this pod even physically/logically run here at all?"

A node that fails even ONE filter is completely removed from consideration — it's not "less preferred," it's genuinely disqualified.

A real, practical consequence worth naming: if every node in the cluster fails at least one filter for a given pod, that pod stays Pending indefinitely, with a FailedScheduling event explaining which filter(s) it couldn't satisfy — kubectl describe pod on a stuck-Pending pod is genuinely the first, most useful diagnostic step, since the event message names the specific unsatisfied constraint directly, rather than requiring guesswork.

This is also the exact trigger condition for autoscaling, worth connecting explicitly: a cluster autoscaler (or Karpenter, in Part 7's EKS-specific treatment) watches for precisely this Pending-with-FailedScheduling-due-to-insufficient-capacity signal to decide when new node capacity is genuinely needed.

Scoring — Ranking the Possible Nodes#

Of the nodes that survive filtering, scoring ranks them by a weighted combination of factors, and the Scheduler picks the highest-scoring one (with some randomization among close ties, to avoid always overloading the exact same "best" node).

The Scheduler Framework — Extension Points#

Filtering and scoring aren't a monolithic, hardcoded process — since Kubernetes 1.19, the scheduler is built on the Scheduler Framework, a pluggable architecture with well-defined extension points where custom logic (in-tree or your own plugin) can hook in.

Why this pluggable architecture matters, worth stating as a concrete, real-world example: it's exactly how specialized scheduling needs get built without forking the entire scheduler — a gang scheduling plugin (used for ML/batch training jobs where an entire group of pods must be scheduled together or not at all, since a partially-scheduled distributed training job wastes resources and never completes) hooks into the Permit extension point specifically to hold pods until the whole group can be placed simultaneously. Kubeflow's and Volcano's batch-scheduling plugins for Kubernetes are real, concrete examples of this pattern in production use.

The PostFilter extension point deserves a specific callout, since it's the mechanism behind preemption: it only runs when the Filter phase found zero feasible nodes for a pod — the default PostFilter plugin implements exactly the preemption logic described later in this Part, evaluating whether evicting lower-priority pods would open up a feasible placement.

Resource Requests and Limits — Why They Drive Scheduling#

This deserves its own dedicated section because it's genuinely one of the most operationally important, most commonly misunderstood settings in all of Kubernetes.

resources:
  requests:
    cpu: "500m"        # 0.5 CPU cores
    memory: "512Mi"
  limits:
    cpu: "1000m"        # 1 full CPU core
    memory: "1Gi"

The critical, frequently-tested distinction: requests drive scheduling; limits drive runtime enforcement. These are two completely separate mechanisms answering two completely different questions — "where should this run" (requests) versus "how much can it actually consume once it's running" (limits).

Quality of Service (QoS) Classes — A Direct Consequence#

Kubernetes automatically assigns every pod one of three QoS classes, purely based on how requests and limits are set — and this classification directly determines which pods get evicted first under real node pressure.

Why this matters practically, and it's a real, specific interview question: "which pod gets killed first if a node runs low on memory?" — the answer is always BestEffort pods first, then Burstable, and Guaranteed pods last, which is exactly why genuinely critical workloads (like a payments service) should always be configured with requests == limits (Guaranteed QoS), to maximize their survival priority during real node-level resource pressure.

ResourceQuota and LimitRange — Namespace-Level Resource Governance#

Individual pod-level requests/limits are only half the picture — real multi-tenant clusters (the pattern deepened further in Part 7's multi-tenancy section) need namespace-level governance on top, worth covering here since it directly shapes scheduling outcomes within a namespace.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-checkout-quota
  namespace: team-checkout
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: team-checkout-limits
  namespace: team-checkout
spec:
  limits:
    - type: Container
      default:
        cpu: "500m"
        memory: 512Mi
      defaultRequest:
        cpu: "250m"
        memory: 256Mi
      max:
        cpu: "2"
        memory: 4Gi
      min:
        cpu: "100m"
        memory: 128Mi

Why a request that would violate the namespace's ResourceQuota is a genuinely different failure than a scheduling failure, worth distinguishing precisely: a pod exceeding a ResourceQuota is rejected outright at admission time (Part 1's admission-control chain — the ResourceQuota admission controller specifically) — it never even reaches the Scheduler at all. This is meaningfully different from a pod that passes admission but then fails to schedule because no single node currently has enough free capacity — the first is a policy rejection (the request itself is disallowed for this namespace), the second is a capacity problem (the request is allowed, but nothing can currently satisfy it). Confusing these two failure modes during troubleshooting is a real, common mistake — kubectl describe pod on a ResourceQuota rejection shows nothing at all (the pod object was never created), while a scheduling failure shows a FailedScheduling event on a pod that does exist.

Why LimitRange's defaultRequest matters concretely, tying directly back to the QoS discussion above: a namespace with a LimitRange setting sensible defaults means a developer who forgets to set resource requests entirely doesn't accidentally get BestEffort QoS (the worst possible eviction priority) by omission — the namespace's default kicks in automatically, giving every pod at least Burstable QoS even when an individual manifest is incomplete. This is a genuinely valuable platform-team safety net, not just a convenience.

Taints and Tolerations#

A taint is applied to a node, and repels pods from scheduling there — unless the pod explicitly has a matching toleration.

Simple analogy: a taint is like a "staff only" sign on a door — by default it repels everyone, and only someone with the specific matching key (a toleration) is allowed through. Note it's opt-in for the pod, not opt-in for the node — the node is closed by default to anything without an explicit toleration.

# Taint a node to reserve it for a specific workload
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule

# A pod that tolerates this specific taint, allowing it to schedule there
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

The three taint effects worth knowing: NoSchedule (won't be scheduled here, but existing pods without the toleration aren't evicted), PreferNoSchedule (a soft version — try to avoid, but not a hard rule), and NoExecute (won't be scheduled AND existing pods without the toleration get actively evicted).

Node Affinity and Anti-Affinity#

While taints are node-side "keep out unless invited," node affinity is pod-side "I specifically want to run on nodes matching these characteristics."

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "disktype"
              operator: In
              values: ["ssd"]
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: "zone"
              operator: In
              values: ["us-east-1a"]

Reading the verbose field name ...IgnoredDuringExecution correctly, a genuinely important, often-missed detail: this means the rule is only checked at scheduling time — if a node's labels change after a pod is already running there (e.g., a label is removed), Kubernetes will NOT retroactively evict that pod, even though it would no longer satisfy the affinity rule if re-evaluated. Affinity rules are a scheduling-time decision, not a continuously-enforced invariant.

The matchExpressions operators, worth knowing the full set, not just In:

Why Exists/DoesNotExist are worth knowing specifically, beyond the more commonly-used In: they're the right tool when the presence of a label matters more than its specific value — e.g., "only schedule on nodes that have been explicitly labeled as GPU-capable at all" (Exists on a gpu-type key) versus needing to enumerate every possible GPU type value with In.

Pod Affinity and Anti-Affinity#

Similar mechanism, but the rule is based on other pods already running, not node labels — this is genuinely important for real availability design.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values: ["checkout-service"]
        topologyKey: "kubernetes.io/hostname"

Why this directly connects to the Reliability & Architecture Patterns series' redundancy discussion: having replicas: 3 in a Deployment guarantees 3 pods exist, but says absolutely nothing about where they end up — without pod anti-affinity, the Scheduler could (and under real conditions, sometimes will) place all 3 on the exact same node, silently creating a single point of failure that looks fine on paper (kubectl get pods shows "3/3 Running") but provides zero actual redundancy against a node failure.

Topology Spread Constraints#

Pod anti-affinity solves "don't co-locate with a specific other pod," but a genuinely common, more precise need is "spread replicas EVENLY across a topology domain (zones, nodes) — not just 'not on the same one.'" This is exactly what Topology Spread Constraints were built for, and they're the modern, more expressive tool for this specific job.

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: checkout-service

maxSkew is the field worth understanding precisely: it's the maximum allowed difference between the zone with the most matching pods and the zone with the fewest. maxSkew: 1 with 3 replicas across 3 zones means the ideal 1-1-1 split is enforced — a 2-1-0 split would violate maxSkew: 1 (difference of 2). whenUnsatisfiable mirrors the hard/soft distinction seen throughout this Part: DoNotSchedule makes it a hard requirement (like requiredDuringScheduling affinity), ScheduleAnyway makes it a soft preference (like preferredDuringScheduling) — the scheduler will still try to minimize skew, but won't refuse to schedule a pod if it can't achieve it.

Why Topology Spread Constraints are generally the more precise, modern tool over pod anti-affinity for this specific "spread evenly" goal, worth stating explicitly in an interview: anti-affinity is fundamentally a pairwise "avoid this other pod" rule, which can produce lopsided results at scale (it satisfies "not co-located" without any notion of overall balance); Topology Spread Constraints reason about the whole distribution across a topology domain directly, which is exactly the actual goal in almost every real-world "spread my replicas out" scenario. Both mechanisms can be used together — anti-affinity for hard co-location rules, spread constraints for balance.

A consolidated comparison across every scheduling-influence mechanism covered so far, worth having as a single reference:

Priority and Preemption#

When cluster resources are genuinely scarce, PriorityClass lets some pods matter more than others — and a sufficiently high-priority pod that can't otherwise be scheduled can actually evict (preempt) lower-priority pods to make room for itself.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 1000000
globalDefault: false
description: "For business-critical workloads only"

A genuinely important, practical caution worth naming explicitly: preemption is powerful but should be used deliberately and sparingly — an over-broadly-applied high PriorityClass can cause unexpected, hard-to-predict evictions of other legitimate workloads across the cluster, so it's typically reserved for a small, carefully-chosen set of genuinely critical services.

Preemption Mechanics and PodDisruptionBudgets#

Preemption's interaction with PodDisruptionBudgets (PDBs) — a mechanism most commonly discussed alongside voluntary disruptions like node drains — deserves precise treatment, since it's a genuinely common point of confusion.

The precise, worth-knowing nuance: the scheduler's preemption logic tries to respect PDBs where possible, but PDB protection is NOT absolute against preemption the way it is against a voluntary node drain. If evicting any combination of lower-priority pods that would satisfy the high-priority pod's needs would violate a PDB, the scheduler's default behavior still allows the preemption to proceed rather than leave the high-priority pod permanently unscheduled — preemption is treated as importance-driven eviction, not a fully PDB-gated voluntary disruption. A strong, precise interview answer states this exact distinction: "PDBs guarantee minimum availability during voluntary disruptions like node drains and cluster upgrades, but preemption for a genuinely higher-priority pod can still override that guarantee if no PDB-respecting eviction set exists — which is exactly why PriorityClass assignment needs real discipline, not liberal use."

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout-service

Graceful preemption, worth naming as a real mechanic, not an abrupt kill: a preempted pod still receives its normal termination grace period (SIGTERM, then SIGKILL after terminationGracePeriodSeconds — the exact same graceful-shutdown mechanics from Part 1's kubelet discussion) — preemption is a targeted eviction decision, not a bypass of normal pod termination behavior.

Deployments — The Workhorse#

A Deployment is the standard, default way to run a stateless application — it manages a ReplicaSet, which in turn manages the actual Pods, giving you declarative scaling, self-healing, and rolling updates.

Why the extra layer (ReplicaSet) exists, rather than the Deployment managing Pods directly — a genuinely insightful, less commonly known fact: this is exactly what makes rolling updates and rollbacks possible. When you update a Deployment's pod template (e.g., a new image version), Kubernetes creates a brand-new ReplicaSet for the new version, while keeping the old ReplicaSet around (scaled to zero) — this is literally how kubectl rollout undo works: it just scales the old ReplicaSet back up and the new one back down, near-instantly, without needing to "remember" the old configuration some other way.

# See both the current AND previous ReplicaSets for a Deployment
kubectl get replicasets -l app=checkout-service

# Roll back to the previous version - literally just flips
# which ReplicaSet is scaled up
kubectl rollout undo deployment/checkout-service

How a Rolling Update Actually Works#

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # at most 1 pod can be DOWN during the rollout
      maxSurge: 1            # at most 1 EXTRA pod above desired count

Why maxUnavailable and maxSurge are worth understanding as a genuine tradeoff, not just default settings to leave alone: maxSurge controls how many extra resources you're willing to briefly consume during a rollout (faster rollout, more resource usage); maxUnavailable controls how much capacity you're willing to briefly lose (faster rollout, less safety margin). Setting both to 0 would make a rollout impossible — there'd be no way to ever have a new pod ready before removing an old one — which is a genuinely good, concrete detail to know if asked to explain the settings' interaction.

Why readiness probes (from Part 1) are absolutely essential to a safe rollout, worth calling out explicitly: the rollout only proceeds to terminate an old pod once a corresponding new pod passes its readiness probe — without a properly configured readiness probe, Kubernetes has no reliable way to know a new pod is actually ready to serve traffic, and could terminate old, working pods while new ones are still silently broken.

StatefulSets — For Things That Need an Identity#

A StatefulSet is for workloads that need something a plain Deployment deliberately doesn't provide: a stable, predictable identity and stable storage tied to that identity, across restarts and rescheduling.

Why this connects directly to the stateful-vs-stateless scaling discussion from the Capacity Planning & Performance series: StatefulSets exist specifically because stateful workloads can't be treated as interchangeable, randomly-named, disposable copies the way stateless Deployment pods can — postgres-0 needs to reliably come back as postgres-0, with its same specific storage volume reattached, every single time it restarts or gets rescheduled, which is exactly the guarantee a StatefulSet (and Deployments deliberately do not) provides.

# StatefulSet pods get PREDICTABLE, stable DNS names too
# e.g. postgres-0.postgres-headless-svc.default.svc.cluster.local

The ordinal index guarantee, worth stating precisely — it governs BOTH creation/deletion order AND scaling: a StatefulSet with replicas: 3 creates postgres-0 first, waits for it to be Running and Ready, THEN creates postgres-1, then postgres-2 — strictly sequential, never in parallel by default. Scaling down reverses this: the highest-ordinal pod (postgres-2) is removed first, never an arbitrary one. Why this ordering matters concretely, a real distributed-systems reason, not just "for tidiness": many stateful systems (etcd itself, from Part 1, is a real example) rely on a specific member joining/leaving order for safe cluster membership changes — removing a random member instead of the most-recently-added one could, for some systems, threaten quorum or replication consistency in ways a strictly ordinal, predictable removal order avoids.

StatefulSet Update Strategies and Partitions#

StatefulSets support the same RollingUpdate strategy concept as Deployments, but with StatefulSet-specific mechanics worth knowing precisely, since a database rollout going wrong is a genuinely higher-stakes mistake than a stateless service rollout going wrong.

Why reverse-ordinal update order specifically, worth explaining as a deliberate design choice: for many real stateful systems, pod-0 is conventionally the first-created, often the "senior" or primary-adjacent member — updating from the highest ordinal downward means the most established, typically most cautious-to-disturb member is touched last, giving the update process the most opportunity to be caught and halted before it reaches the most critical instance.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2   # only postgres-2 and higher get updated; 0 and 1 are left alone

partition is a genuinely powerful, worth-knowing StatefulSet-specific safety mechanism: setting partition: 2 on a 3-replica StatefulSet means only postgres-2 receives the update — postgres-0 and postgres-1 are left entirely untouched, even if the pod template changes. This enables a manual canary pattern for stateful workloads: update the partition to the highest ordinal only, verify the single updated replica behaves correctly under real conditions, then lower the partition value incrementally to roll the update out to the rest — a genuinely important pattern for anything where a bad rollout to a stateful workload (a corrupted migration, an incompatible on-disk format change) is far more costly and harder to reverse than a stateless rollback.

DaemonSets — One Per Node#

A DaemonSet ensures exactly one copy of a pod runs on every (or every matching) node in the cluster — automatically adding a copy when a new node joins, and removing it when a node leaves.

The classic, real-world use case, worth citing specifically: log-shipping agents, monitoring agents (like a Prometheus Node Exporter), and networking/CNI components (covered in Part 3) — anything that genuinely needs to run on every single node, exactly once, rather than being scaled to an arbitrary replica count the way a Deployment would be.

Jobs and CronJobs#

For work that's meant to run to completion and then stop, rather than run continuously forever.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"       # 2 AM every day, standard cron syntax
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: backup-tool:1.0
          restartPolicy: OnFailure

A genuinely important distinction worth naming: a Deployment's restartPolicy is always Always (it's meant to run forever), while a Job's is OnFailure or Never (it's meant to eventually stop, not be restarted indefinitely) — using the wrong workload type entirely for a batch task (e.g., a Deployment for a one-time migration script) means Kubernetes will keep trying to "heal" a container that's supposed to have finished and exited, treating successful completion as if it were a crash.

Job Completion Modes, Backoff, and Parallelism#

Real Job usage goes well beyond "run one pod to completion" — worth knowing the fields that control retries, parallel execution, and completion tracking precisely.

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-image-processing
spec:
  completions: 10          # need 10 SUCCESSFUL pod completions total
  parallelism: 3           # run up to 3 pods AT ONCE
  backoffLimit: 4          # give up after 4 FAILED attempts (per pod slot)
  activeDeadlineSeconds: 3600   # kill the whole Job if it's not done in 1 hour
  completionMode: Indexed  # each pod gets a unique index 0..9 (env var JOB_COMPLETION_INDEX)
  template:
    spec:
      containers:
        - name: worker
          image: image-processor:1.0
      restartPolicy: OnFailure

completionMode: Indexed, worth knowing as the modern, more powerful mode — a real, common upgrade from the legacy NonIndexed default: each pod in an Indexed Job gets a unique, predictable completion index (0 through completions - 1) injected as the JOB_COMPLETION_INDEX environment variable — this is exactly what makes parallel, partitioned batch processing clean to implement (e.g., "pod 3 processes shard 3 of the dataset") without needing an external work-queue coordination system just to hand out unique work assignments.

Why backoffLimit's exponential backoff matters concretely, connecting directly to the retry/backoff patterns in the Reliability & Architecture Patterns series: a Job retrying a failing pod with a fixed, short interval against a genuinely down dependency (a database that's temporarily unreachable) would just hammer that dependency repeatedly — exponential backoff is the same general resilience pattern already covered for application-level retries, applied here at the Job-controller level automatically, with zero extra code needed in the workload itself.

CronJob Concurrency Policy and Missed Schedules#

CronJobs add scheduling on top of Jobs, and two fields genuinely matter for correctness, not just convenience.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec: {}

Why concurrencyPolicy: Allow (the default) is a genuinely common, real production trap, worth stating explicitly: a nightly backup job that occasionally runs long (say, database growth makes it take 90 minutes instead of the usual 30) can, under the default Allow policy, end up with a SECOND backup job starting before the first finishes — two backup processes running simultaneously against the same database is very often not just wasteful but can be genuinely harmful (resource contention, inconsistent concurrent snapshots). Forbid is almost always the correct choice for any CronJob whose runs must never meaningfully overlap (backups, database migrations, anything touching shared state), while Replace fits a workload where only the latest attempt genuinely matters (e.g., "refresh a cache" — an older, superseded run should just be abandoned in favor of the newest one).

startingDeadlineSeconds, a real, worth-knowing edge case around missed schedules: if the CronJob controller itself is down (a control plane issue, per Part 1's leader-election discussion) when a scheduled time passes, the run is "missed." Without startingDeadlineSeconds set, Kubernetes has a default tolerance (historically around 100 seconds) for how late a missed run can start and still be considered "on schedule" rather than skipped entirely — explicitly setting this field makes that tolerance a deliberate, documented decision rather than an implicit default a future on-call engineer has to go discover during an incident.

Init Containers and Native Sidecar Containers#

Two container-lifecycle mechanisms worth understanding precisely, since "just add another container" is not how either of them work — the ordering and lifecycle guarantees are the entire point.

spec:
  initContainers:
    - name: wait-for-db
      image: busybox
      command: ['sh', '-c', 'until nc -z db-host 5432; do sleep 2; done']
  containers:
    - name: log-shipper
      image: fluent-bit
      restartPolicy: Always   # THIS is what makes it a "native sidecar" (1.29+)
    - name: app
      image: checkout:1.2.3

Init containers, the precise mental model worth stating explicitly: they run to completion, strictly sequentially, before ANY main container starts — a genuinely common real use case is exactly what's shown above (block until a dependency is reachable) or running a one-time setup/migration step that the main container assumes has already happened. If an init container fails, the kubelet retries it (respecting the pod's restartPolicy) — the main containers never start until every init container has succeeded.

Native sidecar containers (restartPolicy: Always set on a specific container within containers, not initContainers), a genuinely important, relatively recent (Kubernetes 1.29 stable) improvement worth knowing precisely: before this feature, a "sidecar" was just a convention (a log-shipper or service-mesh proxy container alongside the main app), with a real, concrete problem — a Job's sidecar container that never exits on its own would keep the whole Pod "Running" forever, since Kubernetes had no way to know it was a supporting sidecar versus a co-equal main container. Native sidecars solve this precisely: marking a container with restartPolicy: Always inside the pod spec makes Kubernetes treat it as a genuine sidecar — it starts before the main containers, and critically, the pod is considered complete once the main containers finish, regardless of whether the sidecar is still running — directly fixing the "Job never completes because of a lingering sidecar" problem that plagued the convention-only approach for years.

Choosing the Right Workload Object#

Part 2 CLI Cheat Sheet#

# Scheduling diagnostics
kubectl describe pod <pod> | grep -A 20 Events    # WHY didn't this pod schedule?
kubectl get events --field-selector reason=FailedScheduling
kubectl get pods -o wide --sort-by='.spec.nodeName'   # see actual placement

# Taints and tolerations
kubectl taint nodes <node> key=value:NoSchedule
kubectl taint nodes <node> key=value:NoSchedule-    # remove a taint
kubectl describe node <node> | grep Taints

# Priority classes
kubectl get priorityclasses
kubectl get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority

# Deployments and rollouts
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name> --to-revision=2
kubectl scale deployment/<name> --replicas=5

# StatefulSets
kubectl get statefulset <name> -o jsonpath='{.spec.updateStrategy.rollingUpdate.partition}'
kubectl patch statefulset <name> -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":1}}}}'

# Jobs and CronJobs
kubectl create job manual-run --from=cronjob/<name>   # trigger a CronJob manually, right now
kubectl get jobs --watch
kubectl logs job/<name>

Common Mistakes#

Worked Practice Problems#

Problem 1: A node runs low on memory and Kubernetes needs to evict something. Three pods are running on it: Pod A (no requests/limits set), Pod B (requests=limits), Pod C (requests set, limits higher than requests). In what order are they likely evicted, and why?

Answer: Pod A (BestEffort QoS — no requests or limits at all) is evicted first, since it has no guaranteed resources at all and the lowest eviction priority. Pod C (Burstable QoS — requests set, but limits higher, meaning it's allowed to burst beyond its guarantee) is evicted next if pressure continues. Pod B (Guaranteed QoS — requests exactly equal limits) is evicted last, since it has the strongest resource guarantee and highest survival priority under node pressure.

Problem 2: A team runs 3 replicas of a critical service via a Deployment with no pod anti-affinity configured. During a node failure, all 3 replicas go down simultaneously, causing a full outage despite having "3 replicas." What went wrong, and how would you prevent it going forward?

Answer: Without pod anti-affinity, the Scheduler had no rule preventing it from placing all 3 replicas on the same node — which is exactly what happened, silently creating a single point of failure that kubectl get pods would have shown as healthy (3/3 Running) right up until that one node failed. Fix: add a podAntiAffinity rule (using topologyKey: kubernetes.io/hostname, or even better, topology.kubernetes.io/zone for multi-AZ spread) requiring replicas of this Deployment to avoid scheduling on the same node/zone as each other, ensuring a single node or zone failure can never take down every replica simultaneously.

Problem 3: A one-time database migration script is deployed as a Kubernetes Deployment. After it finishes successfully and the container exits, Kubernetes keeps restarting it in a loop. What's the root cause, and what's the correct fix?

Answer: A Deployment's pods always use restartPolicy: Always, since Deployments are designed for workloads meant to run forever — Kubernetes has no way to know this particular container's successful exit was actually the intended, successful end state rather than a crash, so it keeps restarting it exactly as designed for a continuously-running service. The correct fix is using a Job instead of a Deployment for this one-time task — a Job's semantics (restartPolicy: OnFailure or Never, and tracking actual completion) correctly understand that a successful exit means the work is done, not that something crashed.

Problem 4: A team needs to roll out a risky schema-migrating update to a 5-replica StatefulSet running a database, and wants to verify the change on exactly one replica before touching the rest. What StatefulSet-specific mechanism enables this, and how would they use it?

Answer: The partition field on the StatefulSet's rollingUpdate strategy. Setting partition: 4 on a 5-replica (ordinals 0-4) StatefulSet means only the highest-ordinal pod, postgres-4, receives the updated pod template — ordinals 0 through 3 are left completely untouched. The team can verify postgres-4 behaves correctly under real traffic/data conditions, then lower the partition value incrementally (4, then 3, then 2...) to roll the update out to the rest of the replicas one at a time, with the ability to halt the rollout at any point by simply not lowering the partition further — a genuinely safer, manually-gated canary pattern specifically suited to stateful workloads where a bad rollout is far more costly than a stateless service's.

Problem 5: A nightly CronJob backup occasionally runs long due to database growth, and the team discovers two backup processes briefly ran concurrently against the same database, causing resource contention and an inconsistent snapshot. What field was misconfigured, and what's the fix?

Answer: The CronJob's concurrencyPolicy was left at its default value, Allow, which permits a new scheduled run to start even while a previous run is still executing. The fix is setting concurrencyPolicy: Forbid explicitly — this makes the CronJob controller skip starting a new run entirely if the previous execution hasn't completed yet, guaranteeing backups never overlap. (Replace, which cancels the in-progress run and starts fresh, would be the wrong choice here specifically because an in-progress backup being killed mid-write is arguably worse than simply skipping a run — the correct policy depends on which failure mode is actually more acceptable for the specific workload.)

Problem 6: A batch-processing Job includes a log-shipping container alongside the main worker container. The worker finishes its work and exits successfully, but the Job never shows as Complete — kubectl get jobs shows it stuck indefinitely. What's the root cause, and what's the fix?

Answer: The log-shipping container was added as an ordinary container without being marked as a native sidecar, and it has no natural exit condition of its own (it just keeps running, waiting for logs to ship) — Kubernetes has no way to distinguish "a co-equal main container that's supposed to keep running" from "a supporting sidecar that should be cleaned up once the real work is done," so the Pod (and therefore the Job) never reaches a Complete state as long as ANY container is still running. The fix: set restartPolicy: Always on the log-shipping container specifically (Kubernetes 1.29+'s native sidecar feature) — this tells Kubernetes explicitly that Job/Pod completion should be judged by the main worker container alone, and the sidecar will be automatically terminated once the main container exits, rather than keeping the whole Pod alive indefinitely.

Summary and What's Next#

• Scheduling happens in two phases: filtering (which nodes are even capable of running this pod) and scoring (of those, which is the best choice) — a node failing even one filter is completely disqualified, not just deprioritized.
• Resource requests drive scheduling placement; limits drive runtime enforcement (CPU throttling, memory OOM-kill) — two separate mechanisms answering two different questions, and together they determine a pod's QoS class (Guaranteed > Burstable > BestEffort), which directly determines eviction order under node pressure.
• Taints repel pods from a node unless they carry a matching toleration; node affinity is the pod-side preference/requirement for specific node characteristics — and affinity rules are checked only at scheduling time, not continuously enforced afterward.
• Pod anti-affinity is essential for real multi-replica availability — without it, a Deployment's replica count says nothing about whether they're actually spread across different nodes/zones.
• PriorityClass and preemption let critical pods evict lower-priority ones when the cluster is genuinely full — powerful, and worth applying deliberately and narrowly.
• Deployments manage a ReplicaSet (which manages the actual Pods) — this extra layer is exactly what makes rolling updates and near-instant rollbacks possible, by keeping the old ReplicaSet around, scaled to zero, rather than needing to reconstruct the previous configuration.
• maxSurge/maxUnavailable control the real tradeoff between rollout speed/resource usage and safety margin during a rolling update — and readiness probes are what make the whole rollout process safe in the first place.
• StatefulSets provide stable identity and storage for workloads (like databases) that genuinely need it, create/delete pods in strict ordinal order, and support a partition-based manual canary pattern for high-stakes rollouts; DaemonSets run exactly one pod per node; Jobs/CronJobs are for work meant to run to completion, not forever — using the wrong workload type for a given need causes real, specific, predictable problems.
• The Scheduler Framework's pluggable extension points (PreFilter, Filter, Score, Permit, Bind, and more) are what make specialized scheduling — like gang scheduling for ML/batch training — possible without forking the scheduler itself.
• Topology Spread Constraints are the more precise modern tool for genuine even-distribution goals, complementing (not replacing) pod anti-affinity's pairwise co-location rules.
• Preemption can override a PodDisruptionBudget's guarantee — PDBs fully protect against voluntary disruptions like drains, but not necessarily against a genuinely higher-priority pod needing the space.
• Job completionMode: Indexed gives each pod a predictable completion index, enabling clean partitioned parallel batch processing without external coordination; CronJob concurrencyPolicy (default Allow) is a real, common source of harmful overlapping runs if left unconsidered for workloads that must never run concurrently.
• ResourceQuota rejects at admission time, before a pod object even exists; LimitRange supplies sensible per-container defaults so an incomplete manifest doesn't silently end up BestEffort — two distinct, complementary namespace-level governance mechanisms, easy to confuse with a plain scheduling failure during troubleshooting.
• Init containers run sequentially to completion before any main container starts; native sidecar containers (restartPolicy: Always, Kubernetes 1.29+) start before the main containers and are excluded from the pod's completion condition — directly fixing the long-standing "lingering sidecar keeps a Job running forever" problem.

Continue to Part 3 (03-networking-and-storage.md) to see how pods actually talk to each other and the outside world (the CNI networking model, Services, Ingress) and how they get durable storage (the CSI model, PersistentVolumes).