# Kubernetes Deep Dive — Part 2: Scheduling & Workload Objects
> **Series:** Kubernetes Deep Dive (2 of 9)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** This file — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Questions:** `questions.md`
## Table of Contents
1. [Recap: What the Scheduler's Job Actually Is](#recap-what-the-schedulers-job-actually-is)
2. [The Two-Phase Scheduling Process](#the-two-phase-scheduling-process)
3. [Filtering — Ruling Out Impossible Nodes](#filtering--ruling-out-impossible-nodes)
4. [Scoring — Ranking the Possible Nodes](#scoring--ranking-the-possible-nodes)
5. [The Scheduler Framework — Extension Points](#the-scheduler-framework--extension-points)
6. [Resource Requests and Limits — Why They Drive Scheduling](#resource-requests-and-limits--why-they-drive-scheduling)
7. [ResourceQuota and LimitRange — Namespace-Level Resource Governance](#resourcequota-and-limitrange--namespace-level-resource-governance)
8. [Taints and Tolerations](#taints-and-tolerations)
9. [Node Affinity and Anti-Affinity](#node-affinity-and-anti-affinity)
10. [Pod Affinity and Anti-Affinity](#pod-affinity-and-anti-affinity)
11. [Topology Spread Constraints](#topology-spread-constraints)
12. [Priority and Preemption](#priority-and-preemption)
13. [Preemption Mechanics and PodDisruptionBudgets](#preemption-mechanics-and-poddisruptionbudgets)
14. [Deployments — The Workhorse](#deployments--the-workhorse)
15. [How a Rolling Update Actually Works](#how-a-rolling-update-actually-works)
16. [StatefulSets — For Things That Need an Identity](#statefulsets--for-things-that-need-an-identity)
17. [StatefulSet Update Strategies and Partitions](#statefulset-update-strategies-and-partitions)
18. [DaemonSets — One Per Node](#daemonsets--one-per-node)
19. [Jobs and CronJobs](#jobs-and-cronjobs)
20. [Job Completion Modes, Backoff, and Parallelism](#job-completion-modes-backoff-and-parallelism)
21. [CronJob Concurrency Policy and Missed Schedules](#cronjob-concurrency-policy-and-missed-schedules)
22. [Init Containers and Native Sidecar Containers](#init-containers-and-native-sidecar-containers)
23. [Choosing the Right Workload Object](#choosing-the-right-workload-object)
24. [Part 2 CLI Cheat Sheet](#part-2-cli-cheat-sheet)
25. [Common Mistakes](#common-mistakes)
26. [Worked Practice Problems](#worked-practice-problems)
27. [Summary and What's Next](#summary-and-whats-next)
---
## Recap: What the Scheduler's Job Actually Is
Part 1 introduced the Scheduler at a high level: when a new Pod exists with no node assigned, the Scheduler decides which node it should run on. This part goes deep into exactly *how* that decision gets made, and then covers the higher-level objects (Deployments, StatefulSets, and more) that actually create pods in the first place.
---
## The Two-Phase Scheduling Process
Every scheduling decision happens in two distinct phases, and knowing both by name is a very commonly expected piece of Kubernetes interview knowledge.
```mermaid
flowchart TD
Pod["New, unscheduled Pod"] --> Filter["PHASE 1: FILTERING
('predicates')
Which nodes are even
CAPABLE of running this pod?"]
Filter --> Filtered["A shortlist of
FEASIBLE nodes"]
Filtered --> Score["PHASE 2: SCORING
('priorities')
Of the feasible nodes,
which is the BEST choice?"]
Score --> Best["Highest-scoring node
is selected"]
```
**Simple analogy:** think of hiring for a job. Filtering is the initial resume screen — ruling out anyone who genuinely doesn't meet the minimum requirements (no relevant degree, wrong location). Scoring is the interview process that ranks the remaining, *qualified* candidates against each other to find the best fit. A candidate has to survive filtering before scoring is even relevant.
**Worth stating precisely, since it's a common point of confusion: this two-phase process runs independently for EVERY unscheduled pod, one at a time** — the scheduler doesn't batch-optimize placement across multiple pending pods simultaneously in its default behavior. This matters concretely for understanding scheduling throughput: a burst of many simultaneously-pending pods (a sudden scale-up event) is processed as a queue, each pod going through its own full filter-then-score cycle, which is part of why genuinely large, high-churn clusters care about scheduler performance and queue depth as real, monitorable metrics, not just an abstract implementation detail.
---
## Filtering — Ruling Out Impossible Nodes
Filtering asks, for every node in the cluster: **"could this pod even physically/logically run here at all?"**
```mermaid
graph TD
Filters[Common Filter Checks] --> F1["Does the node have ENOUGH
free CPU/memory to satisfy
the pod's resource requests?"]
Filters --> F2["Does the node have a TAINT
this pod doesn't TOLERATE?
(covered below)"]
Filters --> F3["Does the pod's nodeSelector/
required node affinity
actually MATCH this node's
labels?"]
Filters --> F4["Is the required VOLUME
actually accessible from
this node?"]
Filters --> F5["Does this node ALREADY
have a port conflict with
what the pod needs?"]
```
A node that fails even ONE filter is completely removed from consideration — it's not "less preferred," it's genuinely disqualified.
**A real, practical consequence worth naming:** if every node in the cluster fails at least one filter for a given pod, that pod stays `Pending` indefinitely, with a `FailedScheduling` event explaining which filter(s) it couldn't satisfy — `kubectl describe pod` on a stuck-Pending pod is genuinely the first, most useful diagnostic step, since the event message names the specific unsatisfied constraint directly, rather than requiring guesswork.
**This is also the exact trigger condition for autoscaling**, worth connecting explicitly: a cluster autoscaler (or Karpenter, in Part 7's EKS-specific treatment) watches for precisely this `Pending`-with-`FailedScheduling`-due-to-insufficient-capacity signal to decide when new node capacity is genuinely needed.
---
## Scoring — Ranking the Possible Nodes
Of the nodes that survive filtering, scoring ranks them by a weighted combination of factors, and the Scheduler picks the highest-scoring one (with some randomization among close ties, to avoid always overloading the exact same "best" node).
```mermaid
graph TD
Scoring[Common Scoring Factors] --> S1["Resource balance: prefer a
node that keeps CPU and
memory usage roughly
proportional, rather than
maxing out one resource
while another sits idle"]
Scoring --> S2["Spreading: prefer to spread
pods from the SAME
Deployment/Service across
DIFFERENT nodes (for
availability)"]
Scoring --> S3["Affinity/anti-affinity
PREFERENCES (soft rules,
covered below)"]
Scoring --> S4["Image locality: prefer a
node that ALREADY has the
container image cached
locally (faster startup,
no pull needed)"]
```
---
## The Scheduler Framework — Extension Points
Filtering and scoring aren't a monolithic, hardcoded process — since Kubernetes 1.19, the scheduler is built on the **Scheduler Framework**, a pluggable architecture with well-defined **extension points** where custom logic (in-tree or your own plugin) can hook in.
```mermaid
flowchart TD
QueueSort["QueueSort - order the
scheduling queue"] --> PreFilter["PreFilter - prep/check
before filtering begins"]
PreFilter --> Filter["Filter - the filtering
phase (predicates)"]
Filter --> PostFilter["PostFilter - runs ONLY if
filtering found nothing
feasible (e.g. triggers
preemption logic)"]
Filter --> PreScore["PreScore - prep before
scoring"]
PreScore --> Score["Score - the scoring phase
(priorities)"]
Score --> Reserve["Reserve - reserve resources
on the winning node"]
Reserve --> Permit["Permit - a LAST chance to
delay/deny (e.g. gang
scheduling - wait for a
whole group of pods)"]
Permit --> PreBind["PreBind - work needed
before actually binding
(e.g. provisioning a volume)"]
PreBind --> Bind["Bind - actually assign
the pod to the node"]
Bind --> PostBind["PostBind - informational,
after binding succeeds"]
```
**Why this pluggable architecture matters, worth stating as a concrete, real-world example:** it's exactly how specialized scheduling needs get built without forking the entire scheduler — a **gang scheduling** plugin (used for ML/batch training jobs where an entire group of pods must be scheduled together or not at all, since a partially-scheduled distributed training job wastes resources and never completes) hooks into the `Permit` extension point specifically to hold pods until the whole group can be placed simultaneously. Kubeflow's and Volcano's batch-scheduling plugins for Kubernetes are real, concrete examples of this pattern in production use.
**The `PostFilter` extension point deserves a specific callout, since it's the mechanism behind preemption**: it only runs when the `Filter` phase found zero feasible nodes for a pod — the default `PostFilter` plugin implements exactly the preemption logic described later in this Part, evaluating whether evicting lower-priority pods would open up a feasible placement.
---
## Resource Requests and Limits — Why They Drive Scheduling
This deserves its own dedicated section because it's genuinely one of the most operationally important, most commonly misunderstood settings in all of Kubernetes.
```yaml
resources:
requests:
cpu: "500m" # 0.5 CPU cores
memory: "512Mi"
limits:
cpu: "1000m" # 1 full CPU core
memory: "1Gi"
```
```mermaid
graph TD
Requests["REQUESTS: what the
Scheduler uses to DECIDE
placement — a GUARANTEE
this pod always gets AT
LEAST this much"] --> RequestsNote["The Scheduler will NEVER
place a pod on a node that
doesn't have enough
UNRESERVED capacity to
cover its requests"]
Limits["LIMITS: the MAXIMUM this
container is EVER allowed
to use, enforced by the
kernel's cgroups (Linux &
Networking Fundamentals
series, Part 1)"] --> LimitsNote["Exceeding the CPU limit
gets the process THROTTLED
(slowed down); exceeding the
MEMORY limit gets it
OOM-KILLED (Linux series,
Part 1)"]
```
**The critical, frequently-tested distinction: requests drive scheduling; limits drive runtime enforcement.** These are two completely separate mechanisms answering two completely different questions — "where should this run" (requests) versus "how much can it actually consume once it's running" (limits).
### Quality of Service (QoS) Classes — A Direct Consequence
Kubernetes automatically assigns every pod one of three QoS classes, purely based on how requests and limits are set — and this classification directly determines which pods get evicted first under real node pressure.
```mermaid
graph TD
Guaranteed["Guaranteed:
requests == limits
for EVERY container"] --> GuaranteedNote["Highest priority - LAST
to be evicted under
node memory pressure"]
Burstable["Burstable:
requests SET but LESS
than limits (or limits
not set at all)"] --> BurstableNote["Middle priority"]
BestEffort["BestEffort:
NO requests or limits
set at all"] --> BestEffortNote["Lowest priority - FIRST
to be evicted under
node memory pressure"]
```
**Why this matters practically, and it's a real, specific interview question:** "which pod gets killed first if a node runs low on memory?" — the answer is always **BestEffort pods first, then Burstable, and Guaranteed pods last**, which is exactly why genuinely critical workloads (like a payments service) should always be configured with `requests == limits` (Guaranteed QoS), to maximize their survival priority during real node-level resource pressure.
---
## ResourceQuota and LimitRange — Namespace-Level Resource Governance
Individual pod-level requests/limits are only half the picture — real multi-tenant clusters (the pattern deepened further in Part 7's multi-tenancy section) need namespace-level governance on top, worth covering here since it directly shapes scheduling outcomes within a namespace.
```mermaid
graph TD
NS["Namespace: team-checkout"] --> RQ["ResourceQuota: caps TOTAL
resource consumption across
ALL pods in this namespace"]
NS --> LR["LimitRange: sets DEFAULT
requests/limits for pods
that don't specify their
own, and caps the min/max
a single container can
request"]
```
```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-checkout-quota
namespace: team-checkout
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.cpu: "40"
limits.memory: 80Gi
pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
name: team-checkout-limits
namespace: team-checkout
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: 512Mi
defaultRequest:
cpu: "250m"
memory: 256Mi
max:
cpu: "2"
memory: 4Gi
min:
cpu: "100m"
memory: 128Mi
```
**Why a request that would violate the namespace's ResourceQuota is a genuinely different failure than a scheduling failure, worth distinguishing precisely:** a pod exceeding a ResourceQuota is rejected outright at **admission time** (Part 1's admission-control chain — the `ResourceQuota` admission controller specifically) — it never even reaches the Scheduler at all. This is meaningfully different from a pod that passes admission but then fails to *schedule* because no single node currently has enough free capacity — the first is a policy rejection (the request itself is disallowed for this namespace), the second is a capacity problem (the request is allowed, but nothing can currently satisfy it). **Confusing these two failure modes during troubleshooting is a real, common mistake** — `kubectl describe pod` on a ResourceQuota rejection shows nothing at all (the pod object was never created), while a scheduling failure shows a `FailedScheduling` event on a pod that *does* exist.
**Why `LimitRange`'s `defaultRequest` matters concretely, tying directly back to the QoS discussion above:** a namespace with a `LimitRange` setting sensible defaults means a developer who forgets to set resource requests entirely doesn't accidentally get BestEffort QoS (the worst possible eviction priority) by omission — the namespace's default kicks in automatically, giving every pod at least Burstable QoS even when an individual manifest is incomplete. This is a genuinely valuable platform-team safety net, not just a convenience.
---
## Taints and Tolerations
A **taint** is applied to a *node*, and repels pods from scheduling there — unless the pod explicitly has a matching **toleration**.
```mermaid
graph TD
Node["Node tainted:
'dedicated=gpu:NoSchedule'"] --> Rule["Rule: NO pod may schedule
here UNLESS it explicitly
tolerates this exact taint"]
Pod1["Regular Pod
(no toleration)"] -.->|"❌ REPELLED"| Node
Pod2["GPU-workload Pod
(has matching toleration)"] -.->|"✅ Allowed"| Node
```
**Simple analogy:** a taint is like a "staff only" sign on a door — by default it repels everyone, and only someone with the specific matching key (a toleration) is allowed through. Note it's opt-in for the *pod*, not opt-in for the *node* — the node is closed by default to anything without an explicit toleration.
```bash
# Taint a node to reserve it for a specific workload
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule
```
```yaml
# A pod that tolerates this specific taint, allowing it to schedule there
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
```
**The three taint effects worth knowing:** `NoSchedule` (won't be scheduled here, but existing pods without the toleration aren't evicted), `PreferNoSchedule` (a soft version — try to avoid, but not a hard rule), and `NoExecute` (won't be scheduled AND existing pods without the toleration get actively evicted).
---
## Node Affinity and Anti-Affinity
While taints are node-side "keep out unless invited," **node affinity** is pod-side "I specifically want to run on nodes matching these characteristics."
```mermaid
graph TD
Affinity[Node Affinity Types] --> Required["requiredDuringScheduling...
(HARD requirement - acts
like an extra FILTER)"]
Affinity --> Preferred["preferredDuringScheduling...
(SOFT preference - acts
like an extra SCORING
factor, not a hard rule)"]
```
```yaml
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "disktype"
operator: In
values: ["ssd"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: "zone"
operator: In
values: ["us-east-1a"]
```
**Reading the verbose field name `...IgnoredDuringExecution` correctly, a genuinely important, often-missed detail:** this means the rule is only checked at scheduling time — if a node's labels change *after* a pod is already running there (e.g., a label is removed), Kubernetes will NOT retroactively evict that pod, even though it would no longer satisfy the affinity rule if re-evaluated. Affinity rules are a scheduling-time decision, not a continuously-enforced invariant.
**The `matchExpressions` operators, worth knowing the full set, not just `In`:**
| Operator | Meaning |
|---|---|
| `In` | The label's value is one of the listed values |
| `NotIn` | The label's value is NOT one of the listed values |
| `Exists` | The label key exists, regardless of its value (no `values` field needed) |
| `DoesNotExist` | The label key does NOT exist on this node |
| `Gt` | The label's value, parsed as an integer, is greater than the given value |
| `Lt` | The label's value, parsed as an integer, is less than the given value |
**Why `Exists`/`DoesNotExist` are worth knowing specifically, beyond the more commonly-used `In`:** they're the right tool when the *presence* of a label matters more than its specific value — e.g., "only schedule on nodes that have been explicitly labeled as GPU-capable at all" (`Exists` on a `gpu-type` key) versus needing to enumerate every possible GPU type value with `In`.
---
## Pod Affinity and Anti-Affinity
Similar mechanism, but the rule is based on **other pods already running**, not node labels — this is genuinely important for real availability design.
```mermaid
graph TD
PodAntiAffinity["Pod ANTI-affinity:
'don't schedule me on the
SAME node as another pod
from my own Deployment'"] --> Why["WHY this matters: if all
3 replicas of checkout-
service end up on the
SAME node, that node
dying takes down the
ENTIRE service at once —
defeating the whole point
of having 3 replicas"]
```
```yaml
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: ["checkout-service"]
topologyKey: "kubernetes.io/hostname"
```
**Why this directly connects to the Reliability & Architecture Patterns series' redundancy discussion:** having `replicas: 3` in a Deployment guarantees 3 pods exist, but says absolutely nothing about *where* they end up — without pod anti-affinity, the Scheduler could (and under real conditions, sometimes will) place all 3 on the exact same node, silently creating a single point of failure that looks fine on paper (`kubectl get pods` shows "3/3 Running") but provides zero actual redundancy against a node failure.
---
## Topology Spread Constraints
Pod anti-affinity solves "don't co-locate with a specific other pod," but a genuinely common, more precise need is **"spread replicas EVENLY across a topology domain (zones, nodes) — not just 'not on the same one.'"** This is exactly what **Topology Spread Constraints** were built for, and they're the modern, more expressive tool for this specific job.
```mermaid
graph TD
Bad["Pod anti-affinity ONLY:
guarantees no two
replicas share a node,
but says NOTHING about
BALANCE - could still be
2 in zone A, 1 in zone B,
0 in zone C"] --> BadNote["Uneven distribution still
leaves zone A carrying
disproportionate load/risk"]
Good["Topology Spread
Constraint: 'keep the
MAX DIFFERENCE between
the most-loaded and
least-loaded zone within
maxSkew'"] --> GoodNote["Actively enforces
EVEN distribution, not
just 'no exact
duplicates'"]
```
```yaml
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: checkout-service
```
**`maxSkew` is the field worth understanding precisely:** it's the maximum allowed difference between the zone with the *most* matching pods and the zone with the *fewest*. `maxSkew: 1` with 3 replicas across 3 zones means the ideal 1-1-1 split is enforced — a 2-1-0 split would violate `maxSkew: 1` (difference of 2). **`whenUnsatisfiable` mirrors the hard/soft distinction seen throughout this Part**: `DoNotSchedule` makes it a hard requirement (like `requiredDuringScheduling` affinity), `ScheduleAnyway` makes it a soft preference (like `preferredDuringScheduling`) — the scheduler will still try to minimize skew, but won't refuse to schedule a pod if it can't achieve it.
**Why Topology Spread Constraints are generally the more precise, modern tool over pod anti-affinity for this specific "spread evenly" goal, worth stating explicitly in an interview:** anti-affinity is fundamentally a pairwise "avoid this other pod" rule, which can produce lopsided results at scale (it satisfies "not co-located" without any notion of overall balance); Topology Spread Constraints reason about the *whole* distribution across a topology domain directly, which is exactly the actual goal in almost every real-world "spread my replicas out" scenario. Both mechanisms can be used together — anti-affinity for hard co-location rules, spread constraints for balance.
**A consolidated comparison across every scheduling-influence mechanism covered so far, worth having as a single reference:**
| Mechanism | Governs | Hard or soft? | Typical use case |
|---|---|---|---|
| Taints/tolerations | Which pods may land on a node at all | Hard (unless `PreferNoSchedule`) | Reserve dedicated nodes for specific workloads |
| Node affinity | Pod's preference for node characteristics | Both (required/preferred variants) | "Run only on SSD-backed nodes" |
| Pod affinity | Co-locate with other specific pods | Both | "Run near my cache pod, same zone" |
| Pod anti-affinity | Avoid co-locating with other specific pods | Both | "Never share a node with my own replicas" |
| Topology spread constraints | Even balance across a topology domain | Both (`DoNotSchedule`/`ScheduleAnyway`) | "Keep replicas evenly spread across zones" |
| PriorityClass + preemption | Whether a pod can evict others to get scheduled | N/A — evaluated only when scheduling would otherwise fail | Guarantee scheduling for genuinely critical workloads under contention |
| ResourceQuota | Total namespace-wide resource consumption | Hard — enforced at admission, before scheduling is even reached | Prevent one team/namespace from consuming a shared cluster's entire capacity |
| Init containers | Whether main containers start at all | Hard — main containers never start until every init container succeeds | Block until a dependency is reachable, or run one-time setup |
| Native sidecar (`restartPolicy: Always`) | Whether a supporting container counts toward pod completion | Hard — excluded from the completion condition entirely | A log-shipper or proxy that shouldn't keep a Job's pod alive forever |
---
## Priority and Preemption
When cluster resources are genuinely scarce, **PriorityClass** lets some pods matter more than others — and a sufficiently high-priority pod that can't otherwise be scheduled can actually **evict** (preempt) lower-priority pods to make room for itself.
```mermaid
sequenceDiagram
participant HighPri as High-Priority Pod
participant Sched as Scheduler
participant LowPri as Low-Priority Pod (running)
HighPri->>Sched: Needs to be scheduled,
but cluster is FULL
Sched->>Sched: Checks if evicting lower-
priority pod(s) would
free enough room
Sched->>LowPri: Evicts (terminates) it
Sched->>HighPri: Schedules the
high-priority pod
in the freed space
```
```yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: business-critical
value: 1000000
globalDefault: false
description: "For business-critical workloads only"
```
**A genuinely important, practical caution worth naming explicitly:** preemption is powerful but should be used deliberately and sparingly — an over-broadly-applied high PriorityClass can cause unexpected, hard-to-predict evictions of other legitimate workloads across the cluster, so it's typically reserved for a small, carefully-chosen set of genuinely critical services.
---
## Preemption Mechanics and PodDisruptionBudgets
Preemption's interaction with **PodDisruptionBudgets (PDBs)** — a mechanism most commonly discussed alongside voluntary disruptions like node drains — deserves precise treatment, since it's a genuinely common point of confusion.
```mermaid
flowchart TD
Preempt["High-priority pod
triggers preemption
evaluation"] --> Candidates["Scheduler identifies
candidate lower-priority
pods to evict"]
Candidates --> PDBCheck{"Would evicting THIS
pod violate its
PodDisruptionBudget?"}
PDBCheck -->|Yes| TryNext["Scheduler tries a
DIFFERENT candidate that
wouldn't violate a PDB"]
PDBCheck -->|No PDB violated| Evict["Pod is evicted"]
```
**The precise, worth-knowing nuance: the scheduler's preemption logic tries to respect PDBs where possible, but PDB protection is NOT absolute against preemption the way it is against a voluntary node drain.** If evicting *any* combination of lower-priority pods that would satisfy the high-priority pod's needs would violate a PDB, the scheduler's default behavior still allows the preemption to proceed rather than leave the high-priority pod permanently unscheduled — preemption is treated as importance-driven eviction, not a fully PDB-gated voluntary disruption. **A strong, precise interview answer states this exact distinction:** "PDBs guarantee minimum availability during *voluntary* disruptions like node drains and cluster upgrades, but preemption for a genuinely higher-priority pod can still override that guarantee if no PDB-respecting eviction set exists — which is exactly why PriorityClass assignment needs real discipline, not liberal use."
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: checkout-service
```
**Graceful preemption, worth naming as a real mechanic, not an abrupt kill:** a preempted pod still receives its normal termination grace period (SIGTERM, then SIGKILL after `terminationGracePeriodSeconds` — the exact same graceful-shutdown mechanics from Part 1's kubelet discussion) — preemption is a targeted eviction decision, not a bypass of normal pod termination behavior.
---
## Deployments — The Workhorse
A **Deployment** is the standard, default way to run a stateless application — it manages a **ReplicaSet**, which in turn manages the actual Pods, giving you declarative scaling, self-healing, and rolling updates.
```mermaid
graph TD
Deployment["Deployment
(you edit THIS)"] --> RS["ReplicaSet
(created/managed
AUTOMATICALLY by the
Deployment)"]
RS --> Pod1["Pod"]
RS --> Pod2["Pod"]
RS --> Pod3["Pod"]
```
**Why the extra layer (ReplicaSet) exists, rather than the Deployment managing Pods directly — a genuinely insightful, less commonly known fact:** this is exactly what makes rolling updates and rollbacks possible. When you update a Deployment's pod template (e.g., a new image version), Kubernetes creates a **brand-new ReplicaSet** for the new version, while keeping the **old ReplicaSet** around (scaled to zero) — this is literally how `kubectl rollout undo` works: it just scales the old ReplicaSet back up and the new one back down, near-instantly, without needing to "remember" the old configuration some other way.
```bash
# See both the current AND previous ReplicaSets for a Deployment
kubectl get replicasets -l app=checkout-service
# Roll back to the previous version - literally just flips
# which ReplicaSet is scaled up
kubectl rollout undo deployment/checkout-service
```
---
## How a Rolling Update Actually Works
```yaml
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 # at most 1 pod can be DOWN during the rollout
maxSurge: 1 # at most 1 EXTRA pod above desired count
```
```mermaid
sequenceDiagram
participant Old as Old ReplicaSet (v1)
participant New as New ReplicaSet (v2)
Note over Old,New: Starting state: Old has 3
pods running, New has 0
New->>New: Create 1 new pod (v2)
(maxSurge allows 1 EXTRA)
Note over Old,New: Old: 3, New: 1 (4 total,
within maxSurge=1)
New->>New: Wait for new pod to
pass its READINESS probe
Old->>Old: Terminate 1 old pod (v1)
Note over Old,New: Old: 2, New: 1 (3 total)
New->>New: Create another new pod
Note over Old,New: Repeat until Old: 0, New: 3
```
**Why `maxUnavailable` and `maxSurge` are worth understanding as a genuine tradeoff, not just default settings to leave alone:** `maxSurge` controls how many *extra* resources you're willing to briefly consume during a rollout (faster rollout, more resource usage); `maxUnavailable` controls how much capacity you're willing to briefly lose (faster rollout, less safety margin). **Setting both to 0 would make a rollout impossible** — there'd be no way to ever have a new pod ready before removing an old one — which is a genuinely good, concrete detail to know if asked to explain the settings' interaction.
**Why readiness probes (from Part 1) are absolutely essential to a safe rollout, worth calling out explicitly:** the rollout only proceeds to terminate an old pod once a corresponding new pod passes its readiness probe — without a properly configured readiness probe, Kubernetes has no reliable way to know a new pod is actually ready to serve traffic, and could terminate old, working pods while new ones are still silently broken.
---
## StatefulSets — For Things That Need an Identity
A **StatefulSet** is for workloads that need something a plain Deployment deliberately doesn't provide: a **stable, predictable identity** and **stable storage** tied to that identity, across restarts and rescheduling.
```mermaid
graph TD
Deployment["Deployment pods:
checkout-7d9f8-x7k2p
checkout-7d9f8-m3n9q
(RANDOM names, no
guaranteed order, NO
stable identity)"] --> DeployNote["Fine for stateless apps —
any pod is interchangeable"]
StatefulSet["StatefulSet pods:
postgres-0
postgres-1
postgres-2
(PREDICTABLE, STABLE
names, created/deleted in
ORDER, each with its OWN
persistent storage)"] --> STSNote["Essential for databases,
where 'which specific
replica am I' and 'my own
data volume' genuinely
matter"]
```
**Why this connects directly to the stateful-vs-stateless scaling discussion from the Capacity Planning & Performance series:** StatefulSets exist specifically because stateful workloads can't be treated as interchangeable, randomly-named, disposable copies the way stateless Deployment pods can — `postgres-0` needs to reliably come back as `postgres-0`, with its *same* specific storage volume reattached, every single time it restarts or gets rescheduled, which is exactly the guarantee a StatefulSet (and Deployments deliberately do not) provides.
```bash
# StatefulSet pods get PREDICTABLE, stable DNS names too
# e.g. postgres-0.postgres-headless-svc.default.svc.cluster.local
```
**The ordinal index guarantee, worth stating precisely — it governs BOTH creation/deletion order AND scaling:** a StatefulSet with `replicas: 3` creates `postgres-0` first, waits for it to be Running and Ready, THEN creates `postgres-1`, then `postgres-2` — strictly sequential, never in parallel by default. Scaling down reverses this: the *highest*-ordinal pod (`postgres-2`) is removed first, never an arbitrary one. **Why this ordering matters concretely, a real distributed-systems reason, not just "for tidiness":** many stateful systems (etcd itself, from Part 1, is a real example) rely on a specific member joining/leaving order for safe cluster membership changes — removing a random member instead of the most-recently-added one could, for some systems, threaten quorum or replication consistency in ways a strictly ordinal, predictable removal order avoids.
---
## StatefulSet Update Strategies and Partitions
StatefulSets support the same `RollingUpdate` strategy concept as Deployments, but with StatefulSet-specific mechanics worth knowing precisely, since a database rollout going wrong is a genuinely higher-stakes mistake than a stateless service rollout going wrong.
```mermaid
sequenceDiagram
participant P2 as postgres-2 (highest ordinal)
participant P1 as postgres-1
participant P0 as postgres-0 (lowest ordinal)
Note over P2,P0: RollingUpdate always proceeds
in REVERSE ordinal order
P2->>P2: Updated FIRST
P2->>P2: Must become Ready before
proceeding to the next
P1->>P1: Updated second
P0->>P0: Updated LAST
```
**Why reverse-ordinal update order specifically, worth explaining as a deliberate design choice:** for many real stateful systems, `pod-0` is conventionally the first-created, often the "senior" or primary-adjacent member — updating from the highest ordinal downward means the most established, typically most cautious-to-disturb member is touched last, giving the update process the most opportunity to be caught and halted before it reaches the most critical instance.
```yaml
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 2 # only postgres-2 and higher get updated; 0 and 1 are left alone
```
**`partition` is a genuinely powerful, worth-knowing StatefulSet-specific safety mechanism:** setting `partition: 2` on a 3-replica StatefulSet means only `postgres-2` receives the update — `postgres-0` and `postgres-1` are left entirely untouched, even if the pod template changes. This enables a **manual canary pattern for stateful workloads**: update the partition to the highest ordinal only, verify the single updated replica behaves correctly under real conditions, then lower the partition value incrementally to roll the update out to the rest — a genuinely important pattern for anything where a bad rollout to a stateful workload (a corrupted migration, an incompatible on-disk format change) is far more costly and harder to reverse than a stateless rollback.
---
## DaemonSets — One Per Node
A **DaemonSet** ensures exactly one copy of a pod runs on every (or every matching) node in the cluster — automatically adding a copy when a new node joins, and removing it when a node leaves.
```mermaid
graph TD
DS["DaemonSet"] --> N1["Node 1: 1 pod"]
DS --> N2["Node 2: 1 pod"]
DS --> N3["Node 3: 1 pod"]
NewNode["New Node 4 joins
the cluster"] -.->|"automatically gets
1 pod too"| DS
```
**The classic, real-world use case, worth citing specifically:** log-shipping agents, monitoring agents (like a Prometheus Node Exporter), and networking/CNI components (covered in Part 3) — anything that genuinely needs to run on *every single node*, exactly once, rather than being scaled to an arbitrary replica count the way a Deployment would be.
---
## Jobs and CronJobs
For work that's meant to **run to completion and then stop**, rather than run continuously forever.
```mermaid
graph TD
Job["Job: runs a pod (or several)
until it SUCCEEDS, then stops
— for a ONE-TIME task"] --> JobEx["Example: a database
migration script"]
CronJob["CronJob: runs a Job on a
SCHEDULE (cron syntax)"] --> CronEx["Example: a nightly backup,
a report generated every
hour"]
```
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-backup
spec:
schedule: "0 2 * * *" # 2 AM every day, standard cron syntax
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
image: backup-tool:1.0
restartPolicy: OnFailure
```
**A genuinely important distinction worth naming: a Deployment's `restartPolicy` is always `Always` (it's meant to run forever), while a Job's is `OnFailure` or `Never` (it's meant to eventually stop, not be restarted indefinitely)** — using the wrong workload type entirely for a batch task (e.g., a Deployment for a one-time migration script) means Kubernetes will keep trying to "heal" a container that's supposed to have finished and exited, treating successful completion as if it were a crash.
---
## Job Completion Modes, Backoff, and Parallelism
Real Job usage goes well beyond "run one pod to completion" — worth knowing the fields that control retries, parallel execution, and completion tracking precisely.
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: batch-image-processing
spec:
completions: 10 # need 10 SUCCESSFUL pod completions total
parallelism: 3 # run up to 3 pods AT ONCE
backoffLimit: 4 # give up after 4 FAILED attempts (per pod slot)
activeDeadlineSeconds: 3600 # kill the whole Job if it's not done in 1 hour
completionMode: Indexed # each pod gets a unique index 0..9 (env var JOB_COMPLETION_INDEX)
template:
spec:
containers:
- name: worker
image: image-processor:1.0
restartPolicy: OnFailure
```
```mermaid
graph TD
Completions["completions: 10
TOTAL successful pod
completions needed"] --> Parallelism["parallelism: 3
up to 3 pods run
SIMULTANEOUSLY, cycling
through until 10
successes accumulate"]
Backoff["backoffLimit: 4
a FAILING pod retries with
EXPONENTIAL backoff, up to
4 times, before the Job is
marked Failed"] --> BackoffNote["Exponential backoff -
10s, 20s, 40s... - avoids
hammering a dependency
that's genuinely down"]
```
**`completionMode: Indexed`, worth knowing as the modern, more powerful mode — a real, common upgrade from the legacy `NonIndexed` default:** each pod in an Indexed Job gets a unique, predictable completion index (0 through `completions - 1`) injected as the `JOB_COMPLETION_INDEX` environment variable — this is exactly what makes **parallel, partitioned batch processing** clean to implement (e.g., "pod 3 processes shard 3 of the dataset") without needing an external work-queue coordination system just to hand out unique work assignments.
**Why `backoffLimit`'s exponential backoff matters concretely, connecting directly to the retry/backoff patterns in the Reliability & Architecture Patterns series:** a Job retrying a failing pod with a fixed, short interval against a genuinely down dependency (a database that's temporarily unreachable) would just hammer that dependency repeatedly — exponential backoff is the same general resilience pattern already covered for application-level retries, applied here at the Job-controller level automatically, with zero extra code needed in the workload itself.
---
## CronJob Concurrency Policy and Missed Schedules
CronJobs add scheduling on top of Jobs, and two fields genuinely matter for correctness, not just convenience.
```mermaid
graph TD
Concurrency["concurrencyPolicy"] --> Allow["Allow (default): multiple
runs CAN overlap if a
previous run is still
going when the next
scheduled time arrives"]
Concurrency --> Forbid["Forbid: skip the new run
entirely if the previous
one hasn't finished yet"]
Concurrency --> Replace["Replace: CANCEL the still-
running previous execution
and start the new one"]
```
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-backup
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec: {}
```
**Why `concurrencyPolicy: Allow` (the default) is a genuinely common, real production trap, worth stating explicitly:** a nightly backup job that occasionally runs long (say, database growth makes it take 90 minutes instead of the usual 30) can, under the default `Allow` policy, end up with a SECOND backup job starting before the first finishes — two backup processes running simultaneously against the same database is very often not just wasteful but can be genuinely harmful (resource contention, inconsistent concurrent snapshots). **`Forbid` is almost always the correct choice for any CronJob whose runs must never meaningfully overlap** (backups, database migrations, anything touching shared state), while `Replace` fits a workload where only the *latest* attempt genuinely matters (e.g., "refresh a cache" — an older, superseded run should just be abandoned in favor of the newest one).
**`startingDeadlineSeconds`, a real, worth-knowing edge case around missed schedules:** if the CronJob controller itself is down (a control plane issue, per Part 1's leader-election discussion) when a scheduled time passes, the run is "missed." Without `startingDeadlineSeconds` set, Kubernetes has a default tolerance (historically around 100 seconds) for how late a missed run can start and still be considered "on schedule" rather than skipped entirely — explicitly setting this field makes that tolerance a deliberate, documented decision rather than an implicit default a future on-call engineer has to go discover during an incident.
---
## Init Containers and Native Sidecar Containers
Two container-lifecycle mechanisms worth understanding precisely, since "just add another container" is not how either of them work — the ordering and lifecycle guarantees are the entire point.
```mermaid
sequenceDiagram
participant Init as Init Container(s)
participant App as Main App Container
participant Sidecar as Native Sidecar
Note over Init: Run SEQUENTIALLY,
ONE AT A TIME, each
must SUCCEED before
the next starts
Init->>Init: e.g. wait-for-database
Init->>Init: e.g. run-migrations
Note over Sidecar: Native sidecar STARTS
before the main container
(as of Kubernetes 1.29)
Sidecar->>App: Ready before app starts
App->>App: Main container runs
Note over Sidecar: Native sidecar STOPS
AFTER the main container
terminates
```
```yaml
spec:
initContainers:
- name: wait-for-db
image: busybox
command: ['sh', '-c', 'until nc -z db-host 5432; do sleep 2; done']
containers:
- name: log-shipper
image: fluent-bit
restartPolicy: Always # THIS is what makes it a "native sidecar" (1.29+)
- name: app
image: checkout:1.2.3
```
**Init containers, the precise mental model worth stating explicitly:** they run to completion, strictly sequentially, before ANY main container starts — a genuinely common real use case is exactly what's shown above (block until a dependency is reachable) or running a one-time setup/migration step that the main container assumes has already happened. If an init container fails, the kubelet retries it (respecting the pod's `restartPolicy`) — the main containers never start until every init container has succeeded.
**Native sidecar containers (`restartPolicy: Always` set on a specific container within `containers`, not `initContainers`), a genuinely important, relatively recent (Kubernetes 1.29 stable) improvement worth knowing precisely:** before this feature, a "sidecar" was just a convention (a log-shipper or service-mesh proxy container alongside the main app), with a real, concrete problem — a Job's sidecar container that never exits on its own would keep the whole Pod "Running" forever, since Kubernetes had no way to know it was a supporting sidecar versus a co-equal main container. **Native sidecars solve this precisely**: marking a container with `restartPolicy: Always` inside the pod spec makes Kubernetes treat it as a genuine sidecar — it starts *before* the main containers, and critically, **the pod is considered complete once the main containers finish, regardless of whether the sidecar is still running** — directly fixing the "Job never completes because of a lingering sidecar" problem that plagued the convention-only approach for years.
---
## Choosing the Right Workload Object
```mermaid
flowchart TD
Start{"What kind of workload
is this?"} --> Q1{"Runs continuously,
no persistent identity
needed?"}
Q1 -->|Yes| Deploy["Deployment"]
Q1 -->|No| Q2{"Runs continuously, but
needs stable identity
+ its OWN storage?"}
Q2 -->|Yes| STS["StatefulSet"]
Q2 -->|No| Q3{"Needs to run on
EVERY node, exactly
once?"}
Q3 -->|Yes| DS["DaemonSet"]
Q3 -->|No| Q4{"Runs to completion,
one-time task?"}
Q4 -->|Yes| Job["Job"]
Q4 -->|No| Q5{"Runs to completion,
on a SCHEDULE?"}
Q5 -->|Yes| CronJob["CronJob"]
```
---
## Part 2 CLI Cheat Sheet
```bash
# Scheduling diagnostics
kubectl describe pod | grep -A 20 Events # WHY didn't this pod schedule?
kubectl get events --field-selector reason=FailedScheduling
kubectl get pods -o wide --sort-by='.spec.nodeName' # see actual placement
# Taints and tolerations
kubectl taint nodes key=value:NoSchedule
kubectl taint nodes key=value:NoSchedule- # remove a taint
kubectl describe node | grep Taints
# Priority classes
kubectl get priorityclasses
kubectl get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority
# Deployments and rollouts
kubectl rollout status deployment/
kubectl rollout history deployment/
kubectl rollout undo deployment/ --to-revision=2
kubectl scale deployment/ --replicas=5
# StatefulSets
kubectl get statefulset -o jsonpath='{.spec.updateStrategy.rollingUpdate.partition}'
kubectl patch statefulset -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":1}}}}'
# Jobs and CronJobs
kubectl create job manual-run --from=cronjob/ # trigger a CronJob manually, right now
kubectl get jobs --watch
kubectl logs job/
```
---
## Common Mistakes
| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Not setting resource requests at all | The pod gets BestEffort QoS — first to be evicted under any node memory pressure, and the Scheduler has no real basis for placement decisions | Always set resource requests, sized from real observed usage |
| Assuming `replicas: 3` guarantees availability across node failures | Says nothing about WHERE the 3 pods end up — without anti-affinity, they can all land on the same node | Add pod anti-affinity for genuinely critical, multi-replica workloads |
| Using a Deployment for a one-time batch task | Kubernetes tries to "heal" a container that successfully finished and exited, treating completion like a crash | Use a Job (or CronJob for scheduled tasks) instead |
| Assuming node affinity rules are continuously enforced | `...IgnoredDuringExecution` means the rule is only checked at scheduling time — a pod isn't evicted if the node's labels change later | Understand affinity as a scheduling-time decision, not an ongoing guarantee |
| Confusing requests with limits | Requests drive scheduling placement; limits drive runtime enforcement (throttling/OOM-kill) — very different mechanisms | Set both deliberately, understanding what each one actually controls |
| Applying a high PriorityClass too broadly | Causes unpredictable, hard-to-anticipate evictions of other legitimate workloads across the cluster | Reserve high-priority preemption for a small, carefully-chosen set of genuinely critical services |
| Relying on pod anti-affinity alone for "spread evenly across zones" | Anti-affinity only guarantees no exact co-location, not overall balance — can still produce a lopsided distribution | Use Topology Spread Constraints with an appropriate `maxSkew` for genuine balance requirements |
| Assuming a PodDisruptionBudget fully protects against preemption | PDBs guarantee minimum availability during voluntary disruptions, but preemption for a higher-priority pod can still override that guarantee | Assign PriorityClasses with real discipline — don't assume PDBs are an absolute backstop against preemption |
| Leaving a CronJob's `concurrencyPolicy` at the default `Allow` for a job whose runs must never overlap | An occasional slow run can result in two instances running simultaneously against shared state (e.g. two concurrent backups) | Set `Forbid` (or `Replace`, depending on intent) explicitly for any CronJob where overlapping runs would cause real harm |
| Confusing a ResourceQuota rejection with a scheduling failure during troubleshooting | A quota rejection happens at admission time — the pod object is never even created, so `kubectl describe pod` shows nothing at all | Check `kubectl describe resourcequota -n ` and recent Events when a pod seems to have simply vanished instead of appearing as Pending |
| Adding a log-shipper "sidecar" to a Job without marking it a native sidecar (`restartPolicy: Always`) | A sidecar that never exits on its own keeps the whole Job's pod Running forever, since Kubernetes can't distinguish it from a co-equal main container | Mark supporting sidecar containers with `restartPolicy: Always` (Kubernetes 1.29+) so pod completion is judged only by the main containers |
| Enumerating every possible label value with `In` when `Exists` would do | Verbose, brittle affinity rules that break the moment a new valid value is introduced | Use `Exists`/`DoesNotExist` when only the presence of a label key matters, not its specific value |
---
## Worked Practice Problems
**Problem 1:** A node runs low on memory and Kubernetes needs to evict something. Three pods are running on it: Pod A (no requests/limits set), Pod B (requests=limits), Pod C (requests set, limits higher than requests). In what order are they likely evicted, and why?
*Answer:* Pod A (BestEffort QoS — no requests or limits at all) is evicted first, since it has no guaranteed resources at all and the lowest eviction priority. Pod C (Burstable QoS — requests set, but limits higher, meaning it's allowed to burst beyond its guarantee) is evicted next if pressure continues. Pod B (Guaranteed QoS — requests exactly equal limits) is evicted last, since it has the strongest resource guarantee and highest survival priority under node pressure.
**Problem 2:** A team runs 3 replicas of a critical service via a Deployment with no pod anti-affinity configured. During a node failure, all 3 replicas go down simultaneously, causing a full outage despite having "3 replicas." What went wrong, and how would you prevent it going forward?
*Answer:* Without pod anti-affinity, the Scheduler had no rule preventing it from placing all 3 replicas on the same node — which is exactly what happened, silently creating a single point of failure that `kubectl get pods` would have shown as healthy (3/3 Running) right up until that one node failed. Fix: add a `podAntiAffinity` rule (using `topologyKey: kubernetes.io/hostname`, or even better, `topology.kubernetes.io/zone` for multi-AZ spread) requiring replicas of this Deployment to avoid scheduling on the same node/zone as each other, ensuring a single node or zone failure can never take down every replica simultaneously.
**Problem 3:** A one-time database migration script is deployed as a Kubernetes Deployment. After it finishes successfully and the container exits, Kubernetes keeps restarting it in a loop. What's the root cause, and what's the correct fix?
*Answer:* A Deployment's pods always use `restartPolicy: Always`, since Deployments are designed for workloads meant to run forever — Kubernetes has no way to know this particular container's successful exit was actually the *intended*, successful end state rather than a crash, so it keeps restarting it exactly as designed for a continuously-running service. The correct fix is using a Job instead of a Deployment for this one-time task — a Job's semantics (`restartPolicy: OnFailure` or `Never`, and tracking actual completion) correctly understand that a successful exit means the work is done, not that something crashed.
**Problem 4:** A team needs to roll out a risky schema-migrating update to a 5-replica StatefulSet running a database, and wants to verify the change on exactly one replica before touching the rest. What StatefulSet-specific mechanism enables this, and how would they use it?
*Answer:* The `partition` field on the StatefulSet's `rollingUpdate` strategy. Setting `partition: 4` on a 5-replica (ordinals 0-4) StatefulSet means only the highest-ordinal pod, `postgres-4`, receives the updated pod template — ordinals 0 through 3 are left completely untouched. The team can verify `postgres-4` behaves correctly under real traffic/data conditions, then lower the partition value incrementally (4, then 3, then 2...) to roll the update out to the rest of the replicas one at a time, with the ability to halt the rollout at any point by simply not lowering the partition further — a genuinely safer, manually-gated canary pattern specifically suited to stateful workloads where a bad rollout is far more costly than a stateless service's.
**Problem 5:** A nightly CronJob backup occasionally runs long due to database growth, and the team discovers two backup processes briefly ran concurrently against the same database, causing resource contention and an inconsistent snapshot. What field was misconfigured, and what's the fix?
*Answer:* The CronJob's `concurrencyPolicy` was left at its default value, `Allow`, which permits a new scheduled run to start even while a previous run is still executing. The fix is setting `concurrencyPolicy: Forbid` explicitly — this makes the CronJob controller skip starting a new run entirely if the previous execution hasn't completed yet, guaranteeing backups never overlap. (`Replace`, which cancels the in-progress run and starts fresh, would be the wrong choice here specifically because an in-progress backup being killed mid-write is arguably worse than simply skipping a run — the correct policy depends on which failure mode is actually more acceptable for the specific workload.)
**Problem 6:** A batch-processing Job includes a log-shipping container alongside the main worker container. The worker finishes its work and exits successfully, but the Job never shows as Complete — `kubectl get jobs` shows it stuck indefinitely. What's the root cause, and what's the fix?
*Answer:* The log-shipping container was added as an ordinary container without being marked as a native sidecar, and it has no natural exit condition of its own (it just keeps running, waiting for logs to ship) — Kubernetes has no way to distinguish "a co-equal main container that's supposed to keep running" from "a supporting sidecar that should be cleaned up once the real work is done," so the Pod (and therefore the Job) never reaches a Complete state as long as ANY container is still running. The fix: set `restartPolicy: Always` on the log-shipping container specifically (Kubernetes 1.29+'s native sidecar feature) — this tells Kubernetes explicitly that Job/Pod completion should be judged by the main worker container alone, and the sidecar will be automatically terminated once the main container exits, rather than keeping the whole Pod alive indefinitely.
---
## Summary and What's Next
- Scheduling happens in two phases: **filtering** (which nodes are even capable of running this pod) and **scoring** (of those, which is the best choice) — a node failing even one filter is completely disqualified, not just deprioritized.
- **Resource requests drive scheduling placement; limits drive runtime enforcement** (CPU throttling, memory OOM-kill) — two separate mechanisms answering two different questions, and together they determine a pod's **QoS class** (Guaranteed > Burstable > BestEffort), which directly determines eviction order under node pressure.
- **Taints** repel pods from a node unless they carry a matching **toleration**; **node affinity** is the pod-side preference/requirement for specific node characteristics — and affinity rules are checked only at scheduling time, not continuously enforced afterward.
- **Pod anti-affinity** is essential for real multi-replica availability — without it, a Deployment's replica count says nothing about whether they're actually spread across different nodes/zones.
- **PriorityClass and preemption** let critical pods evict lower-priority ones when the cluster is genuinely full — powerful, and worth applying deliberately and narrowly.
- **Deployments** manage a **ReplicaSet** (which manages the actual Pods) — this extra layer is exactly what makes rolling updates and near-instant rollbacks possible, by keeping the old ReplicaSet around, scaled to zero, rather than needing to reconstruct the previous configuration.
- **`maxSurge`/`maxUnavailable`** control the real tradeoff between rollout speed/resource usage and safety margin during a rolling update — and readiness probes are what make the whole rollout process safe in the first place.
- **StatefulSets** provide stable identity and storage for workloads (like databases) that genuinely need it, create/delete pods in strict ordinal order, and support a `partition`-based manual canary pattern for high-stakes rollouts; **DaemonSets** run exactly one pod per node; **Jobs/CronJobs** are for work meant to run to completion, not forever — using the wrong workload type for a given need causes real, specific, predictable problems.
- The **Scheduler Framework**'s pluggable extension points (`PreFilter`, `Filter`, `Score`, `Permit`, `Bind`, and more) are what make specialized scheduling — like gang scheduling for ML/batch training — possible without forking the scheduler itself.
- **Topology Spread Constraints** are the more precise modern tool for genuine even-distribution goals, complementing (not replacing) pod anti-affinity's pairwise co-location rules.
- **Preemption can override a PodDisruptionBudget's guarantee** — PDBs fully protect against voluntary disruptions like drains, but not necessarily against a genuinely higher-priority pod needing the space.
- **Job `completionMode: Indexed`** gives each pod a predictable completion index, enabling clean partitioned parallel batch processing without external coordination; **CronJob `concurrencyPolicy`** (default `Allow`) is a real, common source of harmful overlapping runs if left unconsidered for workloads that must never run concurrently.
- **ResourceQuota** rejects at admission time, before a pod object even exists; **LimitRange** supplies sensible per-container defaults so an incomplete manifest doesn't silently end up BestEffort — two distinct, complementary namespace-level governance mechanisms, easy to confuse with a plain scheduling failure during troubleshooting.
- **Init containers** run sequentially to completion before any main container starts; **native sidecar containers** (`restartPolicy: Always`, Kubernetes 1.29+) start before the main containers and are excluded from the pod's completion condition — directly fixing the long-standing "lingering sidecar keeps a Job running forever" problem.
**Continue to Part 3** (`03-networking-and-storage.md`) to see how pods actually talk to each other and the outside world (the CNI networking model, Services, Ingress) and how they get durable storage (the CSI model, PersistentVolumes).