Scheduling & Workload Objects

Table of Contents#

1. Recap: What the Scheduler's Job Actually Is
2. The Two-Phase Scheduling Process
3. Filtering — Ruling Out Impossible Nodes
4. Scoring — Ranking the Possible Nodes
5. Resource Requests and Limits — Why They Drive Scheduling
6. Taints and Tolerations
7. Node Affinity and Anti-Affinity
8. Pod Affinity and Anti-Affinity
9. Priority and Preemption
10. Deployments — The Workhorse
11. How a Rolling Update Actually Works
12. StatefulSets — For Things That Need an Identity
13. DaemonSets — One Per Node
14. Jobs and CronJobs
15. Choosing the Right Workload Object
16. Common Mistakes
17. Worked Practice Problems
18. Summary and What's Next

---

Recap: What the Scheduler's Job Actually Is#

Part 1 introduced the Scheduler at a high level: when a new Pod exists with no node assigned, the Scheduler decides which node it should run on. This part goes deep into exactly how that decision gets made, and then covers the higher-level objects (Deployments, StatefulSets, and more) that actually create pods in the first place.

---

The Two-Phase Scheduling Process#

Every scheduling decision happens in two distinct phases, and knowing both by name is a very commonly expected piece of Kubernetes interview knowledge.

Diagram

Simple analogy: think of hiring for a job. Filtering is the initial resume screen — ruling out anyone who genuinely doesn't meet the minimum requirements (no relevant degree, wrong location). Scoring is the interview process that ranks the remaining, qualified candidates against each other to find the best fit. A candidate has to survive filtering before scoring is even relevant.

---

Filtering — Ruling Out Impossible Nodes#

Filtering asks, for every node in the cluster: "could this pod even physically/logically run here at all?"

Diagram

A node that fails even ONE filter is completely removed from consideration — it's not "less preferred," it's genuinely disqualified.

---

Scoring — Ranking the Possible Nodes#

Of the nodes that survive filtering, scoring ranks them by a weighted combination of factors, and the Scheduler picks the highest-scoring one (with some randomization among close ties, to avoid always overloading the exact same "best" node).

Diagram

---

Resource Requests and Limits — Why They Drive Scheduling#

This deserves its own dedicated section because it's genuinely one of the most operationally important, most commonly misunderstood settings in all of Kubernetes.

Copy
resources:
  requests:
    cpu: "500m"        # 0.5 CPU cores
    memory: "512Mi"
  limits:
    cpu: "1000m"        # 1 full CPU core
    memory: "1Gi"

Diagram

The critical, frequently-tested distinction: requests drive scheduling; limits drive runtime enforcement. These are two completely separate mechanisms answering two completely different questions — "where should this run" (requests) versus "how much can it actually consume once it's running" (limits).

Quality of Service (QoS) Classes — A Direct Consequence#

Kubernetes automatically assigns every pod one of three QoS classes, purely based on how requests and limits are set — and this classification directly determines which pods get evicted first under real node pressure.

Diagram

Why this matters practically, and it's a real, specific interview question: "which pod gets killed first if a node runs low on memory?" — the answer is always BestEffort pods first, then Burstable, and Guaranteed pods last, which is exactly why genuinely critical workloads (like a payments service) should always be configured with requests == limits (Guaranteed QoS), to maximize their survival priority during real node-level resource pressure.

---

Taints and Tolerations#

A taint is applied to a node, and repels pods from scheduling there — unless the pod explicitly has a matching toleration.

Diagram

Simple analogy: a taint is like a "staff only" sign on a door — by default it repels everyone, and only someone with the specific matching key (a toleration) is allowed through. Note it's opt-in for the pod, not opt-in for the node — the node is closed by default to anything without an explicit toleration.

Copy
# Taint a node to reserve it for a specific workload
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule

Copy
# A pod that tolerates this specific taint, allowing it to schedule there
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

The three taint effects worth knowing: NoSchedule (won't be scheduled here, but existing pods without the toleration aren't evicted), PreferNoSchedule (a soft version — try to avoid, but not a hard rule), and NoExecute (won't be scheduled AND existing pods without the toleration get actively evicted).

---

Node Affinity and Anti-Affinity#

While taints are node-side "keep out unless invited," node affinity is pod-side "I specifically want to run on nodes matching these characteristics."

Diagram

Copy
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "disktype"
              operator: In
              values: ["ssd"]
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: "zone"
              operator: In
              values: ["us-east-1a"]

Reading the verbose field name ...IgnoredDuringExecution correctly, a genuinely important, often-missed detail: this means the rule is only checked at scheduling time — if a node's labels change after a pod is already running there (e.g., a label is removed), Kubernetes will NOT retroactively evict that pod, even though it would no longer satisfy the affinity rule if re-evaluated. Affinity rules are a scheduling-time decision, not a continuously-enforced invariant.

---

Pod Affinity and Anti-Affinity#

Similar mechanism, but the rule is based on other pods already running, not node labels — this is genuinely important for real availability design.

Diagram

Copy
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values: ["checkout-service"]
        topologyKey: "kubernetes.io/hostname"

Why this directly connects to the Reliability & Architecture Patterns series' redundancy discussion: having replicas: 3 in a Deployment guarantees 3 pods exist, but says absolutely nothing about where they end up — without pod anti-affinity, the Scheduler could (and under real conditions, sometimes will) place all 3 on the exact same node, silently creating a single point of failure that looks fine on paper (kubectl get pods shows "3/3 Running") but provides zero actual redundancy against a node failure.

---

Priority and Preemption#

When cluster resources are genuinely scarce, PriorityClass lets some pods matter more than others — and a sufficiently high-priority pod that can't otherwise be scheduled can actually evict (preempt) lower-priority pods to make room for itself.

Diagram

Copy
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 1000000
globalDefault: false
description: "For business-critical workloads only"

A genuinely important, practical caution worth naming explicitly: preemption is powerful but should be used deliberately and sparingly — an over-broadly-applied high PriorityClass can cause unexpected, hard-to-predict evictions of other legitimate workloads across the cluster, so it's typically reserved for a small, carefully-chosen set of genuinely critical services.

---

Deployments — The Workhorse#

A Deployment is the standard, default way to run a stateless application — it manages a ReplicaSet, which in turn manages the actual Pods, giving you declarative scaling, self-healing, and rolling updates.

Diagram

Why the extra layer (ReplicaSet) exists, rather than the Deployment managing Pods directly — a genuinely insightful, less commonly known fact: this is exactly what makes rolling updates and rollbacks possible. When you update a Deployment's pod template (e.g., a new image version), Kubernetes creates a brand-new ReplicaSet for the new version, while keeping the old ReplicaSet around (scaled to zero) — this is literally how kubectl rollout undo works: it just scales the old ReplicaSet back up and the new one back down, near-instantly, without needing to "remember" the old configuration some other way.

Copy
# See both the current AND previous ReplicaSets for a Deployment
kubectl get replicasets -l app=checkout-service

# Roll back to the previous version - literally just flips
# which ReplicaSet is scaled up
kubectl rollout undo deployment/checkout-service

---

How a Rolling Update Actually Works#

Copy
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # at most 1 pod can be DOWN during the rollout
      maxSurge: 1            # at most 1 EXTRA pod above desired count

Diagram

Why maxUnavailable and maxSurge are worth understanding as a genuine tradeoff, not just default settings to leave alone: maxSurge controls how many extra resources you're willing to briefly consume during a rollout (faster rollout, more resource usage); maxUnavailable controls how much capacity you're willing to briefly lose (faster rollout, less safety margin). Setting both to 0 would make a rollout impossible — there'd be no way to ever have a new pod ready before removing an old one — which is a genuinely good, concrete detail to know if asked to explain the settings' interaction.

Why readiness probes (from Part 1) are absolutely essential to a safe rollout, worth calling out explicitly: the rollout only proceeds to terminate an old pod once a corresponding new pod passes its readiness probe — without a properly configured readiness probe, Kubernetes has no reliable way to know a new pod is actually ready to serve traffic, and could terminate old, working pods while new ones are still silently broken.

---

StatefulSets — For Things That Need an Identity#

A StatefulSet is for workloads that need something a plain Deployment deliberately doesn't provide: a stable, predictable identity and stable storage tied to that identity, across restarts and rescheduling.

Diagram

Why this connects directly to the stateful-vs-stateless scaling discussion from the Capacity Planning & Performance series: StatefulSets exist specifically because stateful workloads can't be treated as interchangeable, randomly-named, disposable copies the way stateless Deployment pods can — postgres-0 needs to reliably come back as postgres-0, with its same specific storage volume reattached, every single time it restarts or gets rescheduled, which is exactly the guarantee a StatefulSet (and Deployments deliberately do not) provides.

Copy
# StatefulSet pods get PREDICTABLE, stable DNS names too
# e.g. postgres-0.postgres-headless-svc.default.svc.cluster.local

---

DaemonSets — One Per Node#

A DaemonSet ensures exactly one copy of a pod runs on every (or every matching) node in the cluster — automatically adding a copy when a new node joins, and removing it when a node leaves.

Diagram

The classic, real-world use case, worth citing specifically: log-shipping agents, monitoring agents (like a Prometheus Node Exporter), and networking/CNI components (covered in Part 3) — anything that genuinely needs to run on every single node, exactly once, rather than being scaled to an arbitrary replica count the way a Deployment would be.

---

Jobs and CronJobs#

For work that's meant to run to completion and then stop, rather than run continuously forever.

Diagram

Copy
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"       # 2 AM every day, standard cron syntax
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: backup-tool:1.0
          restartPolicy: OnFailure

A genuinely important distinction worth naming: a Deployment's restartPolicy is always Always (it's meant to run forever), while a Job's is OnFailure or Never (it's meant to eventually stop, not be restarted indefinitely) — using the wrong workload type entirely for a batch task (e.g., a Deployment for a one-time migration script) means Kubernetes will keep trying to "heal" a container that's supposed to have finished and exited, treating successful completion as if it were a crash.

---

Choosing the Right Workload Object#

Diagram

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A node runs low on memory and Kubernetes needs to evict something. Three pods are running on it: Pod A (no requests/limits set), Pod B (requests=limits), Pod C (requests set, limits higher than requests). In what order are they likely evicted, and why?

Answer: Pod A (BestEffort QoS — no requests or limits at all) is evicted first, since it has no guaranteed resources at all and the lowest eviction priority. Pod C (Burstable QoS — requests set, but limits higher, meaning it's allowed to burst beyond its guarantee) is evicted next if pressure continues. Pod B (Guaranteed QoS — requests exactly equal limits) is evicted last, since it has the strongest resource guarantee and highest survival priority under node pressure.

Problem 2: A team runs 3 replicas of a critical service via a Deployment with no pod anti-affinity configured. During a node failure, all 3 replicas go down simultaneously, causing a full outage despite having "3 replicas." What went wrong, and how would you prevent it going forward?

Answer: Without pod anti-affinity, the Scheduler had no rule preventing it from placing all 3 replicas on the same node — which is exactly what happened, silently creating a single point of failure that kubectl get pods would have shown as healthy (3/3 Running) right up until that one node failed. Fix: add a podAntiAffinity rule (using topologyKey: kubernetes.io/hostname, or even better, topology.kubernetes.io/zone for multi-AZ spread) requiring replicas of this Deployment to avoid scheduling on the same node/zone as each other, ensuring a single node or zone failure can never take down every replica simultaneously.

Problem 3: A one-time database migration script is deployed as a Kubernetes Deployment. After it finishes successfully and the container exits, Kubernetes keeps restarting it in a loop. What's the root cause, and what's the correct fix?

Answer: A Deployment's pods always use restartPolicy: Always, since Deployments are designed for workloads meant to run forever — Kubernetes has no way to know this particular container's successful exit was actually the intended, successful end state rather than a crash, so it keeps restarting it exactly as designed for a continuously-running service. The correct fix is using a Job instead of a Deployment for this one-time task — a Job's semantics (restartPolicy: OnFailure or Never, and tracking actual completion) correctly understand that a successful exit means the work is done, not that something crashed.

---

Summary and What's Next#

• Scheduling happens in two phases: filtering (which nodes are even capable of running this pod) and scoring (of those, which is the best choice) — a node failing even one filter is completely disqualified, not just deprioritized.
• Resource requests drive scheduling placement; limits drive runtime enforcement (CPU throttling, memory OOM-kill) — two separate mechanisms answering two different questions, and together they determine a pod's QoS class (Guaranteed > Burstable > BestEffort), which directly determines eviction order under node pressure.
• Taints repel pods from a node unless they carry a matching toleration; node affinity is the pod-side preference/requirement for specific node characteristics — and affinity rules are checked only at scheduling time, not continuously enforced afterward.
• Pod anti-affinity is essential for real multi-replica availability — without it, a Deployment's replica count says nothing about whether they're actually spread across different nodes/zones.
• PriorityClass and preemption let critical pods evict lower-priority ones when the cluster is genuinely full — powerful, and worth applying deliberately and narrowly.
• Deployments manage a ReplicaSet (which manages the actual Pods) — this extra layer is exactly what makes rolling updates and near-instant rollbacks possible, by keeping the old ReplicaSet around, scaled to zero, rather than needing to reconstruct the previous configuration.
• maxSurge/maxUnavailable control the real tradeoff between rollout speed/resource usage and safety margin during a rolling update — and readiness probes are what make the whole rollout process safe in the first place.
• StatefulSets provide stable identity and storage for workloads (like databases) that genuinely need it; DaemonSets run exactly one pod per node; Jobs/CronJobs are for work meant to run to completion, not forever — using the wrong workload type for a given need causes real, specific, predictable problems.

Continue to Part 3 (03-networking-and-storage.md) to see how pods actually talk to each other and the outside world (the CNI networking model, Services, Ingress) and how they get durable storage (the CSI model, PersistentVolumes).