Verified10 commandsAI-assisted

Cluster Administration & Advanced Usage

Verified against kubectl v1.34.0 (client), flags verified via `kubectl cordon/drain/taint/patch/replace/wait/explain/get/completion --help` run locally, 2026-08-29 · official docs

Node maintenance, low-level resource surgery (patch/replace/wait), self-documenting the API with explain, scripting-friendly output formats, and the productivity tooling (completion, plugins) that turns kubectl from "a command you look up every time" into muscle memory. 🎯

Node maintenance: cordon, drain, uncordon#

kubectl cordon my-node                          # mark unschedulable — no NEW pods will land here
kubectl drain my-node --ignore-daemonsets        # evict existing pods too, in preparation for real maintenance
kubectl drain my-node --ignore-daemonsets --delete-emptydir-data   # also allow evicting pods using emptyDir (their local data is lost)
kubectl uncordon my-node                         # mark schedulable again, once maintenance is done

Diagram

Important

cordon alone stops new pods from scheduling but does NOT move existing ones off the node. drain (which cordons and evicts) is the command that actually clears a node for real maintenance — running only cordon before, say, an OS patch reboot leaves every pod already on that node to go down ungracefully when it reboots.

Warning

drain refuses to proceed on pods it doesn't recognize an owning controller for (bare pods with no Deployment/ReplicaSet/Job/DaemonSet/StatefulSet behind them) unless --force is passed — this is a deliberate safety rail, since a bare pod deleted this way has nothing to recreate it. Confirm that's really what you want before reaching for --force on an unfamiliar cluster.

Taints and tolerations — repelling pods from a node#

kubectl taint nodes my-node dedicated=gpu-workloads:NoSchedule    # new pods without a matching toleration won't schedule here
kubectl taint nodes my-node dedicated=gpu-workloads:NoExecute      # also evicts pods ALREADY running here that lack the toleration
kubectl taint nodes -l node-role=edge dedicated=edge:PreferNoSchedule   # apply to every node matching a label selector
kubectl taint nodes my-node dedicated:NoSchedule-                   # remove a taint (trailing "-")

Effect	What it does
`NoSchedule`	New pods without a matching toleration won't be scheduled here; existing pods are unaffected
`PreferNoSchedule`	The scheduler tries to avoid this node for non-tolerating pods, but it's a soft preference, not a hard rule
`NoExecute`	New pods won't schedule, AND existing non-tolerating pods already here get evicted

Tip

Taints are how you dedicate nodes to a specific workload class (GPU nodes, a specific team, a specific compliance boundary) without a custom scheduler. Pair a taint with a matching nodeSelector/node affinity on the workloads that should land there — the taint alone only repels the wrong pods, it doesn't attract the right ones.

Patching a resource without a full YAML round-trip#

kubectl patch node my-node -p '{"spec":{"unschedulable":true}}'                  # strategic merge patch (default)
kubectl patch deployment my-app -p '{"spec":{"replicas":5}}'
kubectl patch pod my-pod --type=json -p='[{"op":"replace","path":"/spec/containers/0/image","value":"myapp:v2"}]'   # JSON patch, positional
kubectl patch deployment my-app --subresource=scale --type=merge -p '{"spec":{"replicas":2}}'   # patch a subresource specifically

`--type`	Shape	When to use it
`strategic` (default)	Kubernetes-aware merge, understands list merge keys (e.g. matches containers by `name`)	Most day-to-day patches against built-in resource kinds
`merge`	Plain RFC 7386 JSON merge patch	Custom resources, where strategic merge isn't supported
`json`	RFC 6902 JSON Patch — explicit `op`/`path`/`value` operations	Precise array manipulation (insert/remove at a specific index) that a merge patch can't express

Note

patch is for adjusting a live object in place without a full manifest; apply is for converging an object to match a whole manifest file. They overlap (both can change a field), but patch is the right reach for a quick, scripted, single-field change — e.g. an incident-response toggle — where writing/tracking a whole YAML file would be overkill.

Replacing a resource wholesale#

kubectl get pod my-pod -o yaml > pod.yaml    # capture the full current spec first
# edit pod.yaml...
kubectl replace -f pod.yaml                   # the ENTIRE spec must be provided — this isn't a partial update
kubectl replace --force -f pod.yaml            # force: delete and recreate, for fields that can't be updated in place

Warning

Unlike patch, replace requires the complete resource spec — any field you omit from the file gets reset to its default, not left alone. Always get -o yaml first and edit that, rather than hand-writing a partial manifest and expecting replace to merge it.

Waiting for a condition#

kubectl wait --for=condition=Ready pod/my-pod --timeout=60s
kubectl wait --for=condition=Available deployment/my-app --timeout=120s
kubectl wait --for=delete pod/my-pod --timeout=60s                    # wait for something to finish being deleted
kubectl wait --for=create secret/my-secret --timeout=30s               # wait for something to come into existence
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/my-pod       # wait on an arbitrary field value, not just a named condition

Tip

kubectl wait is the scriptable alternative to polling kubectl get in a loop. It's the natural fit for a CI/CD pipeline step — "deploy, then block until the rollout is actually Available before running smoke tests" — without hand-rolling a while ! kubectl get ... ; do sleep 2; done loop.

Self-documenting the API with `explain`#

kubectl explain pod                              # top-level fields on a Pod
kubectl explain pod.spec.containers               # drill into a nested field
kubectl explain deployment.spec.strategy --recursive   # every field under this path, all the way down

Tip

kubectl explain reads live from the connected cluster's OpenAPI schema, not a static bundled doc — it's the fastest way to check the exact field name/type/valid-values for a resource on the actual Kubernetes version you're running, which matters because fields do get added/deprecated/renamed across minor versions. Reach for this before guessing a field name from memory or an outdated blog post.

Output formatting for scripting#

kubectl get pods -o jsonpath='{.items[*].metadata.name}'                       # space-separated pod names
kubectl get pod my-pod -o jsonpath='{.status.phase}'                            # a single field's value
kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase     # a lightweight custom table
kubectl get pods --sort-by='{.status.startTime}'                                 # sort a list by an arbitrary field
kubectl get pod my-pod -o go-template='{{.spec.nodeName}}{{"\n"}}'                # full Go template power when jsonpath isn't enough
kubectl get pods -o name                                                          # just resource/name pairs — handy for piping into xargs

Note

jsonpath is usually enough and easier to write inline; reach for go-template/go-template-file only when you need actual logic (conditionals, loops, custom formatting functions) that JSONPath's query syntax can't express.

Shell completion#

kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl
source <(kubectl completion bash)                # per-session, no file needed
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -o default -F __start_kubectl k' >> ~/.bashrc   # make completion work for the `k` alias too, not just `kubectl`

Tip

The alias-completion snippet above is the single highest-value five-second setup step for daily kubectl use. k get po, k describe deploy, k logs -f with full tab-completion on resource names is a real, compounding time saver across a normal day of cluster work — most experienced operators never type kubectl in full.

Real-world scenario: safely draining a node for a Kubernetes version upgrade#

A rolling node-pool upgrade needs each node emptied of workloads before it's replaced, without an outage for anything running a PodDisruptionBudget-protected workload:

kubectl cordon my-node                                        # step 1: stop new scheduling here
kubectl get pods -o wide --field-selector spec.nodeName=my-node   # step 2: see what's actually running here first
kubectl drain my-node --ignore-daemonsets --delete-emptydir-data --timeout=300s   # step 3: evict, respecting PDBs

drain respects PodDisruptionBudgets automatically when eviction (not raw delete) is used — if evicting a pod would violate its workload's PDB (e.g. "at least 2 of 3 replicas must stay Ready"), drain blocks on that pod rather than violating the budget, until the replacement pod elsewhere is Ready. This is the mechanism that makes a rolling node upgrade genuinely zero-downtime for a correctly-configured workload, rather than just "probably fine."

Real-world scenario: a GitOps-style CI gate using `diff` + `wait`#

# .github/workflows/deploy.yml
- name: Preview changes
  run: kubectl diff -f manifests/ || true    # diff exits 1 on any difference — don't fail the step on that alone
- name: Apply
  run: kubectl apply -f manifests/
- name: Wait for rollout
  run: kubectl wait --for=condition=Available deployment/my-app --timeout=180s
- name: Smoke test
  run: curl -f https://staging.example.com/health

Chaining apply → wait → smoke test is what actually confirms a deploy is healthy, rather than just confirming the API server accepted the manifest — apply returning success only means the desired state was recorded, not that the workload is actually up and serving traffic yet.

Common pitfalls#

Treating cordon as equivalent to drain — see the IMPORTANT callout above.
Hand-writing a partial file for kubectl replace — see the WARNING above; always get -o yaml first.
Guessing a field name instead of running kubectl explain — especially costly on a CRD, where the schema is genuinely specific to that cluster's installed version of the operator.
Forgetting --ignore-daemonsets on drain — without it, drain refuses to proceed at all if any DaemonSet-managed pod is present on the node (which is almost always true — most clusters run at least a CNI or logging DaemonSet on every node).

When to reach for something else#

For declarative, git-tracked cluster state (rather than one-off imperative commands like patch/taint), reach for a GitOps controller (Argo CD, Flux) layered on top of kubectl apply — this page's commands remain the right tool for incident response, node maintenance, and ad-hoc debugging even in a GitOps-managed cluster, but shouldn't be how routine application deploys happen.

← PreviousDebugging & Troubleshooting