# Kubernetes Deep Dive — Part 11: Kubernetes Security Deep Dive (CKS-Aligned) > **Series:** Kubernetes Deep Dive (11 of 19) > **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane > **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects > **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI) > **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators > **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE > **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes > **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth > **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway > **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem > **Part 10:** `10-troubleshooting-kubernetes.md` — Troubleshooting Kubernetes, Systematically > **Part 11:** This file — Kubernetes Security Deep Dive (CKS-Aligned) > **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling > **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale > **Part 14:** `14-ai-ml-workloads-on-kubernetes.md` — Running AI/ML Workloads on Kubernetes > **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management > **Part 16:** `16-capacity-planning.md` — Capacity Planning for Kubernetes Clusters > **Part 17:** `17-cilium-ebpf-networking.md` — Cilium & eBPF Networking Deep Dive > **Part 18:** `18-backup-and-disaster-recovery.md` — Backup & Disaster Recovery for Workloads > **Part 19:** `19-building-custom-controllers-and-operators.md` — Building Custom Controllers and Operators > **Questions:** `questions.md` Assumes you're comfortable with the admission control chain and RBAC basics from Part 1, and the NetworkPolicy mechanics from Part 3 — this chapter goes deep on hardening, supply chain, and runtime security rather than re-introducing either. ## Table of Contents 1. [Why This Part Exists](#why-this-part-exists) 2. [The Defense-in-Depth Model, Extended](#the-defense-in-depth-model-extended) 3. [RBAC Deep Dive — Roles, ClusterRoles, and Aggregation](#rbac-deep-dive--roles-clusterroles-and-aggregation) 4. [RBAC Least Privilege in Practice](#rbac-least-privilege-in-practice) 5. [Auditing RBAC — Finding Over-Privileged Grants](#auditing-rbac--finding-over-privileged-grants) 6. [ServiceAccount Security and Token Hardening](#serviceaccount-security-and-token-hardening) 7. [Pod Security Standards and Pod Security Admission](#pod-security-standards-and-pod-security-admission) 8. [SecurityContext Deep Dive](#securitycontext-deep-dive) 9. [seccomp and AppArmor — Restricting What a Container Can Do](#seccomp-and-apparmor--restricting-what-a-container-can-do) 10. [NetworkPolicy Hardening Patterns](#networkpolicy-hardening-patterns) 11. [Secrets Management and Encryption at Rest](#secrets-management-and-encryption-at-rest) 12. [Supply Chain Security — Scanning, SBOMs, and Signing](#supply-chain-security--scanning-sboms-and-signing) 13. [Enforcing Signature Verification at Admission Time](#enforcing-signature-verification-at-admission-time) 14. [Runtime Security with Falco](#runtime-security-with-falco) 15. [CIS Benchmarks and kube-bench](#cis-benchmarks-and-kube-bench) 16. [Audit Logging — Configuration and What to Watch](#audit-logging--configuration-and-what-to-watch) 17. [Minimizing the Attack Surface: Immutable Containers](#minimizing-the-attack-surface-immutable-containers) 18. [A Full Worked Hardening Pass: Securing the `checkout` Namespace](#a-full-worked-hardening-pass-securing-the-checkout-namespace) 19. [CKS Domain Coverage Map](#cks-domain-coverage-map) 20. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 21. [Worked Practice Problems](#worked-practice-problems) 22. [Summary and What's Next](#summary-and-whats-next) --- ## Why This Part Exists **Every earlier part in this series touched security in passing — the admission chain in Part 1, Pod Security Standards mentioned once in Part 1's admission-controller table, NetworkPolicy basics in Part 3 — but none of them treated security as the primary subject, and the Certified Kubernetes Security Specialist (CKS) exam is an entire, separate certification precisely because securing a cluster is a distinct skill from operating one.** This chapter is organized around the CKS exam's own six domains, not because passing an exam is the point, but because that domain breakdown is a genuinely well-organized map of everything a production cluster owner actually needs to have covered. | CKS domain | Weight | Covered in this chapter | |---|---|---| | Cluster Setup | 15% | NetworkPolicy hardening, CIS benchmarks | | Cluster Hardening | 15% | RBAC deep dive, ServiceAccount security, Pod Security Standards | | System Hardening | 10% | SecurityContext, seccomp, AppArmor | | Minimize Microservice Vulnerabilities | 20% | SecurityContext, Secrets management, NetworkPolicy | | Supply Chain Security | 20% | Image scanning, SBOMs, signing, admission verification | | Monitoring, Logging and Runtime Security | 20% | Falco, audit logging, immutable containers | The throughline system continues: `checkout-service`, `catalog-service`, and `inventory-service` in a `checkout` namespace, now getting the full hardening treatment a real security review would apply before that system goes anywhere near production traffic. ## The Defense-in-Depth Model, Extended **Part 1 covered one slice of this — authentication, authorization, and admission at request time. Real cluster security is layered far beyond that single request path, and no single layer is sufficient alone.** ```mermaid flowchart TD L1["1. Cluster setup:
API server flags, etcd
encryption, network segmentation"] --> L2 L2["2. RBAC & authentication:
who can do what,
to which resources"] --> L3 L3["3. Pod & workload hardening:
SecurityContext, PSS,
seccomp/AppArmor"] --> L4 L4["4. Network segmentation:
NetworkPolicy default-deny"] --> L5 L5["5. Supply chain:
scanned, signed, verified
images only"] --> L6 L6["6. Runtime detection:
Falco, audit logs,
anomaly alerting"] ``` **This diagram is deliberately not a request path — it's a rough ordering from "prevent" to "detect."** Layers 1-4 try to stop a bad outcome before it happens; layers 5-6 assume something will eventually get through despite that and focus on limiting what it can do, and noticing quickly when it does. A cluster investing entirely in layer 2-3 controls while ignoring layer 5-6 detection is common and dangerous — it looks secure until the first incident, at which point there's no signal that anything happened at all. > [!IMPORTANT] > No single layer above is sufficient on its own, and this is the correct way to think about "is my > cluster secure" — the real question is always "which of these six layers have I not addressed," not > "have I secured Kubernetes," because there is no single control that covers all six at once. ## RBAC Deep Dive — Roles, ClusterRoles, and Aggregation **RBAC has exactly four object kinds, and understanding the namespace-scoped vs. cluster-scoped split between them resolves most of the confusion practitioners have with it.** | Object | Scope | Grants access to | |---|---|---| | `Role` | One namespace | Namespaced resources, within that namespace only | | `ClusterRole` | Cluster-wide | Namespaced resources (across *all* namespaces, if bound cluster-wide) **or** cluster-scoped resources (`nodes`, `namespaces`, `persistentvolumes`) | | `RoleBinding` | One namespace | Binds a `Role` **or** a `ClusterRole` to a subject, scoped to that one namespace only | | `ClusterRoleBinding` | Cluster-wide | Binds a `ClusterRole` to a subject, cluster-wide | **The genuinely non-obvious part**: a `ClusterRole` can be bound namespace-scoped via a `RoleBinding` — this is the standard pattern for reusing one well-defined set of permissions (e.g. a `pod-reader` ClusterRole) across many namespaces without duplicating the same `Role` YAML in every one of them. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: read-pods-in-checkout namespace: checkout roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole # a ClusterRole, bound namespace-scoped name: pod-reader subjects: - kind: ServiceAccount name: checkout-readonly namespace: checkout ``` ```mermaid flowchart LR CR["ClusterRole: pod-reader
(defined once)"] --> RB1["RoleBinding in
checkout namespace"] CR --> RB2["RoleBinding in
catalog namespace"] CR --> CRB["ClusterRoleBinding
(cluster-wide)"] RB1 --> Scope1["Effective: read pods
ONLY in checkout"] RB2 --> Scope2["Effective: read pods
ONLY in catalog"] CRB --> Scope3["Effective: read pods
in EVERY namespace"] ``` **Aggregated ClusterRoles** solve a different problem: letting multiple teams or controllers each contribute rules to a shared, composite role without editing a single central YAML file. A ClusterRole with an `aggregationRule` automatically absorbs the rules of every other ClusterRole matching its label selector — Kubernetes's own built-in `admin`/`edit`/`view` ClusterRoles are themselves built this way, and custom controllers/operators extend `admin` the same way by shipping their own small ClusterRole labeled to match. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: custom-monitoring-aggregate-to-view labels: rbac.authorization.k8s.io/aggregate-to-view: "true" # absorbed into the built-in "view" role rules: - apiGroups: ["monitoring.coreos.com"] resources: ["prometheuses", "servicemonitors"] verbs: ["get", "list", "watch"] ``` ## RBAC Least Privilege in Practice **A worked example beats an abstract "follow least privilege" instruction — here's exactly what `checkout-service`'s own permissions should look like if it genuinely only needs to read its own ConfigMap and nothing else.** ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: checkout-service namespace: checkout automountServiceAccountToken: false # see ServiceAccount Security below --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: checkout-config-reader namespace: checkout rules: - apiGroups: [""] resources: ["configmaps"] resourceNames: ["checkout-config"] # scoped to ONE named object, not every ConfigMap verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: checkout-service-config-reader namespace: checkout roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: checkout-config-reader subjects: - kind: ServiceAccount name: checkout-service namespace: checkout ``` **Three least-privilege decisions worth calling out explicitly, since each one is easy to skip under deadline pressure:** - `resourceNames` scopes the grant to exactly one named object, not every ConfigMap in the namespace — a compromised `checkout-service` pod can't enumerate or read any other team's configuration sitting in the same namespace. - The verb list is `["get"]` only, not `["get", "list", "watch"]` — `get` requires already knowing the object's name, while `list`/`watch` let a caller enumerate every object of that kind, which is a meaningfully larger information disclosure if the ServiceAccount's token ever leaks. - No `apiGroups: ["*"]` or `resources: ["*"]` anywhere — every wildcard silently grants access to any future resource type added to that API group, including ones that don't exist yet when the Role was written. ## Auditing RBAC — Finding Over-Privileged Grants **`kubectl auth can-i` answers "can this identity do this specific thing" — the tool for verifying a grant is correctly scoped, not for discovering what's over-scoped across a whole cluster.** ```bash kubectl auth can-i list secrets --as=system:serviceaccount:checkout:checkout-service -n checkout kubectl auth can-i '*' '*' --as=system:serviceaccount:checkout:checkout-service -A kubectl auth can-i --list --as=system:serviceaccount:checkout:checkout-service -n checkout ``` | Question | Command | |---|---| | Can this specific ServiceAccount do this specific thing? | `kubectl auth can-i --as= -n ` | | What can this identity do at all, in a namespace? | `kubectl auth can-i --list --as= -n ` | | Does anything in the cluster have a dangerous wildcard grant? | No single built-in command — this requires listing every `ClusterRole`/`Role` and grepping for `"*"`, or a dedicated RBAC-auditing tool | > [!TIP] > `kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name'` > is worth running as a periodic health check on its own — every subject bound to `cluster-admin` is a > single point of total cluster compromise if that identity's credentials ever leak, and it's common for > this list to accumulate entries added for a one-time debugging session that were never removed afterward. ## ServiceAccount Security and Token Hardening **Every pod runs as some ServiceAccount whether one is explicitly specified or not — the `default` ServiceAccount in every namespace exists automatically, and a pod that doesn't request one gets it implicitly, along with an auto-mounted token, unless that behavior is explicitly disabled.** ```mermaid sequenceDiagram participant Pod participant Kubelet participant API as API Server Note over Pod: Pod created with no
explicit serviceAccountName Kubelet->>API: Request a projected,
time-bound service
account token API-->>Kubelet: Short-lived JWT
(default TTL, auto-rotated) Kubelet->>Pod: Mount token at
/var/run/secrets/kubernetes.io/
serviceaccount/token Note over Pod: Any process inside the
container can now read
this token and call the
API Server as this identity ``` **The modern default (TokenRequest API, stable since 1.20) issues short-lived, audience-bound, auto-rotated tokens** rather than the old long-lived static Secret-backed tokens — a meaningful hardening improvement on its own, since a leaked legacy token was valid indefinitely while a leaked projected token expires within roughly an hour by default. | Hardening step | Why | |---|---| | Set `automountServiceAccountToken: false` on any ServiceAccount/pod that never calls the Kubernetes API | Removes the token entirely — nothing to steal if the container is compromised | | Never use the `default` ServiceAccount for application workloads | It's shared cluster-wide convention, easy to accidentally over-grant, and offers no per-application identity for auditing | | Bind the narrowest possible Role per ServiceAccount (previous section) | A leaked token is only as dangerous as what it's allowed to do | | For cloud-native workloads, prefer workload identity federation (IRSA on EKS, Workload Identity on GKE — Parts 5/7) over static cloud credentials in a Secret | Removes long-lived cloud credentials from the cluster entirely; the ServiceAccount token itself becomes the only secret in play, and it's short-lived | ## Pod Security Standards and Pod Security Admission **`PodSecurityPolicy` was deprecated in 1.21 and fully removed in 1.25 — Pod Security Standards (PSS), a set of three predefined profiles enforced by the built-in Pod Security Admission (PSA) controller, replaced it entirely.** This matters as a version fact worth being precise about: any environment or course material still referencing `PodSecurityPolicy` as current is describing a mechanism that no longer exists on any supported Kubernetes version. | Profile | Allows | |---|---| | `privileged` | Everything — no restrictions, the default if a namespace has no PSS label at all | | `baseline` | Blocks the most severe escalation paths (host namespaces, privileged containers, most dangerous capabilities) while staying broadly compatible with existing workloads | | `restricted` | Heavily hardened — requires `runAsNonRoot`, `allowPrivilegeEscalation: false`, all capabilities dropped, an approved `seccompProfile`, and restricts volume types | **PSA is configured entirely through namespace labels — there is no separate policy object to author for the built-in profiles**, which is the single biggest usability improvement over the old PodSecurityPolicy model: ```yaml apiVersion: v1 kind: Namespace metadata: name: checkout labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: latest pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted ``` ```mermaid flowchart TD Apply["kubectl apply -f pod.yaml
into checkout namespace"] --> Check{"Does the pod spec
satisfy 'restricted'?"} Check -->|"Yes"| Admit["Pod admitted normally"] Check -->|"No"| Modes["Three independent PSA modes,
can all be set at once:"] Modes --> Enforce["enforce: reject the request
outright"] Modes --> Audit["audit: admit it, but log
a violation to the audit log"] Modes --> Warn["warn: admit it, but return
a warning to kubectl"] ``` > [!TIP] > Setting `audit`/`warn` to a *stricter* profile than `enforce` is the standard safe rollout pattern — set > `enforce: baseline` (so nothing breaks today) alongside `warn: restricted` and `audit: restricted`, watch > the warnings and audit log for a rollout window, fix what surfaces, and only then tighten `enforce` to > `restricted` once you know nothing currently running would be rejected by it. ## SecurityContext Deep Dive **`securityContext` is where Pod Security Standards' abstract rules become concrete YAML fields — every `restricted`-profile requirement maps to a specific `securityContext` setting, at either the pod or container level.** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: checkout-service namespace: checkout spec: template: spec: securityContext: runAsNonRoot: true runAsUser: 10001 fsGroup: 10001 seccompProfile: type: RuntimeDefault containers: - name: checkout-service image: registry.internal/checkout-service:1.4.2 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] ``` | Field | What it prevents | |---|---| | `runAsNonRoot: true` | The container process running as UID 0 — the single highest-value target if the container is ever compromised, since root inside the container is one namespace escape away from root on the node in a misconfigured runtime | | `allowPrivilegeEscalation: false` | A process gaining more privileges than its parent had (blocks `setuid` binaries and similar escalation paths) | | `capabilities.drop: ["ALL"]` | Every Linux capability beyond the bare minimum — most containers need zero of the ~40 available capabilities; add back only the specific one a workload genuinely requires (e.g. `NET_BIND_SERVICE` for binding to port 80 as non-root) | | `readOnlyRootFilesystem: true` | Malware or a compromised dependency writing to the container's own filesystem — forces any genuinely needed writable path to be an explicit, auditable volume mount instead | > [!WARNING] > `readOnlyRootFilesystem: true` is one of the highest-value hardening flags and also one of the most > likely to break something on first rollout — plenty of application images write temp files, cache > artifacts, or logs to paths under the container's own root filesystem by default. Test in a `warn`/`audit` > PSA namespace or a staging environment first; the fix is almost always adding a small `emptyDir` volume > mounted at the specific path the app writes to (e.g. `/tmp`), not abandoning the flag entirely. ## seccomp and AppArmor — Restricting What a Container Can Do **seccomp and AppArmor solve two different, complementary problems: seccomp restricts *which system calls* a process may make to the kernel; AppArmor restricts *which files and capabilities* a process may access, via a named profile loaded on the host.** ```mermaid flowchart LR Process["Container process"] --> Syscall{"Makes a syscall"} Syscall --> Seccomp["seccomp filter:
is this syscall on
the allowed list?"] Seccomp -->|"No"| Kill["Process killed or
syscall returns EPERM"] Seccomp -->|"Yes"| FS{"Accesses a file
or capability"} FS --> AppArmor["AppArmor profile:
is this path/capability
allowed for this profile?"] AppArmor -->|"No"| Deny["Access denied"] AppArmor -->|"Yes"| Allow["Operation proceeds"] ``` ```yaml securityContext: seccompProfile: type: RuntimeDefault # the container runtime's own curated default profile — a strong baseline # or, for a custom profile authored for a specific workload's exact syscall needs: seccompProfile: type: Localhost localhostProfile: profiles/checkout-service.json ``` `RuntimeDefault` (the container runtime's own moderately restrictive default profile, blocking roughly 44 of the most dangerous syscalls out of ~300+ available on Linux) is the correct default for nearly every workload and is what the `restricted` Pod Security Standard itself requires. A fully custom `Localhost`-type profile, generated by tracing a workload's actual syscalls under load (`strace` or a tool like `oci-seccomp-bpf-hook`), is worth the extra effort only for a small number of genuinely high-sensitivity workloads — most teams get the large majority of the benefit from `RuntimeDefault` alone. ``` # AppArmor profile, loaded on the node, then referenced by annotation (pre-1.30) # or the modern securityContext.appArmorProfile field (1.30+): securityContext: appArmorProfile: type: Localhost localhostProfile: k8s-checkout-restricted ``` > [!NOTE] > AppArmor is Linux-distribution-dependent (Ubuntu/Debian ship it by default; RHEL/CentOS-family > distributions typically use SELinux instead, which enforces a conceptually similar mandatory access > control model with different tooling). Confirm which is actually available on your nodes before writing > AppArmor profiles for a cluster that might be running on RHEL-based AMIs/images. ## NetworkPolicy Hardening Patterns **Part 3 covered NetworkPolicy mechanics and Part 10 covered debugging a NetworkPolicy that's blocking something unexpected — this section is the hardening pattern a security review actually asks for: a default-deny baseline with explicit, minimal allow rules layered on top.** ```yaml # 1. Deny everything by default, in every namespace that holds workloads apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: checkout spec: podSelector: {} policyTypes: ["Ingress", "Egress"] --- # 2. Then explicitly allow exactly what's needed apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: checkout-allow-catalog-and-dns namespace: checkout spec: podSelector: matchLabels: { app: checkout-service } policyTypes: ["Egress"] egress: - to: - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: catalog } } ports: [{ protocol: TCP, port: 8080 }] - to: - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }] ``` | Hardening pattern | Reason | |---|---| | Default-deny both `Ingress` and `Egress`, not just `Ingress` | An `Ingress`-only default-deny still lets a compromised pod exfiltrate data or call out to a C2 server freely | | Always explicitly allow DNS egress (UDP/TCP 53 to `kube-system`) alongside any default-deny | The single most common self-inflicted outage from NetworkPolicy adoption (see Part 10) | | Prefer `namespaceSelector` + `podSelector` combinations over broad CIDR blocks | Namespace/pod identity survives pod IP churn; a CIDR rule silently stops matching after a redeploy changes IPs | | Label namespaces with `kubernetes.io/metadata.name` (automatic since 1.21+) for `namespaceSelector` targeting | Avoids having to hand-maintain a separate custom label just to select a namespace by name | ## Secrets Management and Encryption at Rest **A Kubernetes `Secret` is base64-encoded, not encrypted, by default — anyone with `get`/`list` RBAC access to Secrets in a namespace, or read access to the etcd data files directly, can trivially recover the plaintext unless encryption at rest is explicitly configured.** ```yaml # /etc/kubernetes/enc/encryption-config.yaml, referenced by kube-apiserver's # --encryption-provider-config flag apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: ["secrets"] providers: - aescbc: keys: - name: key1 secret: - identity: {} # fallback for objects written before encryption was enabled ``` ```mermaid sequenceDiagram participant Client participant API as API Server participant Enc as Encryption provider participant Etcd Client->>API: kubectl apply -f secret.yaml API->>Enc: Encrypt the Secret's data
before persisting Enc-->>API: Ciphertext API->>Etcd: Store ciphertext, not plaintext Note over Etcd: Anyone with raw etcd
disk/backup access sees
only ciphertext, not secrets ``` | Secrets practice | Why it matters | |---|---| | Enable `EncryptionConfiguration` for `secrets` (and ideally `configmaps` holding sensitive data) | Without it, an etcd snapshot backup — routinely copied to S3/GCS for disaster recovery — contains every Secret in plaintext | | Never commit raw `Secret` manifests to git | The most common real leak vector; use Sealed Secrets, External Secrets Operator, or a Vault/cloud-secrets-manager integration instead | | Set `immutable: true` on Secrets/ConfigMaps that never need runtime updates | Prevents an accidental (or malicious) in-place edit, and reduces the kubelet's watch load on that object | | Rotate the `EncryptionConfiguration` key periodically | A static encryption key held indefinitely is itself a long-lived secret worth rotating on the same cadence as any other credential | > [!CAUTION] > Enabling `EncryptionConfiguration` does **not** retroactively encrypt Secrets already written to etcd — > only newly written or updated objects get encrypted going forward. After enabling it, run > `kubectl get secrets -A -o json | kubectl replace -f -` (or the equivalent re-write-in-place operation) > to force every existing Secret to be re-persisted under the new encryption, otherwise old Secrets remain > in plaintext in etcd indefinitely. ## Supply Chain Security — Scanning, SBOMs, and Signing **"Supply chain security" is 20% of the CKS exam and, per the community research earlier in this series' planning, one of the fastest-growing concerns across the industry — the question it answers is "how do you know the image actually running in your cluster is the one you built, unmodified, from known-good dependencies."** ```mermaid flowchart LR Build["1. Build image"] --> Scan["2. Scan for known
CVEs (Trivy)"] Scan --> SBOM["3. Generate SBOM
(Syft) — every package
and version inside"] SBOM --> Sign["4. Sign the image
digest (cosign)"] Sign --> Push["5. Push signed image
+ SBOM to registry"] Push --> Verify["6. Admission-time
verification before
the cluster runs it"] ``` ```bash # 2. Scan for known vulnerabilities before the image ever reaches a registry trivy image registry.internal/checkout-service:1.4.2 # 3. Generate a Software Bill of Materials — every package and version inside syft registry.internal/checkout-service:1.4.2 -o spdx-json > checkout-service-sbom.json # 4. Sign the image digest, keyless (OIDC-backed, via Sigstore's public Fulcio/Rekor infrastructure) cosign sign registry.internal/checkout-service:1.4.2 # Verify the signature independently, e.g. in CI before promoting to a production registry cosign verify registry.internal/checkout-service:1.4.2 \ --certificate-identity=ci@example.com \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com ``` **Keyless signing (via Sigstore's Fulcio certificate authority and Rekor transparency log) is the modern default over long-lived signing keys** — a signing key stored in CI as a static secret is itself a supply chain risk (if it leaks, an attacker can sign arbitrary malicious images indistinguishable from legitimate ones); keyless signing instead issues a short-lived certificate tied to a verified OIDC identity (e.g. "this GitHub Actions workflow, on this exact repository") for each individual signing operation, with the fact of that signature recorded permanently and publicly in Rekor's append-only transparency log. ## Enforcing Signature Verification at Admission Time **Generating a signature is meaningless if nothing in the cluster actually checks it before running the image** — enforcement is a policy-engine admission webhook (Kyverno or the Sigstore Policy Controller), rejecting any Pod whose image reference isn't signed by a trusted identity. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: verify-image-signatures spec: validationFailureAction: Enforce rules: - name: verify-checkout-images match: any: - resources: kinds: ["Pod"] namespaces: ["checkout"] verifyImages: - imageReferences: ["registry.internal/*"] attestors: - entries: - keyless: subject: "ci@example.com" issuer: "https://token.actions.githubusercontent.com" ``` > [!WARNING] > Rolling out image-signature enforcement with `validationFailureAction: Enforce` on day one, before every > currently-deployed image has actually been signed, will block legitimate deployments cluster-wide the > moment the policy applies. Start with `Audit` (log violations, admit anyway), backfill signatures for > every image already in production use, confirm the audit log shows zero violations for a full deployment > cycle, and only then flip to `Enforce` — the same staged-rollout discipline as the Pod Security Admission > `warn`/`enforce` pattern earlier in this chapter. ## Runtime Security with Falco **Everything so far in this chapter is preventive — Falco is the detective control, watching system calls and Kubernetes audit events in real time for behavior that looks like a live compromise rather than a misconfiguration.** ```mermaid sequenceDiagram participant Container participant Kernel participant Falco participant Alert as Alerting pipeline Container->>Kernel: Unexpected syscall
(e.g. opens a shell
inside a running pod) Kernel->>Falco: eBPF probe captures
the syscall event Falco->>Falco: Match against loaded rules Falco->>Alert: Rule "Terminal shell in
container" fired Alert->>Alert: Page on-call / write
to SIEM ``` ```yaml # A representative custom Falco rule — alert if a shell is spawned # inside any container in the checkout namespace - rule: Unexpected shell in checkout namespace desc: Detects a shell being spawned inside a checkout-namespace container condition: > spawned_process and container and k8s.ns.name = "checkout" and proc.name in (shell_binaries) output: > Shell spawned in checkout namespace (user=%user.name container=%container.name image=%container.image.repository command=%proc.cmdline) priority: WARNING ``` **Falco's core insight is that it operates below the application layer entirely** — using eBPF (Part 12 covers eBPF's role in the CNI layer; here it's the same underlying kernel technology applied to security observability) to watch actual syscalls, it detects a compromise regardless of what language or framework the compromised application is written in, and regardless of whether the attacker's technique was even anticipated by name — a rule like the one above catches "someone got a shell in a container that should never need one" as a category, not a specific known exploit. ## CIS Benchmarks and kube-bench **The CIS Kubernetes Benchmark is a detailed, versioned checklist of specific configuration checks (API server flags, file permissions on control plane config files, kubelet flags) — `kube-bench` automates running that checklist against a real cluster and reports pass/fail per control.** ```bash kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml kubectl logs job/kube-bench ``` ``` [FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate [PASS] 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow [WARN] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true ``` **On managed Kubernetes (EKS/AKS/GKE), a large fraction of the control-plane-level checks are simply not applicable** — the cloud provider manages the API server and etcd configuration directly, and `kube-bench` run against a managed cluster should be scoped to the node/worker-level checks it can actually influence, not treated as a report card on checks the cluster operator has no ability to change. ## Audit Logging — Configuration and What to Watch **The Kubernetes audit log records every request to the API Server — who did what, to which resource, when — at a configurable level of detail per rule, and is the primary forensic record after any suspected compromise.** ```yaml # /etc/kubernetes/audit-policy.yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: RequestResponse resources: [{ group: "", resources: ["secrets"] }] - level: Metadata resources: [{ group: "rbac.authorization.k8s.io" }] - level: None users: ["system:kube-proxy"] verbs: ["watch"] - level: Metadata ``` ```bash # kube-apiserver flags that activate the policy above --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kubernetes/audit.log --audit-log-maxage=30 --audit-log-maxbackup=5 --audit-log-maxsize=100 ``` | Audit level | Records | |---|---| | `None` | Nothing — used to exclude high-volume, low-value noise (e.g. a health-check user's routine watches) | | `Metadata` | Who, what, when — request metadata only, no request/response bodies | | `Request` | Metadata plus the request body | | `RequestResponse` | Metadata plus both request and response bodies — the most detail, and the most storage/log volume | **Order matters — rules are evaluated top to bottom, first match wins**, which is why the example policy above puts the high-value `secrets` rule first (always capture full detail on Secret access) and the noisy `system:kube-proxy` exclusion after the rules that matter but before the catch-all `Metadata` default — placing the exclusion first would still work here since it only matches `kube-proxy`, but a catch-all rule placed too early in any audit policy silently shadows every more specific rule that follows it. ## Minimizing the Attack Surface: Immutable Containers **Part 10 covered distroless images from a debugging-ergonomics angle; the security lens on the same choice is that every shell, package manager, and debugging utility present in a production image is also available to an attacker who gains code execution inside it.** | Hardening choice | Attack-surface reduction | |---|---| | Distroless/scratch base image | No shell, no package manager — an attacker with code execution can't easily pivot, download tools, or explore the filesystem interactively | | `readOnlyRootFilesystem: true` (already covered above) | Blocks writing a downloaded payload to disk inside the container at all | | Multi-stage builds, discarding build tooling from the final image | A compiler or build-time dependency present in a runtime image is pure unnecessary attack surface with zero runtime benefit | | Minimal, pinned base image digests (not floating tags) | A floating `:latest` or `:1` tag can silently change under you; a pinned digest guarantees the exact bytes that were scanned and signed are the exact bytes running | ## A Full Worked Hardening Pass: Securing the `checkout` Namespace **Bringing every control in this chapter together on the one throughline system, in the order a real hardening project would actually apply them:** ```mermaid flowchart TD Start["checkout namespace,
unhardened baseline"] --> S1["1. Label namespace with
PSA warn/audit: restricted
(not enforce yet)"] S1 --> S2["2. Fix SecurityContext
violations surfaced by
warn/audit"] S2 --> S3["3. Flip PSA enforce
to restricted"] S3 --> S4["4. Apply default-deny
NetworkPolicy + explicit
allow rules"] S4 --> S5["5. Scope every
ServiceAccount's RBAC
to least privilege"] S5 --> S6["6. Enable Secret
encryption at rest"] S6 --> S7["7. Roll out image signature
verification in Audit mode"] S7 --> S8["8. Flip signature verification
to Enforce"] S8 --> S9["9. Deploy Falco +
audit log shipping
for ongoing detection"] ``` **The ordering itself is a deliberate lesson, not an arbitrary checklist**: every enforcement step (3, 8) is preceded by an audit/warn step (1, 7) that surfaces what would break *before* it's actually blocked — this is the same staged-rollout principle repeated three separate times in this chapter (PSA, NetworkPolicy default-deny per Part 10's incident, and signature verification) because it's the single most important practical lesson for applying any of these controls to a system that's already running in production without causing a self-inflicted outage in the process of trying to secure it. ## CKS Domain Coverage Map | CKS domain | This chapter's sections | |---|---| | Cluster Setup | NetworkPolicy Hardening Patterns, CIS Benchmarks and kube-bench | | Cluster Hardening | RBAC Deep Dive, RBAC Least Privilege, Auditing RBAC, ServiceAccount Security, Pod Security Standards | | System Hardening | SecurityContext Deep Dive, seccomp and AppArmor | | Minimize Microservice Vulnerabilities | SecurityContext Deep Dive, Secrets Management, NetworkPolicy Hardening | | Supply Chain Security | Supply Chain Security (scanning/SBOM/signing), Enforcing Signature Verification | | Monitoring, Logging and Runtime Security | Runtime Security with Falco, Audit Logging, Minimizing the Attack Surface | ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | Correct approach | |---|---|---| | Referencing `PodSecurityPolicy` as a current mechanism | Removed entirely in 1.25 — any cluster on a supported version doesn't have it | Use Pod Security Admission + Pod Security Standards | | Using `apiGroups: ["*"]`/`resources: ["*"]` "to keep things simple" in a Role | Silently grants access to every future resource type added to that API group | Always enumerate exact `apiGroups`, `resources`, and `verbs` | | Assuming base64-encoded Secrets are "encrypted" | Base64 is an encoding, trivially reversible, not encryption | Configure `EncryptionConfiguration` for actual encryption at rest | | Enforcing a new admission policy (PSA, signature verification) cluster-wide with no staged rollout | Blocks legitimate deployments cluster-wide the instant it's misconfigured or something wasn't yet compliant | Always roll out in `audit`/`warn`/non-enforcing mode first | | Treating a signed image as proof it's *safe* rather than proof it's *unmodified* | Signing proves provenance and integrity, not the absence of vulnerabilities | Signing and scanning are complementary controls, not substitutes for each other | | Skipping runtime detection because "we have good preventive controls" | Preventive controls fail; without Falco/audit logging there's no signal when they do | Always pair prevention (layers 1-5) with detection (layer 6) | ## Worked Practice Problems **Problem 1**: A team enables `EncryptionConfiguration` for Secrets on a cluster that's been running for two years. Six months later, a security audit finds several Secrets in an etcd backup snapshot still in plaintext. What went wrong? *Answer*: Enabling `EncryptionConfiguration` only encrypts Secrets written or updated *after* it takes effect — every Secret that existed before that point and was never subsequently updated remains in its original, unencrypted form in etcd indefinitely. The fix required after enabling encryption is a one-time forced rewrite of every existing Secret (e.g. `kubectl get secrets -A -o json | kubectl replace -f -`) to bring pre-existing data under the new encryption; skipping that step is exactly the gap this audit found. **Problem 2**: A cluster owner sets a namespace's Pod Security Admission label straight to `pod-security.kubernetes.io/enforce: restricted` on a namespace already running several Deployments. What's the likely immediate consequence, and what should have been done instead? *Answer*: Any already-running pod is unaffected (PSA only evaluates objects at admission/creation time), but the next rollout, scale event, or pod recreation for any Deployment not already compliant with `restricted` will be rejected outright, potentially in the middle of an unrelated, time-sensitive deploy. The safer approach is setting `audit`/`warn: restricted` first, fixing every violation the warnings surface across existing workloads, and only then flipping `enforce` to `restricted` once nothing currently running would be rejected by it. **Problem 3**: An admission policy requires all images to be signed, enforced cluster-wide. A developer reports that a brand-new internal tool's pod is stuck failing to create with `image signature verification failed`, even though the image was built and pushed correctly. What are the two most likely categories of cause, and how would you tell them apart? *Answer*: Either the image genuinely was never signed (a CI pipeline gap — the build step ran but the signing step was skipped, misconfigured, or failed silently), or it was signed but by an identity the policy doesn't trust (e.g. signed by a personal `cosign` key during local testing rather than the CI pipeline's expected OIDC identity). Running `cosign verify` manually against the exact image reference, comparing the `--certificate-identity`/`--certificate-oidc-issuer` it reports against what the admission policy's `attestors` block expects, distinguishes the two immediately — "no signature found at all" points to a CI gap, while "signature found, wrong identity" points to a trust-configuration mismatch. ## Summary and What's Next Security in Kubernetes is genuinely six separate, complementary disciplines — RBAC and authentication, workload hardening, network segmentation, secrets handling, supply chain integrity, and runtime detection — and the CKS exam's domain breakdown is a reliable map of all six because no cluster is actually secure with only some of them addressed. The staged-rollout pattern (audit/warn before enforce) is worth internalizing as a single transferable principle that applies to every enforcement mechanism covered here. Part 12 turns to a different, equally consequential gap: autoscaling. HPA, VPA, and KEDA all showed up in passing references across earlier parts of this series without ever getting a dedicated treatment — Part 12 covers all three in full depth, including exactly how they interact with each other and with the Karpenter/Cluster Autoscaler node-level scaling from Part 7.