Kubernetes Security Deep Dive (CKS-Aligned)

Assumes you're comfortable with the admission control chain and RBAC basics from Part 1, and the NetworkPolicy mechanics from Part 3 — this chapter goes deep on hardening, supply chain, and runtime security rather than re-introducing either.

Table of Contents#

1. Why This Part Exists
2. The Defense-in-Depth Model, Extended
3. RBAC Deep Dive — Roles, ClusterRoles, and Aggregation
4. RBAC Least Privilege in Practice
5. Auditing RBAC — Finding Over-Privileged Grants
6. ServiceAccount Security and Token Hardening
7. Pod Security Standards and Pod Security Admission
8. SecurityContext Deep Dive
9. seccomp and AppArmor — Restricting What a Container Can Do
10. NetworkPolicy Hardening Patterns
11. Secrets Management and Encryption at Rest
12. Supply Chain Security — Scanning, SBOMs, and Signing
13. Enforcing Signature Verification at Admission Time
14. Runtime Security with Falco
15. CIS Benchmarks and kube-bench
16. Audit Logging — Configuration and What to Watch
17. Minimizing the Attack Surface: Immutable Containers
18. A Full Worked Hardening Pass: Securing the checkout Namespace
19. CKS Domain Coverage Map
20. Common Mistakes and Interview Traps
21. Worked Practice Problems
22. Summary and What's Next

Why This Part Exists#

Every earlier part in this series touched security in passing — the admission chain in Part 1, Pod Security Standards mentioned once in Part 1's admission-controller table, NetworkPolicy basics in Part 3 — but none of them treated security as the primary subject, and the Certified Kubernetes Security Specialist (CKS) exam is an entire, separate certification precisely because securing a cluster is a distinct skill from operating one. This chapter is organized around the CKS exam's own six domains, not because passing an exam is the point, but because that domain breakdown is a genuinely well-organized map of everything a production cluster owner actually needs to have covered.

The throughline system continues: checkout-service, catalog-service, and inventory-service in a checkout namespace, now getting the full hardening treatment a real security review would apply before that system goes anywhere near production traffic.

The Defense-in-Depth Model, Extended#

Part 1 covered one slice of this — authentication, authorization, and admission at request time. Real cluster security is layered far beyond that single request path, and no single layer is sufficient alone.

This diagram is deliberately not a request path — it's a rough ordering from "prevent" to "detect." Layers 1-4 try to stop a bad outcome before it happens; layers 5-6 assume something will eventually get through despite that and focus on limiting what it can do, and noticing quickly when it does. A cluster investing entirely in layer 2-3 controls while ignoring layer 5-6 detection is common and dangerous — it looks secure until the first incident, at which point there's no signal that anything happened at all.

RBAC Deep Dive — Roles, ClusterRoles, and Aggregation#

RBAC has exactly four object kinds, and understanding the namespace-scoped vs. cluster-scoped split between them resolves most of the confusion practitioners have with it.

The genuinely non-obvious part: a ClusterRole can be bound namespace-scoped via a RoleBinding — this is the standard pattern for reusing one well-defined set of permissions (e.g. a pod-reader ClusterRole) across many namespaces without duplicating the same Role YAML in every one of them.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-in-checkout
  namespace: checkout
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole    # a ClusterRole, bound namespace-scoped
  name: pod-reader
subjects:
  - kind: ServiceAccount
    name: checkout-readonly
    namespace: checkout

Aggregated ClusterRoles solve a different problem: letting multiple teams or controllers each contribute rules to a shared, composite role without editing a single central YAML file. A ClusterRole with an aggregationRule automatically absorbs the rules of every other ClusterRole matching its label selector — Kubernetes's own built-in admin/edit/view ClusterRoles are themselves built this way, and custom controllers/operators extend admin the same way by shipping their own small ClusterRole labeled to match.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-monitoring-aggregate-to-view
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"   # absorbed into the built-in "view" role
rules:
  - apiGroups: ["monitoring.coreos.com"]
    resources: ["prometheuses", "servicemonitors"]
    verbs: ["get", "list", "watch"]

RBAC Least Privilege in Practice#

A worked example beats an abstract "follow least privilege" instruction — here's exactly what checkout-service's own permissions should look like if it genuinely only needs to read its own ConfigMap and nothing else.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout-service
  namespace: checkout
automountServiceAccountToken: false   # see ServiceAccount Security below
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: checkout-config-reader
  namespace: checkout
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["checkout-config"]   # scoped to ONE named object, not every ConfigMap
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: checkout-service-config-reader
  namespace: checkout
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: checkout-config-reader
subjects:
  - kind: ServiceAccount
    name: checkout-service
    namespace: checkout

Three least-privilege decisions worth calling out explicitly, since each one is easy to skip under deadline pressure:

• resourceNames scopes the grant to exactly one named object, not every ConfigMap in the namespace — a compromised checkout-service pod can't enumerate or read any other team's configuration sitting in the same namespace.
• The verb list is ["get"] only, not ["get", "list", "watch"] — get requires already knowing the object's name, while list/watch let a caller enumerate every object of that kind, which is a meaningfully larger information disclosure if the ServiceAccount's token ever leaks.
• No apiGroups: ["*"] or resources: ["*"] anywhere — every wildcard silently grants access to any future resource type added to that API group, including ones that don't exist yet when the Role was written.

Auditing RBAC — Finding Over-Privileged Grants#

kubectl auth can-i answers "can this identity do this specific thing" — the tool for verifying a grant is correctly scoped, not for discovering what's over-scoped across a whole cluster.

kubectl auth can-i list secrets --as=system:serviceaccount:checkout:checkout-service -n checkout
kubectl auth can-i '*' '*' --as=system:serviceaccount:checkout:checkout-service -A
kubectl auth can-i --list --as=system:serviceaccount:checkout:checkout-service -n checkout

ServiceAccount Security and Token Hardening#

Every pod runs as some ServiceAccount whether one is explicitly specified or not — the default ServiceAccount in every namespace exists automatically, and a pod that doesn't request one gets it implicitly, along with an auto-mounted token, unless that behavior is explicitly disabled.

The modern default (TokenRequest API, stable since 1.20) issues short-lived, audience-bound, auto-rotated tokens rather than the old long-lived static Secret-backed tokens — a meaningful hardening improvement on its own, since a leaked legacy token was valid indefinitely while a leaked projected token expires within roughly an hour by default.

Pod Security Standards and Pod Security Admission#

PodSecurityPolicy was deprecated in 1.21 and fully removed in 1.25 — Pod Security Standards (PSS), a set of three predefined profiles enforced by the built-in Pod Security Admission (PSA) controller, replaced it entirely. This matters as a version fact worth being precise about: any environment or course material still referencing PodSecurityPolicy as current is describing a mechanism that no longer exists on any supported Kubernetes version.

PSA is configured entirely through namespace labels — there is no separate policy object to author for the built-in profiles, which is the single biggest usability improvement over the old PodSecurityPolicy model:

apiVersion: v1
kind: Namespace
metadata:
  name: checkout
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

SecurityContext Deep Dive#

securityContext is where Pod Security Standards' abstract rules become concrete YAML fields — every restricted-profile requirement maps to a specific securityContext setting, at either the pod or container level.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-service
  namespace: checkout
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: checkout-service
          image: registry.internal/checkout-service:1.4.2
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

seccomp and AppArmor — Restricting What a Container Can Do#

seccomp and AppArmor solve two different, complementary problems: seccomp restricts which system calls a process may make to the kernel; AppArmor restricts which files and capabilities a process may access, via a named profile loaded on the host.

securityContext:
  seccompProfile:
    type: RuntimeDefault   # the container runtime's own curated default profile — a strong baseline
# or, for a custom profile authored for a specific workload's exact syscall needs:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/checkout-service.json

RuntimeDefault (the container runtime's own moderately restrictive default profile, blocking roughly 44 of the most dangerous syscalls out of ~300+ available on Linux) is the correct default for nearly every workload and is what the restricted Pod Security Standard itself requires. A fully custom Localhost-type profile, generated by tracing a workload's actual syscalls under load (strace or a tool like oci-seccomp-bpf-hook), is worth the extra effort only for a small number of genuinely high-sensitivity workloads — most teams get the large majority of the benefit from RuntimeDefault alone.

NetworkPolicy Hardening Patterns#

Part 3 covered NetworkPolicy mechanics and Part 10 covered debugging a NetworkPolicy that's blocking something unexpected — this section is the hardening pattern a security review actually asks for: a default-deny baseline with explicit, minimal allow rules layered on top.

# 1. Deny everything by default, in every namespace that holds workloads
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: checkout
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
# 2. Then explicitly allow exactly what's needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-allow-catalog-and-dns
  namespace: checkout
spec:
  podSelector:
    matchLabels: { app: checkout-service }
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: catalog } }
      ports: [{ protocol: TCP, port: 8080 }]
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]

Secrets Management and Encryption at Rest#

A Kubernetes Secret is base64-encoded, not encrypted, by default — anyone with get/list RBAC access to Secrets in a namespace, or read access to the etcd data files directly, can trivially recover the plaintext unless encryption at rest is explicitly configured.

# /etc/kubernetes/enc/encryption-config.yaml, referenced by kube-apiserver's
# --encryption-provider-config flag
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}   # fallback for objects written before encryption was enabled

Supply Chain Security — Scanning, SBOMs, and Signing#

"Supply chain security" is 20% of the CKS exam and, per the community research earlier in this series' planning, one of the fastest-growing concerns across the industry — the question it answers is "how do you know the image actually running in your cluster is the one you built, unmodified, from known-good dependencies."

# 2. Scan for known vulnerabilities before the image ever reaches a registry
trivy image registry.internal/checkout-service:1.4.2

# 3. Generate a Software Bill of Materials — every package and version inside
syft registry.internal/checkout-service:1.4.2 -o spdx-json > checkout-service-sbom.json

# 4. Sign the image digest, keyless (OIDC-backed, via Sigstore's public Fulcio/Rekor infrastructure)
cosign sign registry.internal/checkout-service:1.4.2

# Verify the signature independently, e.g. in CI before promoting to a production registry
cosign verify registry.internal/checkout-service:1.4.2 \
  --certificate-identity=ci@example.com \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com

Keyless signing (via Sigstore's Fulcio certificate authority and Rekor transparency log) is the modern default over long-lived signing keys — a signing key stored in CI as a static secret is itself a supply chain risk (if it leaks, an attacker can sign arbitrary malicious images indistinguishable from legitimate ones); keyless signing instead issues a short-lived certificate tied to a verified OIDC identity (e.g. "this GitHub Actions workflow, on this exact repository") for each individual signing operation, with the fact of that signature recorded permanently and publicly in Rekor's append-only transparency log.

Enforcing Signature Verification at Admission Time#

Generating a signature is meaningless if nothing in the cluster actually checks it before running the image — enforcement is a policy-engine admission webhook (Kyverno or the Sigstore Policy Controller), rejecting any Pod whose image reference isn't signed by a trusted identity.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-checkout-images
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["checkout"]
      verifyImages:
        - imageReferences: ["registry.internal/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "ci@example.com"
                    issuer: "https://token.actions.githubusercontent.com"

Runtime Security with Falco#

Everything so far in this chapter is preventive — Falco is the detective control, watching system calls and Kubernetes audit events in real time for behavior that looks like a live compromise rather than a misconfiguration.

# A representative custom Falco rule — alert if a shell is spawned
# inside any container in the checkout namespace
- rule: Unexpected shell in checkout namespace
  desc: Detects a shell being spawned inside a checkout-namespace container
  condition: >
    spawned_process and container and
    k8s.ns.name = "checkout" and
    proc.name in (shell_binaries)
  output: >
    Shell spawned in checkout namespace
    (user=%user.name container=%container.name image=%container.image.repository command=%proc.cmdline)
  priority: WARNING

Falco's core insight is that it operates below the application layer entirely — using eBPF (Part 12 covers eBPF's role in the CNI layer; here it's the same underlying kernel technology applied to security observability) to watch actual syscalls, it detects a compromise regardless of what language or framework the compromised application is written in, and regardless of whether the attacker's technique was even anticipated by name — a rule like the one above catches "someone got a shell in a container that should never need one" as a category, not a specific known exploit.

CIS Benchmarks and kube-bench#

The CIS Kubernetes Benchmark is a detailed, versioned checklist of specific configuration checks (API server flags, file permissions on control plane config files, kubelet flags) — kube-bench automates running that checklist against a real cluster and reports pass/fail per control.

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench

On managed Kubernetes (EKS/AKS/GKE), a large fraction of the control-plane-level checks are simply not applicable — the cloud provider manages the API server and etcd configuration directly, and kube-bench run against a managed cluster should be scoped to the node/worker-level checks it can actually influence, not treated as a report card on checks the cluster operator has no ability to change.

Audit Logging — Configuration and What to Watch#

The Kubernetes audit log records every request to the API Server — who did what, to which resource, when — at a configurable level of detail per rule, and is the primary forensic record after any suspected compromise.

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources: [{ group: "", resources: ["secrets"] }]
  - level: Metadata
    resources: [{ group: "rbac.authorization.k8s.io" }]
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  - level: Metadata

# kube-apiserver flags that activate the policy above
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=5
--audit-log-maxsize=100

Order matters — rules are evaluated top to bottom, first match wins, which is why the example policy above puts the high-value secrets rule first (always capture full detail on Secret access) and the noisy system:kube-proxy exclusion after the rules that matter but before the catch-all Metadata default — placing the exclusion first would still work here since it only matches kube-proxy, but a catch-all rule placed too early in any audit policy silently shadows every more specific rule that follows it.

Minimizing the Attack Surface: Immutable Containers#

Part 10 covered distroless images from a debugging-ergonomics angle; the security lens on the same choice is that every shell, package manager, and debugging utility present in a production image is also available to an attacker who gains code execution inside it.

A Full Worked Hardening Pass: Securing the checkout Namespace#

Bringing every control in this chapter together on the one throughline system, in the order a real hardening project would actually apply them:

The ordering itself is a deliberate lesson, not an arbitrary checklist: every enforcement step (3, 8) is preceded by an audit/warn step (1, 7) that surfaces what would break before it's actually blocked — this is the same staged-rollout principle repeated three separate times in this chapter (PSA, NetworkPolicy default-deny per Part 10's incident, and signature verification) because it's the single most important practical lesson for applying any of these controls to a system that's already running in production without causing a self-inflicted outage in the process of trying to secure it.

CKS Domain Coverage Map#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team enables EncryptionConfiguration for Secrets on a cluster that's been running for two years. Six months later, a security audit finds several Secrets in an etcd backup snapshot still in plaintext. What went wrong?

Answer: Enabling EncryptionConfiguration only encrypts Secrets written or updated after it takes effect — every Secret that existed before that point and was never subsequently updated remains in its original, unencrypted form in etcd indefinitely. The fix required after enabling encryption is a one-time forced rewrite of every existing Secret (e.g. kubectl get secrets -A -o json | kubectl replace -f -) to bring pre-existing data under the new encryption; skipping that step is exactly the gap this audit found.

Problem 2: A cluster owner sets a namespace's Pod Security Admission label straight to pod-security.kubernetes.io/enforce: restricted on a namespace already running several Deployments. What's the likely immediate consequence, and what should have been done instead?

Answer: Any already-running pod is unaffected (PSA only evaluates objects at admission/creation time), but the next rollout, scale event, or pod recreation for any Deployment not already compliant with restricted will be rejected outright, potentially in the middle of an unrelated, time-sensitive deploy. The safer approach is setting audit/warn: restricted first, fixing every violation the warnings surface across existing workloads, and only then flipping enforce to restricted once nothing currently running would be rejected by it.

Problem 3: An admission policy requires all images to be signed, enforced cluster-wide. A developer reports that a brand-new internal tool's pod is stuck failing to create with image signature verification failed, even though the image was built and pushed correctly. What are the two most likely categories of cause, and how would you tell them apart?

Answer: Either the image genuinely was never signed (a CI pipeline gap — the build step ran but the signing step was skipped, misconfigured, or failed silently), or it was signed but by an identity the policy doesn't trust (e.g. signed by a personal cosign key during local testing rather than the CI pipeline's expected OIDC identity). Running cosign verify manually against the exact image reference, comparing the --certificate-identity/--certificate-oidc-issuer it reports against what the admission policy's attestors block expects, distinguishes the two immediately — "no signature found at all" points to a CI gap, while "signature found, wrong identity" points to a trust-configuration mismatch.

Summary and What's Next#

Security in Kubernetes is genuinely six separate, complementary disciplines — RBAC and authentication, workload hardening, network segmentation, secrets handling, supply chain integrity, and runtime detection — and the CKS exam's domain breakdown is a reliable map of all six because no cluster is actually secure with only some of them addressed. The staged-rollout pattern (audit/warn before enforce) is worth internalizing as a single transferable principle that applies to every enforcement mechanism covered here.

Part 12 turns to a different, equally consequential gap: autoscaling. HPA, VPA, and KEDA all showed up in passing references across earlier parts of this series without ever getting a dedicated treatment — Part 12 covers all three in full depth, including exactly how they interact with each other and with the Karpenter/Cluster Autoscaler node-level scaling from Part 7.