Container & Kubernetes Security

Table of Contents#

1. Why Containers Need Their Own Security Discussion
2. Container Image Scanning
3. Image Scanning in Practice: Trivy
4. Dockerfile Hardening — A Line-by-Line Walkthrough
5. Distroless and Minimal Base Images
6. Container Runtime Isolation — What a Container Actually Is
7. Kubernetes Security — The Big Picture
8. RBAC — Role-Based Access Control
9. A Full Worked RBAC Example
10. Pod Security Standards
11. Network Policies — Zero Trust Inside the Cluster
12. Admission Controllers and OPA Gatekeeper
13. Secrets in Kubernetes — A Preview
14. Runtime Security: Falco
15. A Full Kubernetes Security Checklist
16. Common Mistakes
17. Worked Practice Problems
18. Summary and What's Next

---

Why Containers Need Their Own Security Discussion#

Containers introduce a genuinely new set of security questions on top of everything in Parts 1-2. It's not enough to know your application code is secure (SAST) or your dependencies are clean (SCA) — you also need to ask: is the container image itself safe, and is it running with more power than it needs?

Diagram

---

Container Image Scanning#

A container image isn't just your application — it's an entire filesystem, usually built on top of a base OS image (like ubuntu or alpine) that comes with its own collection of OS-level packages, which can themselves have known vulnerabilities, completely separate from anything SCA (Part 2) would catch in your application's own dependency manifest.

Diagram

The key insight: image scanning specifically targets the base OS and runtime layers — the part SCA and SAST don't look at at all. This is genuinely its own category of tooling, worth knowing as distinct from Part 2's SAST/SCA.

---

Image Scanning in Practice: Trivy#

Trivy (by Aqua Security) is the most widely used open-source container image scanner — fast, simple, and a very commonly expected tool to have hands-on familiarity with.

Copy
# Scan a container image for known vulnerabilities
trivy image nginx:1.25.0

# Only show HIGH and CRITICAL severity findings
trivy image --severity HIGH,CRITICAL nginx:1.25.0

# Fail the command (useful for CI gating) if any CRITICAL vuln is found
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan for misconfigurations in a Dockerfile itself, not just the built image
trivy config ./Dockerfile

# Scan a Kubernetes manifest for misconfigurations too
trivy config ./k8s-manifests/

# Generate a Software Bill of Materials (SBOM) — covered fully in Part 5
trivy image --format cyclonedx --output sbom.json myapp:latest

A realistic CI gate using Trivy:

Copy
# .github/workflows/image-scan.yml
- name: Build image
  run: docker build -t myapp:${{ github.sha }} .

- name: Scan image with Trivy
  run: |
    trivy image --exit-code 1 --severity CRITICAL,HIGH \
      --ignore-unfixed myapp:${{ github.sha }}

The --ignore-unfixed flag is worth knowing specifically: it excludes vulnerabilities that don't yet have an available patch — because failing a build over a CVE that literally cannot be fixed right now just blocks all progress for no actionable benefit. This is a practical, real-world tuning decision, not just a flag to memorize.

---

Dockerfile Hardening — A Line-by-Line Walkthrough#

A "before and after" comparison is one of the most effective ways to demonstrate real, hands-on Dockerfile security knowledge in an interview.

Before — An Insecure Dockerfile#

Copy
FROM ubuntu:latest

RUN apt-get update && apt-get install -y python3 python3-pip
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt

ENV AWS_SECRET_KEY=AKIAabcdef1234567890

EXPOSE 8080
CMD ["python3", "app.py"]

After — A Hardened Dockerfile#

Copy
# Pin an EXACT version, not "latest" — reproducible, and you know
# exactly what CVEs apply to this specific tag
FROM python:3.12.3-slim AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Multi-stage build — the final image doesn't carry build tools,
# compilers, or anything else only needed to INSTALL dependencies
FROM python:3.12.3-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY . .

# Create and switch to a non-root user — NEVER run as root
RUN useradd --create-home appuser
USER appuser

# Secrets are injected at RUNTIME (see Part 4), never baked into
# the image with ENV or hardcoded values

EXPOSE 8080
CMD ["python3", "app.py"]

What Changed, and Why#

A great, concrete interview line: "The two changes that matter most for real-world risk reduction are: never run as root, and never bake secrets into image layers — everything else (multi-stage builds, minimal base images) reduces attack surface, but those two specifically prevent a contained compromise from becoming a much bigger one."

---

Distroless and Minimal Base Images#

An even more aggressive hardening technique worth naming: distroless images (popularized by Google's gcr.io/distroless project) contain only your application and its direct runtime dependencies — no shell, no package manager, no OS utilities at all.

Diagram

Copy
# Example: a distroless final stage for a Go binary
FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o myapp .

FROM gcr.io/distroless/static-debian12
COPY --from=builder /app/myapp /myapp
USER nonroot:nonroot
ENTRYPOINT ["/myapp"]

The tradeoff worth naming: distroless images are harder to debug directly (you can't docker exec into them and poke around with a shell, because there isn't one) — teams typically compensate with better external observability (logs, traces, from the Observability tutorial series) instead of relying on live shell debugging inside production containers.

---

Container Runtime Isolation — What a Container Actually Is#

A frequently-tested conceptual question: "is a container as isolated as a virtual machine?" The honest answer is no, and understanding why is genuinely important.

Diagram

Plain-English explanation of the isolation mechanisms:

• Namespaces: give each container its own view of things like process IDs, network interfaces, and mounted filesystems — so it looks isolated from the container's own perspective, even though it's sharing the same underlying kernel.
• cgroups (control groups): limit how much CPU, memory, and other resources a container can actually consume — this is what prevents one noisy container from starving every other container on the same host (directly connects to the Saturation/USE concepts from the Monitoring Methodologies series).
• seccomp: restricts which low-level kernel system calls a container is even allowed to make, shrinking the attack surface available to an attacker who does gain code execution inside it.

Why this matters practically: because containers share the host kernel, a serious kernel vulnerability can theoretically allow a "container escape" — code running inside a container gaining access to the host or to other containers. This is exactly why running containers as non-root, using minimal images, and applying Pod Security Standards (below) all matter — they're layered defenses specifically because container isolation is real but genuinely weaker than a VM's.

---

Kubernetes Security — The Big Picture#

Kubernetes adds several more distinct security layers on top of container-level concerns.

Diagram

---

RBAC — Role-Based Access Control#

RBAC controls who (or what) can perform which actions on which Kubernetes resources. The core building blocks:

Diagram

• Role: namespace-scoped permissions (e.g., "can list/get pods, but only in the checkout namespace").
• ClusterRole: cluster-wide permissions (or reusable across namespaces).
• RoleBinding: grants a Role to a subject, within one namespace.
• ClusterRoleBinding: grants a ClusterRole to a subject, cluster-wide.

---

A Full Worked RBAC Example#

A realistic, minimal-privilege setup for a CI/CD deployment pipeline's ServiceAccount — showing exactly the kind of least-privilege thinking interviewers want to see.

Copy
# 1. Create a dedicated ServiceAccount for the CI/CD pipeline
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: checkout

---
# 2. Define a Role with ONLY the specific permissions needed —
#    NOT cluster-admin, NOT wildcard access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer-role
  namespace: checkout
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  # NOTE: deliberately NOT including "delete", "create" on secrets,
  # or access to OTHER namespaces — least privilege

---
# 3. Bind the Role to the ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: checkout
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: checkout
roleRef:
  kind: Role
  name: deployer-role
  apiGroup: rbac.authorization.k8s.io

Copy
# Verify what a ServiceAccount can actually do — a genuinely
# useful, commonly-used auditing command
kubectl auth can-i update deployments \
  --as=system:serviceaccount:checkout:ci-deployer \
  -n checkout
# yes

kubectl auth can-i delete secrets \
  --as=system:serviceaccount:checkout:ci-deployer \
  -n checkout
# no

The core principle worth stating explicitly: least privilege — grant exactly the permissions needed for the task, in exactly the namespace needed, and nothing more. A CI pipeline that only ever needs to update Deployments in one namespace should never be granted cluster-admin "just in case" — that turns a compromised CI credential (a very real, common attack vector) into a full cluster compromise instead of a contained one.

---

Pod Security Standards#

Kubernetes defines three built-in Pod Security Standards (replacing the older, now-deprecated PodSecurityPolicy) that control what privilege level a pod is allowed to run with.

Diagram

A pod spec enforcing the "Restricted" level's key settings directly:

Copy
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: myapp:1.2.3
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

Enforcing this at the namespace level via labels (built into modern Kubernetes, no extra tooling required):

Copy
kubectl label namespace checkout \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted

---

Network Policies — Zero Trust Inside the Cluster#

By default, Kubernetes allows any pod to talk to any other pod, across the entire cluster, with no restrictions at all. This is a genuinely important, commonly-tested fact — and a real security gap in the default configuration.

Diagram

A NetworkPolicy that only allows the checkout service to receive traffic from the api-gateway pods, and denies everything else:

Copy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-allow-from-gateway
  namespace: checkout
spec:
  podSelector:
    matchLabels:
      app: checkout-service
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api-gateway
      ports:
        - protocol: TCP
          port: 8080

Important, commonly-tested caveat: NetworkPolicies require a CNI (Container Network Interface) plugin that actually supports enforcing them — e.g., Calico, Cilium — the default kubenet networking in some setups does not enforce NetworkPolicies at all, meaning the YAML above would silently do nothing. Always verify the cluster's CNI plugin actually supports NetworkPolicy enforcement before relying on it — a genuinely practical gotcha worth naming.

---

Admission Controllers and OPA Gatekeeper#

Admission controllers intercept requests to the Kubernetes API before an object is actually created/updated, letting you enforce policy automatically — a direct, concrete application of the "security as code" idea from Part 1.

Diagram

A real OPA/Gatekeeper constraint (using its ConstraintTemplate + Constraint pattern) blocking the :latest tag:

Copy
# ConstraintTemplate: defines the REUSABLE policy logic (Rego)
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowedtags
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowedTags
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdisallowedtags
        violation[{"msg": msg}] {
          image := input.review.object.spec.containers[_].image
          endswith(image, ":latest")
          msg := sprintf("image '%v' uses the disallowed ':latest' tag", [image])
        }

---
# Constraint: APPLIES the policy above to actual resources
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDisallowedTags
metadata:
  name: no-latest-tag
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]

Why this matters practically, tying back to Part 1's "security as code" idea: instead of a wiki page saying "please don't use :latest tags," this policy is automatically, mechanically enforced on every single deployment attempt, cluster-wide, with zero reliance on any human remembering to check.

---

Secrets in Kubernetes — A Preview#

Kubernetes has a built-in Secret object, but it's genuinely important to know its real limitation: by default, Kubernetes Secrets are only base64-encoded, not encrypted, at rest in etcd (unless encryption-at-rest is explicitly configured).

Copy
# Base64 is NOT encryption — anyone with etcd access,
# or sufficient RBAC permissions, can trivially decode this
kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

This is exactly why Part 4 of this series covers dedicated secrets management tools (like HashiCorp Vault) in depth — native Kubernetes Secrets are a starting point, not a complete solution, for genuinely sensitive data.

---

Runtime Security: Falco#

Everything above is about preventing a problem before it happens. Falco (a CNCF project) represents a different, complementary layer: detecting suspicious behavior in a running cluster, in real time.

Diagram

Copy
# A simplified example Falco rule: alert if a shell is spawned
# inside any container in the 'production' namespace
- rule: Unexpected shell in production container
  desc: A shell was spawned inside a production container
  condition: >
    spawned_process and container
    and k8s.ns.name = "production"
    and proc.name in (bash, sh, zsh)
  output: >
    Shell spawned in production container
    (user=%user.name container=%container.name command=%proc.cmdline)
  priority: WARNING

Why this matters, tying back to the "defense in depth" idea: even with perfect RBAC, network policies, and hardened images, a genuinely novel attack could still get through — Falco's job is to catch it while it's happening, based on behavior (an unexpected shell spawning) rather than a known, pre-identified pattern, which is exactly the kind of "unknown unknown" detection that static scanning (SAST/SCA) fundamentally can't provide.

---

A Full Kubernetes Security Checklist#

A practical, memorable summary worth having ready for an interview:

Diagram

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A security review finds that a production namespace has zero NetworkPolicies defined. What's the actual risk, in concrete terms, and how would you fix it?

Answer: By default, every pod in that namespace (and potentially the whole cluster, depending on other namespaces' policies) can freely communicate with every other pod — meaning if any single pod is compromised (e.g., via a vulnerable dependency exploited through the internet-facing frontend), the attacker can immediately attempt to reach far more sensitive internal services (databases, internal APIs) with zero network-level friction. Fix: implement default-deny NetworkPolicies for the namespace, then add explicit, narrow allow rules only for the specific pod-to-pod communication paths actually required — verifying first that the cluster's CNI plugin supports NetworkPolicy enforcement at all.

Problem 2: A CI/CD pipeline's ServiceAccount currently has cluster-admin bound to it, because "it was easier to get everything working." What's the real risk, and what would a properly scoped alternative look like?

Answer: Cluster-admin means anyone who compromises the CI/CD pipeline's credentials (a genuinely common attack vector, e.g., via a compromised third-party GitHub Action or a leaked token) instantly has full control over the entire cluster — every namespace, every secret, the ability to delete anything. A properly scoped alternative: create a namespace-specific Role granting only the exact verbs (get/list/update/patch) on the exact resources (Deployments, and only Deployments) the pipeline actually needs to perform its job, bound via a RoleBinding scoped to only the namespace(s) it deploys to — verifiable directly with kubectl auth can-i against the ServiceAccount.

Problem 3: Falco fires an alert: "a shell was spawned inside a production checkout-service container." The on-call engineer initially dismisses it as a false positive since "nothing else looks wrong." What would you want them to actually do, and why?

Answer: Investigate before dismissing — this is exactly the kind of anomalous, behavior-based signal that prevention-focused tooling (scanning, RBAC) can't provide, and it's specifically the class of alert Falco exists to surface: something happening that doesn't match expected, normal behavior for that workload (a checkout service shouldn't normally have anyone spawning an interactive shell inside it at all). I'd want them to at minimum check recent access logs/audit logs for who/what triggered it, and treat it as a potential active compromise until proven otherwise, rather than assuming it's noise — dismissing genuinely novel, behavior-based security signals without investigation is exactly how real intrusions go undetected.

---

Summary and What's Next#

• Containers introduce security questions beyond application code: what's inside the image itself (base OS/runtime layers), and how much privilege does the running container actually have.
• Container image scanning (e.g., Trivy) specifically targets base OS/runtime CVEs that SAST/SCA (Part 2) don't cover.
• Dockerfile hardening essentials: pin exact versions (never :latest), use multi-stage builds and minimal/distroless base images, run as a non-root user, and never bake secrets into image layers.
• Containers are not as isolated as VMs — they share the host kernel via namespaces/cgroups/seccomp, which is why running as non-root and minimizing images both matter as layered defenses against a potential container escape.
• Kubernetes RBAC should always follow least privilege — a compromised, over-privileged ServiceAccount (especially in CI/CD) turns a contained incident into a full cluster compromise.
• Pod Security Standards (Restricted level: non-root, no privilege escalation, dropped capabilities, read-only filesystem) constrain what a running pod is even allowed to do.
• Kubernetes has no network isolation between pods by default — NetworkPolicies implement zero-trust, default-deny communication, but only work if the cluster's CNI plugin actually supports enforcing them.
• Admission controllers (OPA Gatekeeper) enforce policy automatically at deploy time — a direct, concrete implementation of "security as code."
• Native Kubernetes Secrets are only base64-encoded by default, not encrypted — a real gap that Part 4 addresses with dedicated secrets management tooling.
• Runtime security tools (Falco) provide a complementary, behavior-based detection layer for genuinely novel attacks that prevention-focused tooling can't catch by pattern-matching alone.

Continue to Part 4 (04-secrets-management-and-iam.md) for a full deep dive into secrets management (Vault, KMS), secret scanning, and least-privilege IAM design.