# Docker & Container Fundamentals — Part 1: Container Model, Images & Runtime Lifecycle

> **Series:** Docker & Container Fundamentals (1 of 4)
> **Part 1:** This file — Container Model, Images & Runtime Lifecycle
> **Part 2:** `02-dockerfiles-buildkit-and-image-delivery.md` — Dockerfiles, BuildKit & Image Delivery
> **Part 3:** `03-networking-storage-and-compose.md` — Networking, Storage & Docker Compose
> **Part 4:** `04-operating-containers-safely.md` — Operating Containers Safely & the Kubernetes Handoff
> **Questions:** `questions.md`

## Table of Contents

1. [Why Containers Changed Delivery](#why-containers-changed-delivery)
2. [Docker's Control Plane on One Host](#dockers-control-plane-on-one-host)
3. [Images, Layers, and Containers](#images-layers-and-containers)
4. [The Lifecycle Behind docker run](#the-lifecycle-behind-docker-run)
5. [Isolation Is a Boundary, Not a Virtual Machine](#isolation-is-a-boundary-not-a-virtual-machine)
6. [Image Pruning and Dangling Images](#image-pruning-and-dangling-images)
7. [Registries, Tags, and Digests](#registries-tags-and-digests)
8. [docker commit: Why It's Rarely the Right Tool](#docker-commit-why-its-rarely-the-right-tool)
9. [save/load vs. export/import: Two Different Artifacts](#saveload-vs-exportimport-two-different-artifacts)
10. [A Practical Runtime Workflow](#a-practical-runtime-workflow)
11. [Interactive Sessions: attach, exec, and TTY Allocation](#interactive-sessions-attach-exec-and-tty-allocation)
12. [Copying Files To and From a Container](#copying-files-to-and-from-a-container)
13. [Environment Variables and Runtime Configuration](#environment-variables-and-runtime-configuration)
14. [Naming, Labels, and Fleet Metadata](#naming-labels-and-fleet-metadata)
15. [Resources, Signals, and Shutdown](#resources-signals-and-shutdown)
16. [Adjusting a Running Container Without Recreating It](#adjusting-a-running-container-without-recreating-it)
17. [Observability and Health](#observability-and-health)
18. [Incident Workflow](#incident-workflow)
19. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
20. [Worked Practice Problems](#worked-practice-problems)
21. [Summary and What's Next](#summary-and-whats-next)

This chapter builds the vocabulary the rest of the series assumes: what an image and a container actually are, how the daemon and its underlying runtime components fit together, how a container moves through its lifecycle, and the everyday commands — inspecting, copying files, adjusting resources, running interactively — that make up most of an engineer's real day-to-day contact with Docker. Part 2 builds on this to construct images deliberately; Parts 3 and 4 build on it to wire multiple containers together and operate them safely in production.

## Why Containers Changed Delivery

Containers make the application artifact explicit. Instead of asking an operator to reconstruct a runtime from a ticket, a package list, and tribal knowledge, a container image captures the executable, libraries, files, default command, and runtime metadata required by one workload.

That is not a promise that every environment is identical. The host kernel, CPU architecture, network policy, credentials, and mounted data still matter. It is a promise that the application filesystem and process contract can move as one versioned unit.

Docker provides the common workflow around that unit: build an image, store it in a registry, create a container from it, attach deliberate configuration, and observe or remove the result. Docker's client sends API requests to `dockerd`; the daemon manages images, containers, networks, and volumes. [Docker's architecture](https://docs.docker.com/get-started/docker-overview/) is therefore a local control plane, not merely a command-line binary.

```mermaid
sequenceDiagram
    participant Dev as Developer or CI
    participant CLI as Docker client
    participant Daemon as dockerd
    participant Registry as Image registry
    participant Runtime as Container runtime
    Dev->>CLI: docker run registry.example.com/payments/api:1.8.4
    CLI->>Daemon: create or start request
    Daemon->>Registry: resolve tag and pull missing layers
    Registry-->>Daemon: image manifest and content layers
    Daemon->>Runtime: create namespaces, cgroups, filesystem, process
    Runtime-->>Daemon: container started or failed
    Daemon-->>CLI: container ID and status
```

### The operational problem it solves

Before containers, application delivery often mixed three change streams: application code, host configuration, and runtime dependencies. A package upgrade on one host could quietly change the behavior of a deployment that had not changed its own repository. Containers turn most of that runtime dependency surface into an immutable artifact that can be tested before promotion.

The useful analogy is a manufacturing release package: a factory does not send a product design and ask every plant to choose compatible parts. It sends a controlled bill of materials and an assembly specification. The container image is not the whole factory — the host and deployment platform still provide power, safety controls, and transport — but it sharply reduces variation in what arrives at the line.

> **From the Trenches:** A container image does not repair an unversioned dependency outside the image. If an application expects `DATABASE_URL`, a TLS private key, and a writable `/data` path, omitting those dependencies still fails at runtime. “It is containerized” is not an operational readiness criterion; the startup contract must be tested in an environment with the same configuration boundaries as production.

| Choose this | When it is appropriate | Avoid it when |
|---|---|---|
| Container image | A service needs repeatable packaging, promotion, and process isolation | The workload needs a full guest kernel or incompatible host architecture |
| Virtual machine | You need kernel-level isolation, a different OS kernel, or legacy host control | You are using it only to compensate for an unrepeatable application install |
| Bare process | A simple host-managed service has a deliberate immutable host image and low deployment complexity | Runtime dependencies drift between machines |

## Docker's Control Plane on One Host

The `docker` CLI is a client. It does not normally create namespaces or mount layers itself. It calls the Docker API exposed by `dockerd`, often over a local Unix socket. This distinction matters when troubleshooting permissions, remote daemons, and CI runners: whoever can control the daemon can usually ask it to create highly privileged host resources.

Docker Engine coordinates several responsibilities:

- image content acquisition and local storage;
- container metadata and lifecycle;
- network and volume objects;
- handoff to the low-level runtime that starts the process.

Inspect the endpoint before assuming which daemon a command targets:

```bash
docker context ls
docker context show
docker version
docker info
```

`docker version` separates client and server details. If the client is present but the server cannot be reached, investigate the selected context, daemon state, socket permissions, or remote endpoint before reinstalling the CLI.

### From dockerd to containerd to runc

`dockerd` itself does not create containers directly. Docker's architecture is layered: `dockerd` delegates container execution to `containerd`, a separate daemon responsible for image transfer, storage, and container lifecycle supervision; `containerd` in turn invokes `runc`, a low-level runtime that implements the [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec) and does the actual work of creating namespaces, cgroups, and the container's initial process.

```mermaid
sequenceDiagram
    participant CLI as docker CLI
    participant Dockerd as dockerd
    participant Containerd as containerd
    participant Shim as containerd-shim
    participant Runc as runc
    CLI->>Dockerd: docker run ...
    Dockerd->>Containerd: create container request
    Containerd->>Shim: spawn shim for this container
    Shim->>Runc: create and start (OCI runtime spec)
    Runc-->>Shim: container process started, runc exits
    Shim-->>Containerd: container running
    Containerd-->>Dockerd: container running
```

The `containerd-shim` process is the detail that explains a behavior every Docker user eventually notices: restarting or upgrading `dockerd` does not kill already-running containers. `runc` exits immediately after starting the container process — it is not a supervisor. The shim, spawned per container, becomes that process's actual parent and stays running independently of both `containerd` and `dockerd`, so a daemon restart only reconnects to already-running containers rather than restarting them.

```bash
ps -ef | grep containerd-shim
docker info --format '{{.ContainerdCommit.ID}}'
```

| Component | Responsibility | Survives a `dockerd` restart? |
|---|---|---|
| `dockerd` | API surface, image/network/volume management, orchestrates the rest | N/A — this is what's restarting |
| `containerd` | Image storage, container lifecycle supervision across the host | Yes — runs as an independent daemon |
| `containerd-shim` | Per-container parent process, holds the container's stdio | Yes — this is exactly why containers survive |
| `runc` | One-shot OCI runtime invocation that creates the container | N/A — exits immediately after container start |

> **From the Trenches:** A team scheduled Docker Engine upgrades during a maintenance window specifically to avoid any container downtime, having assumed (incorrectly) that upgrading `dockerd` would necessarily restart every running container. Testing on a staging host revealed containers stayed running and reachable throughout the entire `dockerd` upgrade, because the shim architecture keeps them alive independently — the "maintenance window" requirement turned out to be unnecessary caution for that specific upgrade path, though the team correctly kept it for upgrades that explicitly documented a containerd-level breaking change instead.

### Docker contexts: which daemon is a command actually talking to

A single `docker` CLI installation can target more than one daemon — a local Docker Engine, a remote host over SSH, or a rootless daemon (Part 4) running under the current user — and **contexts** are the mechanism that decides which one any given command actually reaches. Every command in this series implicitly runs against whichever context is currently active, which matters the moment more than one is configured on the same machine.

```bash
docker context ls
docker context create remote-staging --docker "host=ssh://deploy@staging.internal"
docker context use remote-staging
docker --context remote-staging container ls
```

`docker context use` changes the persistent default for every subsequent command in that shell; `--context` on a single invocation is a one-off override that doesn't change the default. Both are meaningfully safer than the older pattern of exporting `DOCKER_HOST` in a shell profile, because `context ls` gives an explicit, inspectable list of every configured target — an exported environment variable buried in a dotfile is easy to forget about entirely until a command targets the wrong host unexpectedly.

| Context type | Typical use | Risk if forgotten |
|---|---|---|
| Default local | Normal local development and testing | Low — this is almost always the expected target |
| Remote SSH-based | Managing a single remote host without a separate orchestration layer | A command intended for local testing accidentally runs against a shared remote host |
| Rootless (Part 4) | A hardened, unprivileged daemon on the same machine as a root-mode daemon | Confusing which daemon a given container actually belongs to when both exist side by side |

> **From the Trenches:** An engineer ran what they believed was a routine `docker system prune -a` against their local development environment, intending to reclaim disk space before a demo. Their shell had a leftover `remote-staging` context left active from debugging a deployment issue the previous day, and the prune command executed against the shared staging host instead — removing several image caches other engineers were actively relying on for fast rebuilds. The team's corrective action was adding the active context to the shell prompt itself, so which daemon a command targets is visible at a glance rather than only discoverable by running `docker context ls` proactively.

### Docker Desktop is not a Linux host daemon

On macOS and Windows, Docker Desktop commonly runs the Linux engine inside a managed virtual machine. The client experience is intentionally similar, but host networking, filesystem mounts, and kernel behavior are not identical to a native Linux Docker Engine. A bind mount that is fast on a Linux CI runner can be noticeably slower on a desktop VM because file sharing crosses an extra boundary.

> **From the Trenches:** Do not diagnose a Linux production incident by assuming `docker0` or a container IP is reachable from a macOS Docker Desktop host. Docker documents platform-specific networking behavior; use published ports and service DNS as the portable interfaces.

## Images, Layers, and Containers

An image is a read-only package. It is assembled from ordered filesystem layers and configuration metadata. A container is a runnable instance of that image plus a writable layer and runtime configuration such as environment variables, mounts, ports, resource limits, and the command to execute.

Layers explain two important behaviors:

1. A rebuild can reuse unchanged layers, which makes well-ordered Dockerfiles faster.
2. Removing a container discards changes in its writable layer unless data was stored in a volume or another external system.

```mermaid
flowchart TB
    Base["Base image layer\nOS files and runtime"] --> Deps["Dependency layer\napplication libraries"]
    Deps --> App["Application layer\nsource and assets"]
    App --> Image["Immutable image config\nentrypoint and defaults"]
    Image --> RW["Container writable layer\ntemporary runtime changes"]
    RW --> Process["PID 1 application process"]
    classDef artifact fill:#e6f6f2,stroke:#0f766e,color:#102a2a
    class Base,Deps,App,Image artifact
```

The writable layer is not a database. A container can write there, but its lifetime is tied to the container object. Treat it as scratch space for caches, temporary files, and process state that can disappear. Persistent state belongs in a named volume or an external managed service; Part 3 covers the decision in detail.

### The OCI image spec: what's actually inside an image

An image is not a monolithic file — it is a small JSON manifest pointing at a set of independently addressable pieces, standardized by the [OCI Image Format Specification](https://github.com/opencontainers/image-spec) so that any OCI-compliant tool (Docker, Podman, a registry, a scanner) can produce and consume the same artifact.

```bash
docker manifest inspect nginx:1.27
docker image inspect nginx:1.27 --format '{{json .RootFS.Layers}}'
```

```json
{
  "schemaVersion": 2,
  "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:abc..." },
  "layers": [
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:111...", "size": 28123456 },
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:222...", "size": 4021 }
  ]
}
```

The manifest references a **config** blob (the JSON describing environment variables, entrypoint, exposed ports, and the layer digest chain) and an ordered list of **layer** blobs — each one a compressed tarball representing a filesystem diff from the layer below it, including special `.wh.<name>` "whiteout" marker files that record a deletion rather than storing an empty directory. None of these pieces are Docker-specific: a registry stores and serves them as OCI blobs regardless of which OCI-compliant tool built or will run them, which is exactly what makes Part 2's cross-tool build and scan tooling interoperable in the first place.

### Copy-on-write storage: how layers become a filesystem

Pulling an image downloads its layer blobs; running a container requires assembling those layers, plus a new writable layer, into one coherent filesystem view. Docker's default storage driver on modern Linux, `overlay2`, does this with the kernel's OverlayFS: each image layer is a read-only directory, stacked in order, with the container's writable layer mounted as the topmost `upperdir`.

```bash
docker info --format '{{.Driver}}'
docker inspect catalog-api --format '{{.GraphDriver.Data}}'
```

Copy-on-write means a container never mutates an image layer directly — the first write to any file causes `overlay2` to copy that file up into the container's own writable layer, and only the copy is modified from then on. This is what makes launching a hundred containers from the same image cheap: every container shares the same read-only image layers on disk and only pays storage cost for files it actually changes.

```mermaid
flowchart TB
    subgraph Shared["Shared, read-only (one copy on disk regardless of container count)"]
        L1["Base layer"]
        L2["Dependency layer"]
        L3["Application layer"]
    end
    subgraph C1["Container A writable layer"]
        W1["Modified file (copied up on first write)"]
    end
    subgraph C2["Container B writable layer"]
        W2["Different modified file"]
    end
    L1 --> L2 --> L3
    L3 --> C1
    L3 --> C2
```

| Choose this mental model | When it matters |
|---|---|
| "A write copies the file up to this container's own layer" | Explains why editing a large file inside a container has a real, sometimes surprising, first-write cost |
| "Deleting a file adds a whiteout marker, not a smaller layer" | Explains why deleting a large file in a later Dockerfile `RUN` doesn't shrink the image — Part 2 covers the build-time consequence |
| "Layers are shared across every container using that image" | Explains why disk usage from `docker system df` doesn't grow linearly with container count for identical images |

> **From the Trenches:** A host running dozens of containers from the same base image showed far less disk usage under `docker system df` than a naive "layers × containers" estimate predicted, which briefly worried a capacity-planning exercise into believing usage was being under-reported. Understanding that image layers are genuinely shared, copy-on-write, read-only content — not duplicated per container — resolved the apparent discrepancy immediately once someone checked the storage driver's actual behavior instead of assuming per-container layer duplication.

### Image metadata and process semantics

The image contains defaults, not magic behavior. `ENTRYPOINT` establishes the executable contract; `CMD` supplies default arguments. A run command can override those values. The container remains alive while its primary process remains alive. A web server that daemonizes and exits may leave Docker with no foreground process to supervise, so the container stops even though the application attempted to start.

```bash
docker image inspect nginx:stable
docker run --rm nginx:stable nginx -T
docker run --rm --entrypoint /bin/sh alpine:3.21 -c 'id && uname -a'
```

Use `--rm` for disposable experiments so stopped containers do not accumulate. Do not use it for a workload whose logs or crash state you have not captured.

## The Lifecycle Behind docker run

`docker run` is convenient shorthand. Conceptually, Docker resolves an image, creates a container object, assigns a writable layer and requested resources, connects networking and mounts, then starts the configured process. If the image is absent locally, the daemon pulls the manifest and content layers from a registry first.

```bash
docker pull nginx:1.27
docker container create --name demo-web -p 127.0.0.1:8080:80 nginx:1.27
docker container start demo-web
docker container ls --filter name=demo-web
docker container logs --tail 50 demo-web
docker container stop --time 20 demo-web
docker container rm demo-web
```

Separating `create` from `start` is useful when debugging configuration. You can inspect a created but stopped container to verify its command, mounts, environment, and port bindings before the service starts.

```bash
docker container inspect demo-web
docker container stats --no-stream demo-web
docker container exec demo-web nginx -T
```

`docker exec` starts an additional process in an already-running container. It is excellent for incident diagnosis, but it is not a deployment mechanism. Changes made interactively are usually confined to the writable layer and disappear on replacement. Capture the corrective action in the image, configuration, or runbook.

### The full container state machine

`docker run` and `docker stop` show only two states in casual use, but Docker actually tracks a container through a small, well-defined state machine — knowing the full set explains behavior that otherwise looks inconsistent, like why a "stopped" container still shows up in `docker ps -a` with an exit code, or why `pause` freezes a container without stopping it.

```mermaid
stateDiagram-v2
    [*] --> Created: docker container create
    Created --> Running: docker container start
    Running --> Paused: docker container pause
    Paused --> Running: docker container unpause
    Running --> Exited: process exits, or docker stop
    Exited --> Running: docker container start (restart)
    Running --> Dead: daemon failed to stop/remove cleanly
    Exited --> [*]: docker container rm
    Dead --> [*]: docker container rm
```

```bash
docker container create --name demo nginx:1.27
docker container ls -a --filter name=demo --format '{{.Status}}'
docker container pause demo
docker container unpause demo
```

`Paused` is a distinct, less commonly used state worth knowing explicitly: it freezes every process in the container via the cgroup freezer, with no CPU scheduling at all, while keeping memory state intact — useful for a brief, deliberate freeze (a live migration handoff, a consistent-snapshot window) but not a substitute for `stop`, since a paused container still holds all of its resources and network connections open without doing anything with them.

`Dead` is a state most engineers never intentionally trigger — it indicates the daemon attempted to remove or stop a container and failed partway, leaving inconsistent state. A container stuck in `Dead` typically needs `docker container rm -f` and, if that also fails, an investigation into daemon or storage-driver health rather than repeated retries of the same removal command.

### Zombie processes and the --init flag

A container's PID 1 has a responsibility beyond just being the main process: on Linux, PID 1 is also responsible for "reaping" zombie processes — child processes that have exited but whose exit status hasn't been collected by a parent that called `wait()`. A normal init system (systemd, or a shell in an interactive session) does this automatically; an application binary running directly as PID 1 often does not, because most applications were never written with PID-1 responsibilities in mind.

```bash
docker run -d --name worker --init worker:1.4
```

The `--init` flag injects a minimal init process (`docker-init`, based on `tini`) as the container's actual PID 1, which then execs the application as its child — the application still behaves as the "main" process for all practical purposes (its exit code becomes the container's exit code), but zombie reaping and signal forwarding are now handled correctly underneath it.

| Symptom | Likely cause | Fix |
|---|---|---|
| Process count inside a long-running container grows slowly over time despite no obvious leak | The application spawns short-lived child processes it doesn't `wait()` on, accumulating zombies | Run with `--init`, or fix the application to reap its own children |
| A container doesn't respond to `docker stop` promptly | PID 1 doesn't forward `SIGTERM` to a child it spawned via a shell | `--init` forwards signals correctly; alternatively use exec-form entrypoints as covered in Part 4 |

> **From the Trenches:** A container running a script-based application via `CMD ["sh", "-c", "python worker.py"]` accumulated defunct zombie processes over weeks of operation, each one a residual few bytes in the process table, until the container's PID limit (Part 4 covers `--pids-limit`) was eventually exhausted by the accumulated zombies rather than by any real workload growth — a confusing symptom, since the application's own memory and CPU usage looked completely normal the entire time. Adding `--init` resolved it without any application code change, because the actual problem was PID 1 never reaping the short-lived subprocesses the application spawned internally.

### Restart policies are a limited tool

Restart policies can restart an exited process on one Docker host. They do not add scheduling across hosts, rollout control, capacity placement, or dependency orchestration. Use them for a bounded host-local service only after its startup failure mode is understood.

| Policy | Use it for | Operational caveat |
|---|---|---|
| `no` | One-shot jobs and explicit operator control | A failed service stays failed until action is taken |
| `on-failure` | Retryable process failures with a clear exit code | A bad configuration can create a restart loop |
| `unless-stopped` | A deliberately host-resident service | It does not replace health-aware orchestration |
| `always` | Rare cases needing restart after daemon restart | It can conceal a persistent application failure |

> **From the Trenches:** A rapidly restarting container can make dashboards look “alive” because a container exists, while users receive no successful requests. Alert on service-level health and restart rate, not merely the Docker container state.

## Isolation Is a Boundary, Not a Virtual Machine

Linux containers rely primarily on namespaces for isolation and cgroups for resource accounting and limits. Namespaces give a process a constrained view of identifiers such as processes, mounts, networks, and users. Cgroups govern resource usage such as CPU and memory. Docker makes these kernel primitives operationally accessible, but they do not create a separate kernel.

This has two consequences. First, containers generally start faster and consume less overhead than a full guest operating system. Second, a kernel vulnerability, overly broad capability, host mount, or access to the Docker socket changes the risk model substantially. Treat a container boundary as meaningful defense in depth, not as an excuse to run untrusted code with broad host access.

The Linux & Networking Fundamentals series explains namespaces and cgroups in kernel depth. Here, the Docker decision is practical: request only the privileges and host interfaces the process needs.

### The namespace types Docker uses, at a glance

Docker doesn't apply one undifferentiated "isolation" — it composes several distinct Linux namespace types per container, each hiding a different kind of host-wide identifier. Knowing which namespace is responsible for which illusion explains specific, otherwise-confusing behavior.

| Namespace | Hides/isolates | Explains |
|---|---|---|
| PID | Process IDs | Why a container's main process sees itself as PID 1, even though it has a different PID on the host |
| Mount | Filesystem mount points | Why a container has its own root filesystem view, distinct from the host's |
| Network | Network interfaces, routing, ports | Why a container gets its own IP and interfaces (Part 3 covers this in depth) |
| UTS | Hostname and domain name | Why `hostname` inside a container returns the container ID by default, not the host's hostname |
| IPC | System V IPC objects, POSIX message queues | Why two containers can't accidentally share a semaphore or shared-memory segment meant to be process-local |
| User | UID/GID mapping | Why rootless Docker (Part 4) can map a container's "root" to an unprivileged host UID |

```bash
docker run --rm --pid host alpine:3.21 ps aux | head -5
docker run --rm alpine:3.21 hostname
```

`--pid host` (rare, and worth treating as a specific, justified exception rather than a routine flag) shares the host's PID namespace instead of creating a new one — the diagnostic sidecar pattern from Part 3's networking debugging section uses exactly this, plus a shared network namespace, to see a target container's real process tree from outside it.

```bash
docker run --rm \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop ALL \
  --user 10001:10001 \
  --security-opt no-new-privileges:true \
  nginx:1.27
```

The command is a starting posture, not a universal drop-in. Nginx may need writable cache or PID paths depending on its configuration. Validate the workload, then make each permission explicit instead of reverting to `--privileged`.

## Image Pruning and Dangling Images

Rebuilding an image under the same tag doesn't delete the previous build's layers — it moves the tag to point at the new image and leaves the old one behind, now untagged. Docker displays these as `<none>:<none>` in `docker images`, commonly called "dangling" images.

```bash
docker images --filter dangling=true
docker image prune
docker image prune -a --filter "until=168h"
```

`docker image prune` removes dangling images only — safe to run routinely, since nothing can reference an untagged image by name anymore. `docker image prune -a` is a materially different, more aggressive operation: it removes every image with no container currently using it, tagged or not, which can delete an image a deployment script still expects to find locally for a future `docker run` even though nothing is running from it *right now*. The `--filter "until=168h"` age filter is what makes routine, scheduled pruning safe — it only touches images old enough that a legitimate rebuild-and-redeploy cycle would have already superseded them.

| Command | Removes | Safe to run unattended on a schedule? |
|---|---|---|
| `docker image prune` | Only dangling (untagged) images | Yes |
| `docker image prune -a` | Every image with no container currently running from it, tagged or not | Only with an age filter and clear understanding of what's expected to be pre-pulled |
| `docker system prune -a` | Images, stopped containers, unused networks, and the build cache | No — always review what it would remove first (`--dry-run` isn't native; inspect each resource type separately first) |

> **From the Trenches:** A disk-cleanup cron job ran `docker image prune -a` unconditionally every night on a host that pre-pulled several images specifically so a fast-response deployment script could skip the pull step during an incident. The prune job silently deleted those pre-pulled images every night, and the first genuine incident response after the cron job was introduced was slower than the runbook assumed, because "the image is already local" turned out to no longer be true. Switching the nightly job to plain `docker image prune` (dangling only) restored the pre-pull optimization's actual benefit.

## Registries, Tags, and Digests

A registry stores image repositories. A tag is a mutable human-friendly reference such as `1.8.4` or `stable`. A digest is a content-addressed immutable identifier such as `sha256:...`. Tags are useful for release workflows; digests are useful when an environment must run exactly the artifact that was approved.

```bash
docker build -t registry.example.com/payments/api:1.8.4 .
docker push registry.example.com/payments/api:1.8.4
docker buildx imagetools inspect registry.example.com/payments/api:1.8.4
docker pull registry.example.com/payments/api@sha256:REPLACE_WITH_APPROVED_DIGEST
```

Never invent a digest in a production command. Copy the approved digest from the registry or deployment metadata. The image reference structure and tag behavior are documented by Docker; a tag can be moved, so it is not evidence of immutable provenance.

> **From the Trenches:** “latest” is an environment-dependent instruction, not a release identifier. Two hosts can run different bytes under the same tag when they pull at different times. An incident rollback is slow and ambiguous when nobody can state the exact digest that was live.

### Multi-platform manifest lists

A single tag can actually resolve to different bytes depending on the pulling machine's CPU architecture. What `docker pull registry.example.com/catalog/api:2.6.0` retrieves is often not a single image manifest but a **manifest list** (also called an image index) — a small JSON document listing several architecture-specific manifests, letting the daemon pick the one matching its own platform automatically.

```bash
docker manifest inspect registry.example.com/catalog/api:2.6.0
docker buildx imagetools inspect registry.example.com/catalog/api:2.6.0
```

```json
{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    { "platform": { "architecture": "amd64", "os": "linux" }, "digest": "sha256:aaa..." },
    { "platform": { "architecture": "arm64", "os": "linux" }, "digest": "sha256:bbb..." }
  ]
}
```

This is why the same `docker pull` command run on an Intel CI runner and an Apple Silicon developer laptop can silently retrieve genuinely different image bytes under an identical tag — both are correct, expected behavior, not a caching bug. Part 2's multi-architecture build coverage is what actually produces this manifest list; this chapter's concern is simply recognizing that a tag and a specific set of bytes are not a one-to-one mapping once more than one platform is involved.

> **From the Trenches:** A "the image behaves differently between my laptop and CI" bug report turned out not to be an environment configuration difference at all — the developer's Apple Silicon laptop and the x86 CI runner were pulling different manifests from the same manifest list, and one architecture's build had a genuine, unrelated bug the other didn't share. Recognizing that `docker pull <same tag>` doesn't guarantee identical bytes across architectures redirected the investigation to the actual per-architecture build difference within a few minutes, instead of continuing to search for a phantom environment misconfiguration.

## docker commit: Why It's Rarely the Right Tool

`docker commit` creates a new image from a running or stopped container's current filesystem state — technically simple, and worth understanding precisely because it undermines nearly everything the rest of this series is built on.

```bash
docker exec -it demo-web sh -c 'apt-get update && apt-get install -y curl'
docker commit demo-web demo-web:patched
```

The resulting `demo-web:patched` image has no Dockerfile, no build history, no record of what command produced it beyond "whatever happened to be true about this specific container's filesystem at commit time." It cannot be rebuilt, cannot be meaningfully code-reviewed, and cannot be reproduced from source — every property Part 2 spent an entire chapter establishing for a trustworthy build artifact is absent by construction.

| Choose this | When it is appropriate | Avoid it when |
|---|---|---|
| `docker commit` | A one-off, throwaway debugging snapshot you'll discard, or genuinely exploratory local experimentation | Anything intended to run again, be shared with a teammate, or reach any shared environment |
| A proper Dockerfile change + rebuild | Any real fix, dependency addition, or configuration change | Never really — this is the default, not a special case |

The one narrow legitimate use is capturing a container's exact broken state for offline forensic analysis during an incident (per Part 4's incident guidance) — even then, the resulting image is evidence to inspect, not something to redeploy.

> **From the Trenches:** A production incident was "fixed" under time pressure by exec-ing into the running container, applying a manual patch, and committing the result as the new "production image" to unblock a release deadline. Six months later, nobody could explain why that specific version behaved differently from every subsequent Dockerfile-built image, because the commit had captured an undocumented, non-reproducible filesystem state with no corresponding source change — the team eventually had to treat the entire image lineage from that point as untrusted and rebuild from a known-good Dockerfile to regain a reproducible baseline.

## save/load vs. export/import: Two Different Artifacts

Docker has two pairs of commands that sound similar and are frequently confused, but operate on genuinely different things: an **image** (with its full layer history and metadata) versus a **container's current filesystem** (a flat snapshot, with history discarded).

```bash
docker save nginx:1.27 -o nginx-image.tar
docker load -i nginx-image.tar

docker export demo-web -o demo-web-rootfs.tar
docker import demo-web-rootfs.tar demo-web:snapshot
```

`save`/`load` round-trip a real, OCI-compliant image — layers, manifest, config, and history all intact — the same artifact this chapter's OCI image spec section described. `export`/`import` instead flatten a *running or stopped container's* current filesystem into a single tarball with no layer history and no image config metadata (`ENTRYPOINT`, `CMD`, exposed ports, and environment defaults are all lost) — `import` creates a brand-new single-layer image from that flat filesystem, not a faithful reconstruction of the original image.

| Choose this | When it is appropriate | Avoid it when |
|---|---|---|
| `save`/`load` | Transporting a real image between hosts without a registry (the air-gapped delivery pattern Part 2 covers in depth) | You need to inspect or modify a running container's accumulated filesystem state |
| `export`/`import` | Rare cases needing a flat filesystem snapshot with no layer history, disconnected from any image lineage | Anything that needs to preserve the original image's entrypoint, environment, or build provenance — those are silently dropped |

> **From the Trenches:** An engineer used `docker export`/`docker import` to "back up" a running container before an experiment, reasoning it was equivalent to a `docker commit` or a proper image save. The imported result ran, but with none of the original image's environment variables or entrypoint configuration — because `export`/`import` never carried that metadata in the first place — and the experiment's rollback plan had to be redone from the actual source image and its build configuration, the artifact that should have been the backup all along.

## A Practical Runtime Workflow

Use a deliberate sequence for a new service:

1. Start a named container with an explicit image version and host binding.
2. Inspect the exact command, image, ports, mounts, and health output.
3. Read logs from the primary process.
4. Verify the service through its published interface.
5. Capture durable configuration in source-controlled Compose or deployment manifests.

```bash
docker run -d \
  --name catalog-api \
  --publish 127.0.0.1:8080:8080 \
  --env-file ./catalog-api.env \
  --memory 512m \
  --cpus 1.0 \
  registry.example.com/catalog/api:2.4.1

docker container inspect catalog-api --format '{{json .NetworkSettings.Ports}}'
docker container logs --since 10m catalog-api
docker container stats --no-stream catalog-api
```

Binding to `127.0.0.1` deliberately exposes the port only on the host loopback interface. Publishing to all host interfaces is a security decision, not the default answer for every backing service.

### What to record in a runbook

| Question | Evidence to record |
|---|---|
| What bytes run? | Registry repository, immutable digest, build provenance |
| What starts the process? | Entrypoint, command, user, working directory |
| Where is state? | Volume, database, object store, or explicit ephemeral path |
| How is it reachable? | Published port, reverse proxy, network, DNS name |
| What proves it is healthy? | Endpoint, expected response, log line, SLO signal |
| How is it stopped safely? | Grace period, draining behavior, shutdown signal |

## Interactive Sessions: attach, exec, and TTY Allocation

Three commands let a human interact with a container's process, and they are not interchangeable — confusing them produces symptoms ranging from a garbled terminal to accidentally sending input to a production process.

```bash
docker run -it --name debug-shell alpine:3.21 sh
docker attach debug-shell
docker exec -it debug-shell sh
```

`docker run -it` allocates a pseudo-TTY (`-t`) and keeps stdin open (`-i`) for the container's *own* PID 1 process — appropriate for a container whose primary purpose is an interactive session. `docker attach` connects to that same PID 1 process's existing stdio streams after the fact; critically, it does not start a new process, and typing `exit` inside an attached session terminates the container's main process, not just your terminal connection — a mistake that has stopped production containers more than once. `docker exec -it` starts a brand-new process inside the container's namespaces, independent of PID 1, which is why `exit` there only ends that one exec session and leaves the main process untouched.

| Command | Connects to | Typing `exit` does |
|---|---|---|
| `docker run -it` | A brand-new container's PID 1, at creation | Stops the container (PID 1 exits) |
| `docker attach` | An existing container's PID 1, already running | Stops the container (PID 1 exits) — the most common accidental-outage mistake |
| `docker exec -it` | A brand-new, separate process inside a running container | Ends only that exec session; PID 1 and the container keep running |

> **From the Trenches:** An engineer intending to check logs on a production container ran `docker attach` out of habit from local development, then typed `exit` to leave — which sent the exit to the container's actual PID 1 and stopped a live production service, rather than merely disconnecting a terminal. The corrective action was a team-wide convention: `attach` is reserved for the rare case of genuinely needing to interact with PID 1's own stdio, and `exec -it <container> sh` is the default for any exploratory or diagnostic session, specifically because it can never accidentally terminate the main process.

### Detached mode and log visibility

`docker run -d` starts a container without attaching to it at all — the common production pattern, since a real service shouldn't depend on an open terminal session to keep running. This makes `docker logs` the correct tool for observing a detached container's output, not `attach`, which for a long-running production service is rarely the right choice even when available.

```bash
docker run -d --name catalog-api catalog-api:2.6.0
docker logs --follow --tail 100 catalog-api
```

## Copying Files To and From a Container

`docker cp` moves files between the host and a container's filesystem directly, bypassing both the image build process and any mounted volume — useful for a specific diagnostic or one-off need, but easy to reach for as a substitute for a mechanism that should really be a proper volume, bind mount, or image rebuild.

```bash
docker cp catalog-api:/app/logs/error.log ./error.log
docker cp ./patched-config.yaml catalog-api:/app/config.yaml
```

Both directions have the same fundamental property as `docker exec`-driven changes: anything copied *into* a container lands only in that container's writable layer (Part 1's copy-on-write model) and disappears the moment the container is replaced, exactly like a manual `exec`-based edit. Copying *out* of a container is genuinely useful and low-risk — pulling a crash log or core dump for offline analysis during an incident is a legitimate, common use.

| Direction | Reasonable use | Anti-pattern |
|---|---|---|
| Host → container | Never, for anything that should survive a restart or be reproducible | "Patching" a running container's config or code instead of fixing the image or a real bind mount |
| Container → host | Pulling a diagnostic artifact (a log file, a core dump, a heap snapshot) during an incident | Using it as a routine backup mechanism instead of Part 3's proper volume backup workflow |

> **From the Trenches:** A recurring production issue was "fixed" repeatedly by an on-call engineer copying an updated configuration file into the running container with `docker cp`, each time believing it was a temporary stopgap until a proper fix landed. Because the fix worked immediately and the underlying Dockerfile/config-management change was never prioritized, the same manual `docker cp` had been silently repeated after every container replacement for months — an invisible, undocumented dependency that a new team member discovered only when a routine redeploy "mysteriously" reintroduced the original bug.

## Environment Variables and Runtime Configuration

An image's `Dockerfile` can declare default environment variables with `ENV`; a running container can override any of them at `docker run` time — the same override relationship Part 2 covered for `ARG` vs `ENV` at build time, now at the runtime layer.

```dockerfile
ENV LOG_LEVEL=info
ENV PORT=8080
```

```bash
docker run -d --name catalog-api -e LOG_LEVEL=debug -e PORT=9090 catalog-api:2.6.0
docker run -d --name catalog-api --env-file ./catalog-api.env catalog-api:2.6.0
docker container inspect catalog-api --format '{{json .Config.Env}}'
```

`-e` overrides one variable at a time; `--env-file` loads many from a file in one flag, useful once a service has more than a handful of configuration values. Both are visible in full via `docker container inspect` — this is the same non-secret-appropriate visibility Part 3 and Part 4 flagged for Compose `environment:` blocks, worth restating here at the single-container level: an `-e` flag is fine for a log level or a feature flag, and wrong for a credential, which belongs in the mounted-file mechanisms those later chapters cover instead.

### Precedence, made explicit

| Source | Precedence | Where it's declared |
|---|---|---|
| `-e VAR=value` at `docker run` | Highest — overrides everything below | The `docker run` command line |
| `--env-file` | Overrides the image default, overridden by `-e` for the same variable | A file passed at `docker run` |
| `ENV` in the Dockerfile | Lowest — the image's own default | The Dockerfile, baked into the image |

A variable set by both `--env-file` and a same-named `-e` flag on the same command resolves to the `-e` value — worth confirming explicitly with `docker container inspect` rather than assuming, especially once a deployment script combines both mechanisms and it's no longer obvious at a glance which one "wins" for a given variable.

> **From the Trenches:** A staging environment's `LOG_LEVEL` refused to change no matter how many times an engineer edited the environment file the deployment script referenced, because that same script also passed an unrelated, older `-e LOG_LEVEL=info` flag left over from an earlier debugging session — and the `-e` flag silently took precedence over the file every single time. `docker container inspect --format '{{json .Config.Env}}'` on the actually-running container, rather than re-reading the source files and assuming they were authoritative, revealed the conflicting flag within seconds.

## Naming, Labels, and Fleet Metadata

A single-host deployment with a handful of containers can get by with memorable names alone. Anything beyond that needs a deliberate metadata convention — Docker's own tooling has no opinion on what a "service," "environment," or "owner" means, so that structure has to come from labels applied consistently.

```bash
docker run -d --name catalog-api-prod \
  --label app=catalog-api \
  --label env=production \
  --label team=platform \
  --label version=2.6.0 \
  catalog-api:2.6.0

docker container ls --filter label=env=production --filter label=team=platform
docker container ls --filter label=app=catalog-api --format 'table {{.Names}}\t{{.Label "version"}}'
```

Labels turn ad hoc container names into queryable fleet metadata — `docker container ls --filter label=...` answers "which containers belong to this team" or "which containers are still running the previous version" without relying on naming convention discipline holding perfectly across every engineer and script that ever starts a container. `LABEL` instructions baked into the image (Part 2) and `--label` flags applied at run time (here) compose together; the image can declare static facts about itself (`org.opencontainers.image.revision`, from Part 2's OCI annotation guidance) while runtime labels capture deployment-specific facts (which environment, which specific host role) that don't belong in the image itself.

| Metadata mechanism | Set when | Good for |
|---|---|---|
| `LABEL` in the Dockerfile | Build time | Facts about the image itself — version, revision, maintainer, OCI annotations |
| `--label` at `docker run` | Run time | Facts about this specific deployment — environment, team, instance role |
| Container name | Run time | A single human-memorable identifier — doesn't scale as a query mechanism the way labels do |

> **From the Trenches:** An incident review needed to quickly answer "which containers across this host are still running the vulnerable image version" during a CVE response, and the only available identifying information was container names following an inconsistent, engineer-specific naming convention that had drifted over two years — some included a version number, most didn't, none used a queryable format. Answering the question took a slow, manual `docker inspect` sweep across every container instead of one `docker container ls --filter label=version=...` call. The team's post-incident action item was a mandatory `--label version=` (and `app=`, `env=`) convention enforced by their deployment tooling, not left to individual habit.

## Resources, Signals, and Shutdown

A container is still a process. Resource controls protect a shared host, but they do not replace capacity planning. Set a measured memory ceiling, CPU budget, and PID limit, then validate the resulting latency and failure behavior under load.

```bash
docker run -d --name invoice-api \
  --memory 768m --memory-reservation 512m --cpus 1.5 --pids-limit 256 \
  registry.example.com/invoice/api:3.2.0
docker container stats --no-stream invoice-api
```

`docker stop` sends `SIGTERM`, waits for its grace period, then uses `SIGKILL` if required. Applications must stop accepting new work, drain in-flight requests, and exit before that deadline. `SIGKILL` cannot be handled.

> **From the Trenches:** Restarting a process can make a dashboard look healthy while every request fails during repeated startup. Alert on successful service behavior and restart rate, not merely a running container state.

| Control | Purpose | Verify with |
|---|---|---|
| Memory limit | Bound a memory leak's blast radius | OOM events, RSS, latency |
| CPU limit | Protect neighbor workloads | Throttling and request latency |
| PID limit | Prevent runaway process creation | Process count and exit behavior |
| Stop timeout | Allow orderly shutdown | Draining and completed requests |

## Adjusting a Running Container Without Recreating It

Most configuration changes covered so far in this chapter — a new environment variable, a different image version — genuinely require recreating the container, since they're baked in at creation time. A narrow set of runtime properties are the exception: `docker update` can change a live container's resource limits and restart policy in place, without stopping or recreating it.

```bash
docker update --memory 1g --memory-swap 1g --cpus 1.5 catalog-api
docker update --restart unless-stopped catalog-api
```

This is genuinely useful for an urgent, temporary adjustment during an incident — raising a memory ceiling immediately while a proper fix is prepared, without the brief availability gap a full recreation would cause. It is not a substitute for making the corrected value permanent in whatever source (a Compose file, a deployment script) originally created the container: a `docker update` change lives only on that one running container and is lost the next time it's recreated from its original definition, silently reverting exactly when the change might be needed most.

| Property | Changeable via `docker update`? | Requires recreation instead |
|---|---|---|
| Memory/CPU limits | Yes | — |
| Restart policy | Yes | — |
| Environment variables | No | New container from updated configuration |
| Mounted volumes/networks | No | New container from updated configuration |
| The image itself | No | New container from the new image |

> **From the Trenches:** A container was hitting its memory limit repeatedly during a traffic spike, and the on-call engineer used `docker update --memory 2g` to immediately relieve the pressure while a proper capacity review was scheduled for the following week. The capacity review happened, concluded the higher limit was correct, and updated the team's documentation — but nobody updated the actual Compose file, so the next routine redeploy silently reverted the container to its original, too-low limit and reintroduced the exact incident the review had "fixed." The lesson generalized into a standing checklist item: any `docker update` applied during an incident gets a linked follow-up ticket to make the same change in the source configuration, closed only when the source itself reflects it.

## Observability and Health

Containers should write structured application events to stdout and stderr. `docker logs` is useful for one-host diagnosis, but centralized collection, correlation, retention, and deployment identity are required in production.

```bash
docker container logs --timestamps --tail 100 catalog-api
docker container logs --follow catalog-api
docker events --filter container=catalog-api
docker inspect --format '{{json .State.Health}}' catalog-api
```

A Docker `HEALTHCHECK` records container health metadata. It must test the correct boundary: a liveness check proves the process can run, while readiness should prove the service can accept its intended work. Do not make every probe execute an expensive dependency transaction.

```dockerfile
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD wget --spider --quiet http://127.0.0.1:8080/healthz || exit 1
```

### Exit codes and restart counts as a first triage signal

Before reading a single log line, a container's exit code and restart count already narrow down the likely failure category — cheap, structured signals worth checking first in any triage, including the incident workflow later in this chapter.

```bash
docker container inspect catalog-api --format '{{.State.ExitCode}} {{.RestartCount}}'
```

| Exit code | Typical meaning | First thing to check |
|---|---|---|
| `0` | Clean exit — the process finished intentionally | Was this a one-shot job that's supposed to exit, or a service that shouldn't have exited at all? |
| `1` | Generic application error | Application logs, immediately before the exit |
| `137` | `SIGKILL` (128 + 9) — often an OOM kill | `docker events --filter event=oom`; compare against the memory limit |
| `143` | `SIGTERM` (128 + 15) — a requested, graceful stop | Confirm this was an intentional `docker stop`, not an unexpected external signal |
| `126`/`127` | The command itself couldn't be executed, or wasn't found | The image's `ENTRYPOINT`/`CMD` and whether the referenced binary actually exists at that path |

A high `RestartCount` on a container using an `on-failure` or `always` policy is itself a symptom worth investigating even if the container currently shows as running — it means the process has been dying and coming back repeatedly, which per this chapter's earlier restart-policy guidance can hide behind a dashboard that only checks "is a container running" rather than "is it succeeding."

### HEALTHCHECK command forms and inheriting from a base image

`HEALTHCHECK` supports two command forms with a meaningful difference in how failures surface, plus a way to explicitly opt a container out of a check inherited from its base image.

```dockerfile
HEALTHCHECK CMD ["curl", "-f", "http://127.0.0.1:8080/healthz"]
HEALTHCHECK CMD curl -f http://127.0.0.1:8080/healthz || exit 1
```

The exec form (`CMD [...]`, no shell involved) runs the binary directly — if `curl` itself succeeds but the response is a non-2xx status, `curl -f` still needs to be told to fail on that explicitly, since there's no shell present to interpret an `||` fallback in this form at all. The shell form (`CMD curl ... || exit 1`, no brackets) runs through `/bin/sh -c`, which does let you chain a fallback exit code — but inherits the same PID-1-signal-forwarding caveat from earlier in this chapter, since the health check itself now runs as a subshell rather than a direct exec.

```dockerfile
HEALTHCHECK NONE
```

A derived image (one using `FROM some-base-image-with-a-healthcheck`) inherits that base image's `HEALTHCHECK` automatically unless explicitly overridden — `HEALTHCHECK NONE` disables an inherited check that doesn't make sense for the derived image's actual purpose, which matters more than it might seem: a base image's health check assumptions (a specific port, a specific endpoint path) don't automatically hold for everything built on top of it.

> **From the Trenches:** A team's internal base image shipped a `HEALTHCHECK` testing port 8080, and a service built from it that legitimately listened on a different port inherited that check unmodified — the derived image reported unhealthy in every environment, and the confusing part was that nothing in the derived image's own Dockerfile mentioned a health check at all, since it had never explicitly declared one, only inherited one silently. `docker image history` on the derived image, not just reading its own Dockerfile, was what revealed the inherited instruction's actual origin.

## Incident Workflow

Preserve evidence before replacing a failed container. Work from user symptoms through service reachability, logs, configuration, resource state, and dependencies.

### A quick-reference table of `docker inspect` recipes

`docker inspect` returns a large JSON document per container or image; `--format` with a Go template extracts exactly the field a specific investigation needs, avoiding the need to pipe through `jq` for common questions. Keeping a short reference of the recipes actually used during an incident, rather than re-deriving the template syntax under pressure, is a small thing that measurably speeds up triage.

```bash
docker inspect catalog-api --format '{{.State.Status}}'
docker inspect catalog-api --format '{{.State.ExitCode}} {{.RestartCount}}'
docker inspect catalog-api --format '{{.Config.Image}}'
docker inspect catalog-api --format '{{json .Config.Env}}'
docker inspect catalog-api --format '{{json .HostConfig.Memory}}'
docker inspect catalog-api --format '{{json .NetworkSettings.Networks}}'
docker inspect catalog-api --format '{{json .Mounts}}'
```

| Question during an incident | Recipe |
|---|---|
| Is it running, and what's its exact state? | `{{.State.Status}}` |
| How did it die, and has this happened before? | `{{.State.ExitCode}} {{.RestartCount}}` |
| What image and digest is actually running? | `{{.Config.Image}}` |
| What configuration did it actually start with? | `{{json .Config.Env}}` |
| What resource ceiling is it running under? | `{{json .HostConfig.Memory}}` |
| What networks and mounts does it actually have? | `{{json .NetworkSettings.Networks}}` / `{{json .Mounts}}` |

Each of these answers a question this chapter has already covered in isolation — the exit-code table, the resource-limit discussion, the environment-precedence section — this table exists purely to collect the actual commands in one place, since an incident is the wrong moment to be reconstructing Go template syntax from memory.

```mermaid
flowchart TD
    A["User symptom"] --> B["Endpoint and error rate"]
    B --> C["Logs and exit state"]
    C --> D["Image, command, mounts, ports"]
    D --> E["CPU, memory, PID, disk pressure"]
    E --> F["DNS and dependency checks"]
    F --> G["Source-controlled correction"]
```

```bash
docker container ls -a --no-trunc
docker container inspect catalog-api --format '{{json .State}}'
docker container logs --since 15m --timestamps catalog-api
docker container stats --no-stream catalog-api
```

Record the image digest, exit code, restart count, timestamps, and relevant configuration structure. Never paste raw `docker inspect` output into an incident channel without checking it for secrets.

### Failure classification before recovery

Classify the failure before selecting recovery. A configuration failure repeats predictably on every restart. A transient dependency failure may recover with bounded retries. A capacity failure needs admission control, scaling, or resource tuning. A corrupted image or registry reference needs a known-good artifact, not a host-level workaround.

| Failure class | Evidence | Correct response |
|---|---|---|
| Startup configuration | Immediate non-zero exit and missing setting | Correct source-controlled configuration |
| Dependency outage | Timeouts after a healthy process starts | Restore dependency or apply bounded retry behavior |
| Resource exhaustion | OOM, throttling, or PID-limit events | Measure load and adjust capacity deliberately |
| Image regression | Failure begins at a new digest | Roll back to a verified digest and investigate |

### The immutable recovery rule

Do not repair a production container by installing packages, editing application files, or changing its process interactively. Those actions cannot be reviewed, reproduced, or reliably carried into the next replacement. Use the incident to identify the missing build, configuration, or deployment control, then make that correction in the appropriate repository.

This discipline is what turns containers from a convenience wrapper into a reliable delivery unit: each replacement is created from declared inputs, and each investigation leaves a durable improvement behind.

## Common Mistakes and Interview Traps

| Mistake or claim | Why it is wrong | Better answer |
|---|---|---|
| “A container is a small VM.” | Containers share the host kernel and differ materially in isolation and lifecycle. | A container is an isolated process plus filesystem and runtime configuration. |
| “An image is a running application.” | Images are immutable artifacts; containers are runnable instances. | Build or pull an image, then create and start a container from it. |
| “Container data persists automatically.” | The writable layer is removed with the container. | Persist intentional state in a volume or external service. |
| “Restart always makes a service reliable.” | Restart policies do not provide health-aware orchestration or root-cause correction. | Monitor the service and use an orchestrator when placement and rollout control are needed. |
| “Tags identify exact releases.” | Tags can be repointed. | Record the image digest for audited promotions. |
| “`docker attach` is a safe way to check on a running container.” | Typing `exit` in an attached session terminates the container's PID 1, not just the terminal connection. | Use `docker exec -it <container> sh` for exploratory or diagnostic sessions instead. |
| “`docker commit` is a fine way to patch a container quickly.” | The result has no Dockerfile, no build history, and can't be reproduced or reviewed. | Make the fix in the Dockerfile and rebuild; reserve `commit` for throwaway experiments or forensic snapshots. |
| “An `-e` flag and an `--env-file` entry for the same variable are equally authoritative.” | `-e` always overrides a same-named `--env-file` entry, which itself overrides the Dockerfile's `ENV` default. | Confirm the actually-applied value with `docker container inspect`, not by re-reading source files alone. |
| “`docker image prune -a` is a safe routine cleanup command.” | It removes every image with no container currently using it, including ones a script expects to find pre-pulled. | Use plain `docker image prune` (dangling only) for unattended scheduled cleanup, or add a generous age filter to `-a`. |

## Worked Practice Problems

### 1. A service exits immediately after docker run. What do you check first?

Start with `docker container logs <name>` and `docker container inspect <name>` to identify the exit code, command, and configuration. Confirm that the configured entrypoint launches the intended foreground process. Then check missing environment variables, a bad bind mount, an unavailable dependency, or a port conflict. Do not immediately add a restart policy: that converts a clear failure into a noisy loop.

### 2. A team says a containerized PostgreSQL database has durable data because the image is versioned. Correct them.

An image version preserves database software, not runtime writes. The database files are created in the container writable layer unless a volume or external storage target is mounted. Replacing the container replaces that writable layer. Define a named volume or managed database service, test restore behavior, and document the backup and recovery objective.

### 3. A CVE response requires knowing whether production runs the fixed image. What evidence is sufficient?

The deployment record must identify the repository and immutable digest that is running, then compare that digest with the remediated build's digest and SBOM or vulnerability scan evidence. A tag alone is insufficient because it can move. Roll forward by promoting the verified artifact and confirm running containers reference that digest.

### 4. A container that was working fine yesterday exited overnight with no obvious application error in its logs. Where do you look before assuming the application itself is at fault?

Check the exit code and restart count first with `docker container inspect --format '{{.State.ExitCode}} {{.RestartCount}}'` — an exit code of `137` points to a `SIGKILL`, commonly an OOM kill, which `docker events --filter event=oom` can confirm independently of anything the application itself logged (a process killed by the kernel usually has no chance to log its own death). This reframes the investigation from "why did the application silently fail" to "why did memory usage cross the configured limit," a meaningfully different and more tractable question.

### 5. Two engineers on the same team consistently get different results running the same `docker pull` command against the same tag — one gets a working image, the other reports a bug that the first can't reproduce. Both are on the same network and registry. What's a likely explanation that has nothing to do with caching?

If the two engineers are on different CPU architectures (one on an Apple Silicon laptop, one on an Intel machine, for instance), the tag may resolve to a manifest list with genuinely different per-architecture builds, and a bug present in only one architecture's build would reproduce exclusively on that architecture. Confirm this with `docker manifest inspect <tag>` to see whether multiple platform-specific manifests exist, and check `docker image inspect --format '{{.Architecture}}'` on each engineer's locally pulled image before assuming the difference is environmental configuration or a caching artifact.

### 6. A junior engineer proposes using `docker cp` to push a hotfix configuration file into every production container as an immediate, temporary stopgap while a proper fix goes through review. What's the risk, and what should happen instead?

The immediate risk is exactly what Part 1's `docker commit` and `docker cp` guidance warns about: a host-filesystem change that lives only in each container's writable layer, invisible to version control, and silently lost the next time any of those containers is replaced — which means the "temporary" stopgap either has to be manually reapplied after every future deployment (a growing, undocumented maintenance burden) or quietly disappears and the original bug reappears without explanation. The correct response under real time pressure is still to make the fix in the actual configuration source (the image, a mounted config file, or an environment variable) and deploy it through the normal pipeline — even an expedited version of that pipeline is safer than a change that exists nowhere durable.

## Summary and What's Next

Docker packages an application into a versioned image and turns that image into a container with explicit runtime configuration. The operational discipline is to distinguish immutable image content from mutable runtime state, treat the daemon as privileged infrastructure, and make ports, mounts, identities, and resources deliberate. Underneath the CLI, `dockerd` delegates to `containerd` and `runc` through a shim that's what actually keeps containers alive across a daemon restart; an image is a small OCI manifest pointing at content-addressed, copy-on-write layers shared across every container using it; and the tools that touch a running container directly — `attach`, `exec`, `cp`, `commit` — each have a specific, narrow legitimate use and a much more common way to misuse them as a substitute for a real, reproducible fix.

None of this chapter's tools replace the discipline of treating the image and its declared configuration as the source of truth. A container you can `exec` into, patch, and walk away from is not more reliable than one you can't — it's a container whose actual running state has quietly diverged from anything checked into source control, discoverable only during the next incident. Part 2 turns that runtime model into a reliable delivery artifact: Dockerfiles, BuildKit cache behavior, multi-stage builds, image tagging, and registry promotion — the practices that make sure the image this chapter has been running actually deserves the trust this chapter has been placing in it.