Container Model, Images & Runtime Lifecycle

Table of Contents#

1. Why Containers Changed Delivery
2. Docker's Control Plane on One Host
3. Images, Layers, and Containers
4. The Lifecycle Behind docker run
5. Isolation Is a Boundary, Not a Virtual Machine
6. Image Pruning and Dangling Images
7. Registries, Tags, and Digests
8. docker commit: Why It's Rarely the Right Tool
9. save/load vs. export/import: Two Different Artifacts
10. A Practical Runtime Workflow
11. Interactive Sessions: attach, exec, and TTY Allocation
12. Copying Files To and From a Container
13. Environment Variables and Runtime Configuration
14. Naming, Labels, and Fleet Metadata
15. Resources, Signals, and Shutdown
16. Adjusting a Running Container Without Recreating It
17. Observability and Health
18. Incident Workflow
19. Common Mistakes and Interview Traps
20. Worked Practice Problems
21. Summary and What's Next

This chapter builds the vocabulary the rest of the series assumes: what an image and a container actually are, how the daemon and its underlying runtime components fit together, how a container moves through its lifecycle, and the everyday commands — inspecting, copying files, adjusting resources, running interactively — that make up most of an engineer's real day-to-day contact with Docker. Part 2 builds on this to construct images deliberately; Parts 3 and 4 build on it to wire multiple containers together and operate them safely in production.

Why Containers Changed Delivery#

Containers make the application artifact explicit. Instead of asking an operator to reconstruct a runtime from a ticket, a package list, and tribal knowledge, a container image captures the executable, libraries, files, default command, and runtime metadata required by one workload.

That is not a promise that every environment is identical. The host kernel, CPU architecture, network policy, credentials, and mounted data still matter. It is a promise that the application filesystem and process contract can move as one versioned unit.

Docker provides the common workflow around that unit: build an image, store it in a registry, create a container from it, attach deliberate configuration, and observe or remove the result. Docker's client sends API requests to dockerd; the daemon manages images, containers, networks, and volumes. Docker's architecture is therefore a local control plane, not merely a command-line binary.

The operational problem it solves#

Before containers, application delivery often mixed three change streams: application code, host configuration, and runtime dependencies. A package upgrade on one host could quietly change the behavior of a deployment that had not changed its own repository. Containers turn most of that runtime dependency surface into an immutable artifact that can be tested before promotion.

The useful analogy is a manufacturing release package: a factory does not send a product design and ask every plant to choose compatible parts. It sends a controlled bill of materials and an assembly specification. The container image is not the whole factory — the host and deployment platform still provide power, safety controls, and transport — but it sharply reduces variation in what arrives at the line.

From the Trenches: A container image does not repair an unversioned dependency outside the image. If an application expects DATABASE_URL, a TLS private key, and a writable /data path, omitting those dependencies still fails at runtime. “It is containerized” is not an operational readiness criterion; the startup contract must be tested in an environment with the same configuration boundaries as production.

Docker's Control Plane on One Host#

The docker CLI is a client. It does not normally create namespaces or mount layers itself. It calls the Docker API exposed by dockerd, often over a local Unix socket. This distinction matters when troubleshooting permissions, remote daemons, and CI runners: whoever can control the daemon can usually ask it to create highly privileged host resources.

Docker Engine coordinates several responsibilities:

• image content acquisition and local storage;
• container metadata and lifecycle;
• network and volume objects;
• handoff to the low-level runtime that starts the process.

Inspect the endpoint before assuming which daemon a command targets:

docker context ls
docker context show
docker version
docker info

docker version separates client and server details. If the client is present but the server cannot be reached, investigate the selected context, daemon state, socket permissions, or remote endpoint before reinstalling the CLI.

From dockerd to containerd to runc#

dockerd itself does not create containers directly. Docker's architecture is layered: dockerd delegates container execution to containerd, a separate daemon responsible for image transfer, storage, and container lifecycle supervision; containerd in turn invokes runc, a low-level runtime that implements the OCI Runtime Specification and does the actual work of creating namespaces, cgroups, and the container's initial process.

The containerd-shim process is the detail that explains a behavior every Docker user eventually notices: restarting or upgrading dockerd does not kill already-running containers. runc exits immediately after starting the container process — it is not a supervisor. The shim, spawned per container, becomes that process's actual parent and stays running independently of both containerd and dockerd, so a daemon restart only reconnects to already-running containers rather than restarting them.

ps -ef | grep containerd-shim
docker info --format '{{.ContainerdCommit.ID}}'

From the Trenches: A team scheduled Docker Engine upgrades during a maintenance window specifically to avoid any container downtime, having assumed (incorrectly) that upgrading dockerd would necessarily restart every running container. Testing on a staging host revealed containers stayed running and reachable throughout the entire dockerd upgrade, because the shim architecture keeps them alive independently — the "maintenance window" requirement turned out to be unnecessary caution for that specific upgrade path, though the team correctly kept it for upgrades that explicitly documented a containerd-level breaking change instead.

Docker contexts: which daemon is a command actually talking to#

A single docker CLI installation can target more than one daemon — a local Docker Engine, a remote host over SSH, or a rootless daemon (Part 4) running under the current user — and contexts are the mechanism that decides which one any given command actually reaches. Every command in this series implicitly runs against whichever context is currently active, which matters the moment more than one is configured on the same machine.

docker context ls
docker context create remote-staging --docker "host=ssh://deploy@staging.internal"
docker context use remote-staging
docker --context remote-staging container ls

docker context use changes the persistent default for every subsequent command in that shell; --context on a single invocation is a one-off override that doesn't change the default. Both are meaningfully safer than the older pattern of exporting DOCKER_HOST in a shell profile, because context ls gives an explicit, inspectable list of every configured target — an exported environment variable buried in a dotfile is easy to forget about entirely until a command targets the wrong host unexpectedly.

From the Trenches: An engineer ran what they believed was a routine docker system prune -a against their local development environment, intending to reclaim disk space before a demo. Their shell had a leftover remote-staging context left active from debugging a deployment issue the previous day, and the prune command executed against the shared staging host instead — removing several image caches other engineers were actively relying on for fast rebuilds. The team's corrective action was adding the active context to the shell prompt itself, so which daemon a command targets is visible at a glance rather than only discoverable by running docker context ls proactively.

Docker Desktop is not a Linux host daemon#

On macOS and Windows, Docker Desktop commonly runs the Linux engine inside a managed virtual machine. The client experience is intentionally similar, but host networking, filesystem mounts, and kernel behavior are not identical to a native Linux Docker Engine. A bind mount that is fast on a Linux CI runner can be noticeably slower on a desktop VM because file sharing crosses an extra boundary.

From the Trenches: Do not diagnose a Linux production incident by assuming docker0 or a container IP is reachable from a macOS Docker Desktop host. Docker documents platform-specific networking behavior; use published ports and service DNS as the portable interfaces.

Images, Layers, and Containers#

An image is a read-only package. It is assembled from ordered filesystem layers and configuration metadata. A container is a runnable instance of that image plus a writable layer and runtime configuration such as environment variables, mounts, ports, resource limits, and the command to execute.

Layers explain two important behaviors:

1. A rebuild can reuse unchanged layers, which makes well-ordered Dockerfiles faster.
2. Removing a container discards changes in its writable layer unless data was stored in a volume or another external system.

The writable layer is not a database. A container can write there, but its lifetime is tied to the container object. Treat it as scratch space for caches, temporary files, and process state that can disappear. Persistent state belongs in a named volume or an external managed service; Part 3 covers the decision in detail.

The OCI image spec: what's actually inside an image#

An image is not a monolithic file — it is a small JSON manifest pointing at a set of independently addressable pieces, standardized by the OCI Image Format Specification so that any OCI-compliant tool (Docker, Podman, a registry, a scanner) can produce and consume the same artifact.

docker manifest inspect nginx:1.27
docker image inspect nginx:1.27 --format '{{json .RootFS.Layers}}'

{
  "schemaVersion": 2,
  "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:abc..." },
  "layers": [
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:111...", "size": 28123456 },
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:222...", "size": 4021 }
  ]
}

The manifest references a config blob (the JSON describing environment variables, entrypoint, exposed ports, and the layer digest chain) and an ordered list of layer blobs — each one a compressed tarball representing a filesystem diff from the layer below it, including special .wh.<name> "whiteout" marker files that record a deletion rather than storing an empty directory. None of these pieces are Docker-specific: a registry stores and serves them as OCI blobs regardless of which OCI-compliant tool built or will run them, which is exactly what makes Part 2's cross-tool build and scan tooling interoperable in the first place.

Copy-on-write storage: how layers become a filesystem#

Pulling an image downloads its layer blobs; running a container requires assembling those layers, plus a new writable layer, into one coherent filesystem view. Docker's default storage driver on modern Linux, overlay2, does this with the kernel's OverlayFS: each image layer is a read-only directory, stacked in order, with the container's writable layer mounted as the topmost upperdir.

docker info --format '{{.Driver}}'
docker inspect catalog-api --format '{{.GraphDriver.Data}}'

Copy-on-write means a container never mutates an image layer directly — the first write to any file causes overlay2 to copy that file up into the container's own writable layer, and only the copy is modified from then on. This is what makes launching a hundred containers from the same image cheap: every container shares the same read-only image layers on disk and only pays storage cost for files it actually changes.

From the Trenches: A host running dozens of containers from the same base image showed far less disk usage under docker system df than a naive "layers × containers" estimate predicted, which briefly worried a capacity-planning exercise into believing usage was being under-reported. Understanding that image layers are genuinely shared, copy-on-write, read-only content — not duplicated per container — resolved the apparent discrepancy immediately once someone checked the storage driver's actual behavior instead of assuming per-container layer duplication.

Image metadata and process semantics#

The image contains defaults, not magic behavior. ENTRYPOINT establishes the executable contract; CMD supplies default arguments. A run command can override those values. The container remains alive while its primary process remains alive. A web server that daemonizes and exits may leave Docker with no foreground process to supervise, so the container stops even though the application attempted to start.

docker image inspect nginx:stable
docker run --rm nginx:stable nginx -T
docker run --rm --entrypoint /bin/sh alpine:3.21 -c 'id && uname -a'

Use --rm for disposable experiments so stopped containers do not accumulate. Do not use it for a workload whose logs or crash state you have not captured.

The Lifecycle Behind docker run#

docker run is convenient shorthand. Conceptually, Docker resolves an image, creates a container object, assigns a writable layer and requested resources, connects networking and mounts, then starts the configured process. If the image is absent locally, the daemon pulls the manifest and content layers from a registry first.

docker pull nginx:1.27
docker container create --name demo-web -p 127.0.0.1:8080:80 nginx:1.27
docker container start demo-web
docker container ls --filter name=demo-web
docker container logs --tail 50 demo-web
docker container stop --time 20 demo-web
docker container rm demo-web

Separating create from start is useful when debugging configuration. You can inspect a created but stopped container to verify its command, mounts, environment, and port bindings before the service starts.

docker container inspect demo-web
docker container stats --no-stream demo-web
docker container exec demo-web nginx -T

docker exec starts an additional process in an already-running container. It is excellent for incident diagnosis, but it is not a deployment mechanism. Changes made interactively are usually confined to the writable layer and disappear on replacement. Capture the corrective action in the image, configuration, or runbook.

The full container state machine#

docker run and docker stop show only two states in casual use, but Docker actually tracks a container through a small, well-defined state machine — knowing the full set explains behavior that otherwise looks inconsistent, like why a "stopped" container still shows up in docker ps -a with an exit code, or why pause freezes a container without stopping it.

docker container create --name demo nginx:1.27
docker container ls -a --filter name=demo --format '{{.Status}}'
docker container pause demo
docker container unpause demo

Paused is a distinct, less commonly used state worth knowing explicitly: it freezes every process in the container via the cgroup freezer, with no CPU scheduling at all, while keeping memory state intact — useful for a brief, deliberate freeze (a live migration handoff, a consistent-snapshot window) but not a substitute for stop, since a paused container still holds all of its resources and network connections open without doing anything with them.

Dead is a state most engineers never intentionally trigger — it indicates the daemon attempted to remove or stop a container and failed partway, leaving inconsistent state. A container stuck in Dead typically needs docker container rm -f and, if that also fails, an investigation into daemon or storage-driver health rather than repeated retries of the same removal command.

Zombie processes and the --init flag#

A container's PID 1 has a responsibility beyond just being the main process: on Linux, PID 1 is also responsible for "reaping" zombie processes — child processes that have exited but whose exit status hasn't been collected by a parent that called wait(). A normal init system (systemd, or a shell in an interactive session) does this automatically; an application binary running directly as PID 1 often does not, because most applications were never written with PID-1 responsibilities in mind.

docker run -d --name worker --init worker:1.4

The --init flag injects a minimal init process (docker-init, based on tini) as the container's actual PID 1, which then execs the application as its child — the application still behaves as the "main" process for all practical purposes (its exit code becomes the container's exit code), but zombie reaping and signal forwarding are now handled correctly underneath it.

From the Trenches: A container running a script-based application via CMD ["sh", "-c", "python worker.py"] accumulated defunct zombie processes over weeks of operation, each one a residual few bytes in the process table, until the container's PID limit (Part 4 covers --pids-limit) was eventually exhausted by the accumulated zombies rather than by any real workload growth — a confusing symptom, since the application's own memory and CPU usage looked completely normal the entire time. Adding --init resolved it without any application code change, because the actual problem was PID 1 never reaping the short-lived subprocesses the application spawned internally.

Restart policies are a limited tool#

Restart policies can restart an exited process on one Docker host. They do not add scheduling across hosts, rollout control, capacity placement, or dependency orchestration. Use them for a bounded host-local service only after its startup failure mode is understood.

From the Trenches: A rapidly restarting container can make dashboards look “alive” because a container exists, while users receive no successful requests. Alert on service-level health and restart rate, not merely the Docker container state.

Isolation Is a Boundary, Not a Virtual Machine#

Linux containers rely primarily on namespaces for isolation and cgroups for resource accounting and limits. Namespaces give a process a constrained view of identifiers such as processes, mounts, networks, and users. Cgroups govern resource usage such as CPU and memory. Docker makes these kernel primitives operationally accessible, but they do not create a separate kernel.

This has two consequences. First, containers generally start faster and consume less overhead than a full guest operating system. Second, a kernel vulnerability, overly broad capability, host mount, or access to the Docker socket changes the risk model substantially. Treat a container boundary as meaningful defense in depth, not as an excuse to run untrusted code with broad host access.

The Linux & Networking Fundamentals series explains namespaces and cgroups in kernel depth. Here, the Docker decision is practical: request only the privileges and host interfaces the process needs.

The namespace types Docker uses, at a glance#

Docker doesn't apply one undifferentiated "isolation" — it composes several distinct Linux namespace types per container, each hiding a different kind of host-wide identifier. Knowing which namespace is responsible for which illusion explains specific, otherwise-confusing behavior.

docker run --rm --pid host alpine:3.21 ps aux | head -5
docker run --rm alpine:3.21 hostname

--pid host (rare, and worth treating as a specific, justified exception rather than a routine flag) shares the host's PID namespace instead of creating a new one — the diagnostic sidecar pattern from Part 3's networking debugging section uses exactly this, plus a shared network namespace, to see a target container's real process tree from outside it.

docker run --rm \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop ALL \
  --user 10001:10001 \
  --security-opt no-new-privileges:true \
  nginx:1.27

The command is a starting posture, not a universal drop-in. Nginx may need writable cache or PID paths depending on its configuration. Validate the workload, then make each permission explicit instead of reverting to --privileged.

Image Pruning and Dangling Images#

Rebuilding an image under the same tag doesn't delete the previous build's layers — it moves the tag to point at the new image and leaves the old one behind, now untagged. Docker displays these as <none>:<none> in docker images, commonly called "dangling" images.

docker images --filter dangling=true
docker image prune
docker image prune -a --filter "until=168h"

docker image prune removes dangling images only — safe to run routinely, since nothing can reference an untagged image by name anymore. docker image prune -a is a materially different, more aggressive operation: it removes every image with no container currently using it, tagged or not, which can delete an image a deployment script still expects to find locally for a future docker run even though nothing is running from it right now. The --filter "until=168h" age filter is what makes routine, scheduled pruning safe — it only touches images old enough that a legitimate rebuild-and-redeploy cycle would have already superseded them.

From the Trenches: A disk-cleanup cron job ran docker image prune -a unconditionally every night on a host that pre-pulled several images specifically so a fast-response deployment script could skip the pull step during an incident. The prune job silently deleted those pre-pulled images every night, and the first genuine incident response after the cron job was introduced was slower than the runbook assumed, because "the image is already local" turned out to no longer be true. Switching the nightly job to plain docker image prune (dangling only) restored the pre-pull optimization's actual benefit.

Registries, Tags, and Digests#

A registry stores image repositories. A tag is a mutable human-friendly reference such as 1.8.4 or stable. A digest is a content-addressed immutable identifier such as sha256:.... Tags are useful for release workflows; digests are useful when an environment must run exactly the artifact that was approved.

docker build -t registry.example.com/payments/api:1.8.4 .
docker push registry.example.com/payments/api:1.8.4
docker buildx imagetools inspect registry.example.com/payments/api:1.8.4
docker pull registry.example.com/payments/api@sha256:REPLACE_WITH_APPROVED_DIGEST

Never invent a digest in a production command. Copy the approved digest from the registry or deployment metadata. The image reference structure and tag behavior are documented by Docker; a tag can be moved, so it is not evidence of immutable provenance.

From the Trenches: “latest” is an environment-dependent instruction, not a release identifier. Two hosts can run different bytes under the same tag when they pull at different times. An incident rollback is slow and ambiguous when nobody can state the exact digest that was live.

Multi-platform manifest lists#

A single tag can actually resolve to different bytes depending on the pulling machine's CPU architecture. What docker pull registry.example.com/catalog/api:2.6.0 retrieves is often not a single image manifest but a manifest list (also called an image index) — a small JSON document listing several architecture-specific manifests, letting the daemon pick the one matching its own platform automatically.

docker manifest inspect registry.example.com/catalog/api:2.6.0
docker buildx imagetools inspect registry.example.com/catalog/api:2.6.0

{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    { "platform": { "architecture": "amd64", "os": "linux" }, "digest": "sha256:aaa..." },
    { "platform": { "architecture": "arm64", "os": "linux" }, "digest": "sha256:bbb..." }
  ]
}

This is why the same docker pull command run on an Intel CI runner and an Apple Silicon developer laptop can silently retrieve genuinely different image bytes under an identical tag — both are correct, expected behavior, not a caching bug. Part 2's multi-architecture build coverage is what actually produces this manifest list; this chapter's concern is simply recognizing that a tag and a specific set of bytes are not a one-to-one mapping once more than one platform is involved.

From the Trenches: A "the image behaves differently between my laptop and CI" bug report turned out not to be an environment configuration difference at all — the developer's Apple Silicon laptop and the x86 CI runner were pulling different manifests from the same manifest list, and one architecture's build had a genuine, unrelated bug the other didn't share. Recognizing that docker pull <same tag> doesn't guarantee identical bytes across architectures redirected the investigation to the actual per-architecture build difference within a few minutes, instead of continuing to search for a phantom environment misconfiguration.

docker commit: Why It's Rarely the Right Tool#

docker commit creates a new image from a running or stopped container's current filesystem state — technically simple, and worth understanding precisely because it undermines nearly everything the rest of this series is built on.

docker exec -it demo-web sh -c 'apt-get update && apt-get install -y curl'
docker commit demo-web demo-web:patched

The resulting demo-web:patched image has no Dockerfile, no build history, no record of what command produced it beyond "whatever happened to be true about this specific container's filesystem at commit time." It cannot be rebuilt, cannot be meaningfully code-reviewed, and cannot be reproduced from source — every property Part 2 spent an entire chapter establishing for a trustworthy build artifact is absent by construction.

The one narrow legitimate use is capturing a container's exact broken state for offline forensic analysis during an incident (per Part 4's incident guidance) — even then, the resulting image is evidence to inspect, not something to redeploy.

From the Trenches: A production incident was "fixed" under time pressure by exec-ing into the running container, applying a manual patch, and committing the result as the new "production image" to unblock a release deadline. Six months later, nobody could explain why that specific version behaved differently from every subsequent Dockerfile-built image, because the commit had captured an undocumented, non-reproducible filesystem state with no corresponding source change — the team eventually had to treat the entire image lineage from that point as untrusted and rebuild from a known-good Dockerfile to regain a reproducible baseline.

save/load vs. export/import: Two Different Artifacts#

Docker has two pairs of commands that sound similar and are frequently confused, but operate on genuinely different things: an image (with its full layer history and metadata) versus a container's current filesystem (a flat snapshot, with history discarded).

docker save nginx:1.27 -o nginx-image.tar
docker load -i nginx-image.tar

docker export demo-web -o demo-web-rootfs.tar
docker import demo-web-rootfs.tar demo-web:snapshot

save/load round-trip a real, OCI-compliant image — layers, manifest, config, and history all intact — the same artifact this chapter's OCI image spec section described. export/import instead flatten a running or stopped container's current filesystem into a single tarball with no layer history and no image config metadata (ENTRYPOINT, CMD, exposed ports, and environment defaults are all lost) — import creates a brand-new single-layer image from that flat filesystem, not a faithful reconstruction of the original image.

From the Trenches: An engineer used docker export/docker import to "back up" a running container before an experiment, reasoning it was equivalent to a docker commit or a proper image save. The imported result ran, but with none of the original image's environment variables or entrypoint configuration — because export/import never carried that metadata in the first place — and the experiment's rollback plan had to be redone from the actual source image and its build configuration, the artifact that should have been the backup all along.

A Practical Runtime Workflow#

Use a deliberate sequence for a new service:

1. Start a named container with an explicit image version and host binding.
2. Inspect the exact command, image, ports, mounts, and health output.
3. Read logs from the primary process.
4. Verify the service through its published interface.
5. Capture durable configuration in source-controlled Compose or deployment manifests.

docker run -d \
  --name catalog-api \
  --publish 127.0.0.1:8080:8080 \
  --env-file ./catalog-api.env \
  --memory 512m \
  --cpus 1.0 \
  registry.example.com/catalog/api:2.4.1

docker container inspect catalog-api --format '{{json .NetworkSettings.Ports}}'
docker container logs --since 10m catalog-api
docker container stats --no-stream catalog-api

Binding to 127.0.0.1 deliberately exposes the port only on the host loopback interface. Publishing to all host interfaces is a security decision, not the default answer for every backing service.

What to record in a runbook#

Interactive Sessions: attach, exec, and TTY Allocation#

Three commands let a human interact with a container's process, and they are not interchangeable — confusing them produces symptoms ranging from a garbled terminal to accidentally sending input to a production process.

docker run -it --name debug-shell alpine:3.21 sh
docker attach debug-shell
docker exec -it debug-shell sh

docker run -it allocates a pseudo-TTY (-t) and keeps stdin open (-i) for the container's own PID 1 process — appropriate for a container whose primary purpose is an interactive session. docker attach connects to that same PID 1 process's existing stdio streams after the fact; critically, it does not start a new process, and typing exit inside an attached session terminates the container's main process, not just your terminal connection — a mistake that has stopped production containers more than once. docker exec -it starts a brand-new process inside the container's namespaces, independent of PID 1, which is why exit there only ends that one exec session and leaves the main process untouched.

From the Trenches: An engineer intending to check logs on a production container ran docker attach out of habit from local development, then typed exit to leave — which sent the exit to the container's actual PID 1 and stopped a live production service, rather than merely disconnecting a terminal. The corrective action was a team-wide convention: attach is reserved for the rare case of genuinely needing to interact with PID 1's own stdio, and exec -it <container> sh is the default for any exploratory or diagnostic session, specifically because it can never accidentally terminate the main process.

Detached mode and log visibility#

docker run -d starts a container without attaching to it at all — the common production pattern, since a real service shouldn't depend on an open terminal session to keep running. This makes docker logs the correct tool for observing a detached container's output, not attach, which for a long-running production service is rarely the right choice even when available.

docker run -d --name catalog-api catalog-api:2.6.0
docker logs --follow --tail 100 catalog-api

Copying Files To and From a Container#

docker cp moves files between the host and a container's filesystem directly, bypassing both the image build process and any mounted volume — useful for a specific diagnostic or one-off need, but easy to reach for as a substitute for a mechanism that should really be a proper volume, bind mount, or image rebuild.

docker cp catalog-api:/app/logs/error.log ./error.log
docker cp ./patched-config.yaml catalog-api:/app/config.yaml

Both directions have the same fundamental property as docker exec-driven changes: anything copied into a container lands only in that container's writable layer (Part 1's copy-on-write model) and disappears the moment the container is replaced, exactly like a manual exec-based edit. Copying out of a container is genuinely useful and low-risk — pulling a crash log or core dump for offline analysis during an incident is a legitimate, common use.

From the Trenches: A recurring production issue was "fixed" repeatedly by an on-call engineer copying an updated configuration file into the running container with docker cp, each time believing it was a temporary stopgap until a proper fix landed. Because the fix worked immediately and the underlying Dockerfile/config-management change was never prioritized, the same manual docker cp had been silently repeated after every container replacement for months — an invisible, undocumented dependency that a new team member discovered only when a routine redeploy "mysteriously" reintroduced the original bug.

Environment Variables and Runtime Configuration#

An image's Dockerfile can declare default environment variables with ENV; a running container can override any of them at docker run time — the same override relationship Part 2 covered for ARG vs ENV at build time, now at the runtime layer.

ENV LOG_LEVEL=info
ENV PORT=8080

docker run -d --name catalog-api -e LOG_LEVEL=debug -e PORT=9090 catalog-api:2.6.0
docker run -d --name catalog-api --env-file ./catalog-api.env catalog-api:2.6.0
docker container inspect catalog-api --format '{{json .Config.Env}}'

-e overrides one variable at a time; --env-file loads many from a file in one flag, useful once a service has more than a handful of configuration values. Both are visible in full via docker container inspect — this is the same non-secret-appropriate visibility Part 3 and Part 4 flagged for Compose environment: blocks, worth restating here at the single-container level: an -e flag is fine for a log level or a feature flag, and wrong for a credential, which belongs in the mounted-file mechanisms those later chapters cover instead.

Precedence, made explicit#

A variable set by both --env-file and a same-named -e flag on the same command resolves to the -e value — worth confirming explicitly with docker container inspect rather than assuming, especially once a deployment script combines both mechanisms and it's no longer obvious at a glance which one "wins" for a given variable.

From the Trenches: A staging environment's LOG_LEVEL refused to change no matter how many times an engineer edited the environment file the deployment script referenced, because that same script also passed an unrelated, older -e LOG_LEVEL=info flag left over from an earlier debugging session — and the -e flag silently took precedence over the file every single time. docker container inspect --format '{{json .Config.Env}}' on the actually-running container, rather than re-reading the source files and assuming they were authoritative, revealed the conflicting flag within seconds.

Naming, Labels, and Fleet Metadata#

A single-host deployment with a handful of containers can get by with memorable names alone. Anything beyond that needs a deliberate metadata convention — Docker's own tooling has no opinion on what a "service," "environment," or "owner" means, so that structure has to come from labels applied consistently.

docker run -d --name catalog-api-prod \
  --label app=catalog-api \
  --label env=production \
  --label team=platform \
  --label version=2.6.0 \
  catalog-api:2.6.0

docker container ls --filter label=env=production --filter label=team=platform
docker container ls --filter label=app=catalog-api --format 'table {{.Names}}\t{{.Label "version"}}'

Labels turn ad hoc container names into queryable fleet metadata — docker container ls --filter label=... answers "which containers belong to this team" or "which containers are still running the previous version" without relying on naming convention discipline holding perfectly across every engineer and script that ever starts a container. LABEL instructions baked into the image (Part 2) and --label flags applied at run time (here) compose together; the image can declare static facts about itself (org.opencontainers.image.revision, from Part 2's OCI annotation guidance) while runtime labels capture deployment-specific facts (which environment, which specific host role) that don't belong in the image itself.

From the Trenches: An incident review needed to quickly answer "which containers across this host are still running the vulnerable image version" during a CVE response, and the only available identifying information was container names following an inconsistent, engineer-specific naming convention that had drifted over two years — some included a version number, most didn't, none used a queryable format. Answering the question took a slow, manual docker inspect sweep across every container instead of one docker container ls --filter label=version=... call. The team's post-incident action item was a mandatory --label version= (and app=, env=) convention enforced by their deployment tooling, not left to individual habit.

Resources, Signals, and Shutdown#

A container is still a process. Resource controls protect a shared host, but they do not replace capacity planning. Set a measured memory ceiling, CPU budget, and PID limit, then validate the resulting latency and failure behavior under load.

docker run -d --name invoice-api \
  --memory 768m --memory-reservation 512m --cpus 1.5 --pids-limit 256 \
  registry.example.com/invoice/api:3.2.0
docker container stats --no-stream invoice-api

docker stop sends SIGTERM, waits for its grace period, then uses SIGKILL if required. Applications must stop accepting new work, drain in-flight requests, and exit before that deadline. SIGKILL cannot be handled.

From the Trenches: Restarting a process can make a dashboard look healthy while every request fails during repeated startup. Alert on successful service behavior and restart rate, not merely a running container state.

Adjusting a Running Container Without Recreating It#

Most configuration changes covered so far in this chapter — a new environment variable, a different image version — genuinely require recreating the container, since they're baked in at creation time. A narrow set of runtime properties are the exception: docker update can change a live container's resource limits and restart policy in place, without stopping or recreating it.

docker update --memory 1g --memory-swap 1g --cpus 1.5 catalog-api
docker update --restart unless-stopped catalog-api

This is genuinely useful for an urgent, temporary adjustment during an incident — raising a memory ceiling immediately while a proper fix is prepared, without the brief availability gap a full recreation would cause. It is not a substitute for making the corrected value permanent in whatever source (a Compose file, a deployment script) originally created the container: a docker update change lives only on that one running container and is lost the next time it's recreated from its original definition, silently reverting exactly when the change might be needed most.

From the Trenches: A container was hitting its memory limit repeatedly during a traffic spike, and the on-call engineer used docker update --memory 2g to immediately relieve the pressure while a proper capacity review was scheduled for the following week. The capacity review happened, concluded the higher limit was correct, and updated the team's documentation — but nobody updated the actual Compose file, so the next routine redeploy silently reverted the container to its original, too-low limit and reintroduced the exact incident the review had "fixed." The lesson generalized into a standing checklist item: any docker update applied during an incident gets a linked follow-up ticket to make the same change in the source configuration, closed only when the source itself reflects it.

Observability and Health#

Containers should write structured application events to stdout and stderr. docker logs is useful for one-host diagnosis, but centralized collection, correlation, retention, and deployment identity are required in production.

docker container logs --timestamps --tail 100 catalog-api
docker container logs --follow catalog-api
docker events --filter container=catalog-api
docker inspect --format '{{json .State.Health}}' catalog-api

A Docker HEALTHCHECK records container health metadata. It must test the correct boundary: a liveness check proves the process can run, while readiness should prove the service can accept its intended work. Do not make every probe execute an expensive dependency transaction.

HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD wget --spider --quiet http://127.0.0.1:8080/healthz || exit 1

Exit codes and restart counts as a first triage signal#

Before reading a single log line, a container's exit code and restart count already narrow down the likely failure category — cheap, structured signals worth checking first in any triage, including the incident workflow later in this chapter.

docker container inspect catalog-api --format '{{.State.ExitCode}} {{.RestartCount}}'

A high RestartCount on a container using an on-failure or always policy is itself a symptom worth investigating even if the container currently shows as running — it means the process has been dying and coming back repeatedly, which per this chapter's earlier restart-policy guidance can hide behind a dashboard that only checks "is a container running" rather than "is it succeeding."

HEALTHCHECK command forms and inheriting from a base image#

HEALTHCHECK supports two command forms with a meaningful difference in how failures surface, plus a way to explicitly opt a container out of a check inherited from its base image.

HEALTHCHECK CMD ["curl", "-f", "http://127.0.0.1:8080/healthz"]
HEALTHCHECK CMD curl -f http://127.0.0.1:8080/healthz || exit 1

The exec form (CMD [...], no shell involved) runs the binary directly — if curl itself succeeds but the response is a non-2xx status, curl -f still needs to be told to fail on that explicitly, since there's no shell present to interpret an || fallback in this form at all. The shell form (CMD curl ... || exit 1, no brackets) runs through /bin/sh -c, which does let you chain a fallback exit code — but inherits the same PID-1-signal-forwarding caveat from earlier in this chapter, since the health check itself now runs as a subshell rather than a direct exec.

HEALTHCHECK NONE

A derived image (one using FROM some-base-image-with-a-healthcheck) inherits that base image's HEALTHCHECK automatically unless explicitly overridden — HEALTHCHECK NONE disables an inherited check that doesn't make sense for the derived image's actual purpose, which matters more than it might seem: a base image's health check assumptions (a specific port, a specific endpoint path) don't automatically hold for everything built on top of it.

From the Trenches: A team's internal base image shipped a HEALTHCHECK testing port 8080, and a service built from it that legitimately listened on a different port inherited that check unmodified — the derived image reported unhealthy in every environment, and the confusing part was that nothing in the derived image's own Dockerfile mentioned a health check at all, since it had never explicitly declared one, only inherited one silently. docker image history on the derived image, not just reading its own Dockerfile, was what revealed the inherited instruction's actual origin.

Incident Workflow#

Preserve evidence before replacing a failed container. Work from user symptoms through service reachability, logs, configuration, resource state, and dependencies.

A quick-reference table of docker inspect recipes#

docker inspect returns a large JSON document per container or image; --format with a Go template extracts exactly the field a specific investigation needs, avoiding the need to pipe through jq for common questions. Keeping a short reference of the recipes actually used during an incident, rather than re-deriving the template syntax under pressure, is a small thing that measurably speeds up triage.

docker inspect catalog-api --format '{{.State.Status}}'
docker inspect catalog-api --format '{{.State.ExitCode}} {{.RestartCount}}'
docker inspect catalog-api --format '{{.Config.Image}}'
docker inspect catalog-api --format '{{json .Config.Env}}'
docker inspect catalog-api --format '{{json .HostConfig.Memory}}'
docker inspect catalog-api --format '{{json .NetworkSettings.Networks}}'
docker inspect catalog-api --format '{{json .Mounts}}'

Each of these answers a question this chapter has already covered in isolation — the exit-code table, the resource-limit discussion, the environment-precedence section — this table exists purely to collect the actual commands in one place, since an incident is the wrong moment to be reconstructing Go template syntax from memory.

docker container ls -a --no-trunc
docker container inspect catalog-api --format '{{json .State}}'
docker container logs --since 15m --timestamps catalog-api
docker container stats --no-stream catalog-api

Record the image digest, exit code, restart count, timestamps, and relevant configuration structure. Never paste raw docker inspect output into an incident channel without checking it for secrets.

Failure classification before recovery#

Classify the failure before selecting recovery. A configuration failure repeats predictably on every restart. A transient dependency failure may recover with bounded retries. A capacity failure needs admission control, scaling, or resource tuning. A corrupted image or registry reference needs a known-good artifact, not a host-level workaround.

The immutable recovery rule#

Do not repair a production container by installing packages, editing application files, or changing its process interactively. Those actions cannot be reviewed, reproduced, or reliably carried into the next replacement. Use the incident to identify the missing build, configuration, or deployment control, then make that correction in the appropriate repository.

This discipline is what turns containers from a convenience wrapper into a reliable delivery unit: each replacement is created from declared inputs, and each investigation leaves a durable improvement behind.

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. A service exits immediately after docker run. What do you check first?#

Start with docker container logs <name> and docker container inspect <name> to identify the exit code, command, and configuration. Confirm that the configured entrypoint launches the intended foreground process. Then check missing environment variables, a bad bind mount, an unavailable dependency, or a port conflict. Do not immediately add a restart policy: that converts a clear failure into a noisy loop.

2. A team says a containerized PostgreSQL database has durable data because the image is versioned. Correct them.#

An image version preserves database software, not runtime writes. The database files are created in the container writable layer unless a volume or external storage target is mounted. Replacing the container replaces that writable layer. Define a named volume or managed database service, test restore behavior, and document the backup and recovery objective.

3. A CVE response requires knowing whether production runs the fixed image. What evidence is sufficient?#

The deployment record must identify the repository and immutable digest that is running, then compare that digest with the remediated build's digest and SBOM or vulnerability scan evidence. A tag alone is insufficient because it can move. Roll forward by promoting the verified artifact and confirm running containers reference that digest.

4. A container that was working fine yesterday exited overnight with no obvious application error in its logs. Where do you look before assuming the application itself is at fault?#

Check the exit code and restart count first with docker container inspect --format '{{.State.ExitCode}} {{.RestartCount}}' — an exit code of 137 points to a SIGKILL, commonly an OOM kill, which docker events --filter event=oom can confirm independently of anything the application itself logged (a process killed by the kernel usually has no chance to log its own death). This reframes the investigation from "why did the application silently fail" to "why did memory usage cross the configured limit," a meaningfully different and more tractable question.

5. Two engineers on the same team consistently get different results running the same docker pull command against the same tag — one gets a working image, the other reports a bug that the first can't reproduce. Both are on the same network and registry. What's a likely explanation that has nothing to do with caching?#

If the two engineers are on different CPU architectures (one on an Apple Silicon laptop, one on an Intel machine, for instance), the tag may resolve to a manifest list with genuinely different per-architecture builds, and a bug present in only one architecture's build would reproduce exclusively on that architecture. Confirm this with docker manifest inspect <tag> to see whether multiple platform-specific manifests exist, and check docker image inspect --format '{{.Architecture}}' on each engineer's locally pulled image before assuming the difference is environmental configuration or a caching artifact.

6. A junior engineer proposes using docker cp to push a hotfix configuration file into every production container as an immediate, temporary stopgap while a proper fix goes through review. What's the risk, and what should happen instead?#

The immediate risk is exactly what Part 1's docker commit and docker cp guidance warns about: a host-filesystem change that lives only in each container's writable layer, invisible to version control, and silently lost the next time any of those containers is replaced — which means the "temporary" stopgap either has to be manually reapplied after every future deployment (a growing, undocumented maintenance burden) or quietly disappears and the original bug reappears without explanation. The correct response under real time pressure is still to make the fix in the actual configuration source (the image, a mounted config file, or an environment variable) and deploy it through the normal pipeline — even an expedited version of that pipeline is safer than a change that exists nowhere durable.

Summary and What's Next#

Docker packages an application into a versioned image and turns that image into a container with explicit runtime configuration. The operational discipline is to distinguish immutable image content from mutable runtime state, treat the daemon as privileged infrastructure, and make ports, mounts, identities, and resources deliberate. Underneath the CLI, dockerd delegates to containerd and runc through a shim that's what actually keeps containers alive across a daemon restart; an image is a small OCI manifest pointing at content-addressed, copy-on-write layers shared across every container using it; and the tools that touch a running container directly — attach, exec, cp, commit — each have a specific, narrow legitimate use and a much more common way to misuse them as a substitute for a real, reproducible fix.

None of this chapter's tools replace the discipline of treating the image and its declared configuration as the source of truth. A container you can exec into, patch, and walk away from is not more reliable than one you can't — it's a container whose actual running state has quietly diverged from anything checked into source control, discoverable only during the next incident. Part 2 turns that runtime model into a reliable delivery artifact: Dockerfiles, BuildKit cache behavior, multi-stage builds, image tagging, and registry promotion — the practices that make sure the image this chapter has been running actually deserves the trust this chapter has been placing in it.