Dockerfiles, BuildKit & Image Delivery

Table of Contents#

1. From Runtime Model to Build Pipeline
2. BuildKit Is the Build Engine, Not Just a Flag
3. Writing a Dockerfile That Caches Well
4. Multi-Stage Builds as an Artifact Boundary
5. Cache Mounts, Secrets, and Build-Time Data
6. Tagging, Digests, and Promotion Workflows
7. Supply-Chain Controls: SBOM, Provenance, Signing
8. Multi-Architecture Builds with buildx
9. A CI Build Pipeline End to End
10. Choosing a Base Image
11. Dockerfile Instruction Reference Notes
12. Vulnerability Scanning in the Pipeline
13. Linting and Build Reproducibility
14. Registry Choices and Image Retention
15. A Complete GitHub Actions Build Pipeline
16. Rootless and Daemonless Build Alternatives
17. Layer Count, Squashing, and Image Flattening
18. Offline and Air-Gapped Image Delivery
19. Build Context Sources and Private Dependencies
20. Debugging a Failed or Suspicious Build
21. Running Tests as a Build Stage
22. Common Mistakes and Interview Traps
23. Worked Practice Problems
24. Summary and What's Next

From Runtime Model to Build Pipeline#

Part 1 treated the image as a given: something pulled from a registry and run. This chapter builds that artifact deliberately. The Dockerfile is the specification; BuildKit is the engine that turns it into layered, content-addressed, cacheable image content; the registry is where the result becomes shareable and auditable.

Treat a Dockerfile the way a release engineer treats a build manifest, not a shell script that happens to run in a container. Every instruction produces a layer with a content hash. Two builds with identical instructions and identical inputs should produce identical layers — that determinism is what makes caching, promotion, and rollback trustworthy. A Dockerfile that downloads "latest" packages, embeds a timestamp, or depends on unpinned network state breaks that guarantee quietly, and the failure only shows up as "it built differently on Tuesday."

From the Trenches: A team once "fixed" a flaky CI build by adding apt-get update && apt-get install -y curl with no version pin, reasoning that curl rarely changes. Six months later, a Debian security update changed a transitive dependency's default TLS behavior, and only the container build picked up the new package — the host toolchain used elsewhere in CI did not. The image passed CI and failed a production TLS handshake. The fix was pinning package versions and rebuilding on a schedule instead of relying on floating installs to "just work."

BuildKit Is the Build Engine, Not Just a Flag#

Modern Docker builds run on BuildKit by default; the legacy builder is effectively retired. BuildKit changes more than performance — it changes the execution model. It parses the Dockerfile into a build graph (a DAG of stages and instructions), resolves which nodes are independent, and can execute unrelated stages concurrently instead of strictly top to bottom. It also introduces first-class primitives — cache mounts, bind mounts, and secret mounts — that a naive docker build on the classic builder never had.

docker buildx version
docker buildx ls
docker build --progress=plain -t catalog-api:dev .

docker buildx is the CLI front end for BuildKit. On recent Docker Engine releases it is the default builder, exposed transparently through docker build; docker buildx build gives access to the fuller feature set — multiple output types, multi-platform builds, and remote cache backends — that plain docker build does not always expose depending on driver.

Builder instances and drivers#

buildx builds run through a named builder instance backed by a driver. The default docker driver runs inside the existing Docker daemon and has real limitations: it cannot export multi-platform manifests to a local image store, and it cannot always store attestations without the containerd image store enabled. The docker-container driver runs BuildKit inside a dedicated container and unlocks multi-platform output, remote cache export, and attestations without daemon reconfiguration.

docker buildx create --name ci-builder --driver docker-container --use
docker buildx inspect --bootstrap

From the Trenches: A pipeline that worked on an engineer's laptop failed in CI with a cryptic "multiple platforms feature is currently not supported" error. The laptop had implicitly created a docker-container builder at some point; the CI runner was still on the default docker driver. Standardizing on an explicit buildx create --driver docker-container step in the pipeline — rather than trusting whatever driver happened to be active — removed the class of failure entirely.

Writing a Dockerfile That Caches Well#

BuildKit's cache is layer-keyed: each instruction's cache key depends on the instruction text and the content hashes of everything it reads. Reordering instructions so volatile inputs (source code) come after stable inputs (dependency manifests) means a source-only change does not invalidate the expensive dependency-install layer.

# syntax=docker/dockerfile:1
FROM node:20.17-bookworm-slim AS base
WORKDIR /app

FROM base AS deps
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

FROM base AS build
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
RUN npm run build

Two details matter more than they look:

1. COPY package.json package-lock.json ./ happens before COPY . .. Editing application source will invalidate the build stage's later layers but not the dependency-install layer, because that layer's cache key never saw the changed files.
2. The # syntax=docker/dockerfile:1 directive pins the Dockerfile frontend. Without it, cache-mount and bind-mount syntax silently fail to parse on older BuildKit frontends. Pin a frontend version deliberately in a regulated pipeline instead of always floating to :1.

.dockerignore is part of caching, not just hygiene#

Every file in the build context is hashed by BuildKit to decide whether a COPY layer's cache is still valid. A .dockerignore that excludes .git, node_modules, build artifacts, and local .env files does two things: it shrinks the context sent to the daemon, and it prevents an unrelated local file (an editor swap file, a stale build output) from invalidating a cache that should have been a hit.

From the Trenches: A Python service's Dockerfile did COPY . . before pip install -r requirements.txt. Every commit — including documentation-only changes — triggered a full dependency reinstall in CI, adding four minutes to every pipeline run regardless of what actually changed. Splitting the COPY requirements.txt . step out ahead of the source copy cut median build time by more than half with no other change.

Multi-Stage Builds as an Artifact Boundary#

A multi-stage Dockerfile uses one or more FROM ... AS <name> stages to build software, then copies only the finished artifact into a final, minimal runtime stage. This is the mechanism that keeps compilers, build tools, and source archives out of the image that actually runs in production.

# syntax=docker/dockerfile:1
FROM golang:1.23-bookworm AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12:nonroot AS runtime
COPY --from=build /out/api /usr/local/bin/api
USER nonroot:nonroot
ENTRYPOINT ["/usr/local/bin/api"]

The build stage carries the entire Go toolchain and module cache. None of that reaches the final image — only the compiled binary crosses the COPY --from=build boundary. The runtime stage uses a distroless base with no shell, no package manager, and a non-root user baked in, which shrinks both image size and the attack surface available to anyone who gains code execution inside the container.

Independent stages build in parallel#

BuildKit's DAG scheduler builds stages that do not depend on each other concurrently. A Dockerfile with a separate FROM node:20 AS frontend-build and FROM golang:1.23 AS backend-build, both copied into a final stage, builds both toolchains at the same time rather than sequentially — a real wall-clock win in CI, not just a stylistic preference.

From the Trenches: An image audit found a "slim" production image was still 640MB because a single-stage Dockerfile shipped gcc, make, and the full apt package cache alongside the compiled binary they were only needed to produce. Converting to a two-stage build with a distroless runtime target dropped the image to 24MB and removed an entire class of "there's a shell in prod, can an attacker use it" security review questions.

Cache Mounts, Secrets, and Build-Time Data#

Three BuildKit mount types solve three distinct problems, and conflating them is a common mistake.

RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

docker buildx build --secret id=npmrc,src=$HOME/.npmrc -t catalog-api:dev .

Never use ARG or ENV to pass a credential into a build. Both are recorded in image history and inspectable with docker history or docker image inspect, even if a later instruction "removes" the file — the layer that contained it is still part of the image's content-addressed history. The type=secret mount is the only mechanism that keeps a credential out of the resulting layers entirely.

From the Trenches: A security scan flagged a production image because docker history --no-trunc revealed a plaintext registry password from an ARG NPM_TOKEN passed three build stages earlier — the team believed a later RUN rm step had removed it. It had removed the file from the final filesystem, not from the image's layer history. Every image built with that Dockerfile had to be treated as a credential leak and the token rotated.

Tagging, Digests, and Promotion Workflows#

Part 1 established that a tag is mutable and a digest is not. This chapter's concern is the workflow that produces both deliberately, instead of treating tagging as an afterthought.

A practical promotion scheme separates a build-time identifier from a deployment-time identifier:

docker buildx build \
  --tag registry.example.com/catalog/api:2.6.0 \
  --tag registry.example.com/catalog/api:git-3f9a2c1 \
  --label org.opencontainers.image.revision=3f9a2c1 \
  --label org.opencontainers.image.created="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --push .

docker buildx imagetools inspect registry.example.com/catalog/api:2.6.0 --format '{{json .Manifest.Digest}}'

• 2.6.0 is the human-facing release tag that a change log or release note refers to.
• git-3f9a2c1 ties the image directly to the commit that produced it, useful for bisecting a regression.
• org.opencontainers.image.* labels follow the OCI image spec's annotation keys, which most registry UIs and scanning tools already understand.
• The digest returned by imagetools inspect is what a deployment manifest should ultimately reference, not the tag.

A promotion pipeline, not a single build#

Re-tagging, not rebuilding, is the important discipline: the artifact that passed staging must be bit-for-bit the artifact that reaches production. Rebuilding "the same" Dockerfile a second time for the release tag reintroduces exactly the nondeterminism this chapter opened with — a floating base-image tag, a package mirror serving a newer patch version, or a timestamp baked into a layer can all produce a different digest the second time.

docker buildx imagetools create \
  --tag registry.example.com/catalog/api:2.6.0-release \
  registry.example.com/catalog/api@sha256:REPLACE_WITH_VERIFIED_DIGEST

buildx imagetools create copies an existing manifest under a new tag without rebuilding anything, which is exactly the operation a promotion step needs.

Supply-Chain Controls: SBOM, Provenance, Signing#

An image that reaches production should be able to answer three questions on demand: what software is inside it, how was it built, and can its authenticity be verified without trusting whoever is asking.

docker buildx build \
  --sbom=true \
  --provenance=mode=max \
  --tag registry.example.com/catalog/api:2.6.0 \
  --push .

docker buildx imagetools inspect registry.example.com/catalog/api:2.6.0 --format '{{ json .SBOM }}'
docker buildx imagetools inspect registry.example.com/catalog/api:2.6.0 --format '{{ json .Provenance }}'

• SBOM (--sbom=true) attaches a software bill of materials — every OS package and, depending on the generator, application dependency — as an attestation on the image index, inspectable without pulling the full image.
• Provenance (--provenance=mode=max) records how the image was built: the Dockerfile, build arguments, source materials, and builder identity. mode=min is the current BuildKit default when provenance is not explicitly disabled; mode=max is the level a compliance-driven pipeline should request explicitly.
• Attestations require an image index — the classic Docker image store cannot hold them. A docker-container (or remote/Kubernetes) buildx driver supports attestations unconditionally; the default docker driver needs the containerd image store enabled first.

Signing what you built#

An SBOM and provenance record describe the artifact; a signature proves it has not been substituted since. Signing an image (with cosign or an equivalent) and verifying that signature as an admission-control gate closes the loop between "we built this and scanned it" and "this exact digest is what's running."

cosign sign --yes registry.example.com/catalog/api@sha256:REPLACE_WITH_BUILT_DIGEST
cosign verify --certificate-identity-regexp '.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  registry.example.com/catalog/api@sha256:REPLACE_WITH_BUILT_DIGEST

From the Trenches: During a CVE response, a security team needed to know within the hour whether any running workload contained a vulnerable library version. Because every production image had an SBOM attestation generated at build time, the answer came from a registry query against stored SBOMs — no emergency re-scan of live containers, no guessing based on a base-image tag. Teams without that attestation had to pull and re-scan every candidate image live, under incident pressure, to answer the same question.

Multi-Architecture Builds with buildx#

Fleets increasingly mix linux/amd64 and linux/arm64 (Graviton, Apple Silicon CI runners, edge devices). buildx can produce a single manifest list that resolves to the correct architecture-specific image automatically at pull time.

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag registry.example.com/catalog/api:2.6.0 \
  --push .

This requires the docker-container driver (or an equivalent that isn't the plain daemon-backed docker driver) and, for architectures that can't run natively on the build host, QEMU emulation registered via binfmt. Emulated builds are correct but noticeably slower than native cross-compilation; for compiled languages, cross-compiling inside the Dockerfile (as the Go example above already does with CGO_ENABLED=0) and copying architecture-specific binaries into per-platform stages is often faster than emulating the whole build.

From the Trenches: A team added Apple Silicon developer laptops to their build fleet and started seeing "exec format error" on containers deployed to amd64 production nodes, because a laptop had built and pushed an arm64-only image under a shared tag. Standardizing CI as the only path allowed to push release tags — with multi-platform buildx builds — removed architecture mismatch as a class of incident.

A CI Build Pipeline End to End#

Putting the chapter's pieces together, a realistic pipeline stage looks like this:

docker buildx create --name ci --driver docker-container --use
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag registry.example.com/catalog/api:git-${CI_COMMIT_SHA:0:7} \
  --tag registry.example.com/catalog/api:${CI_TAG} \
  --label org.opencontainers.image.revision=${CI_COMMIT_SHA} \
  --sbom=true \
  --provenance=mode=max \
  --cache-from type=registry,ref=registry.example.com/catalog/api:buildcache \
  --cache-to type=registry,ref=registry.example.com/catalog/api:buildcache,mode=max \
  --secret id=npmrc,src=$HOME/.npmrc \
  --push .

• --cache-from / --cache-to type=registry share the build cache across ephemeral CI runners — without it, every fresh runner starts with a cold cache and none of this chapter's cache-mount work helps in CI the way it helps locally.
• The pipeline pushes once, with both a traceable commit tag and the release tag, and attaches SBOM and provenance in the same build.
• A downstream scan stage inspects the pushed digest's SBOM before the deployment stage is allowed to promote it.

Cache export modes#

mode=max is usually correct for a multi-stage Dockerfile built repeatedly in CI; mode=min is acceptable for a single-stage build where there is nothing intermediate worth caching.

Choosing a Base Image#

The final FROM in a multi-stage Dockerfile is a security and operability decision, not a style preference. Base images sit on a spectrum from "full OS with a shell and package manager" to "nothing but the application's own bytes."

From the Trenches: A service quietly worked on alpine in every environment except one customer's air-gapped deployment, where DNS resolution to an internal resolver failed intermittently. The root cause was musl libc's stricter, non-glibc-compatible resolver behavior against a resolver configuration that glibc tolerated. Switching to a distroless glibc-based image resolved it without changing a line of application code — the lesson wasn't "avoid Alpine," it was "test DNS behavior against your actual resolver topology before standardizing on musl."

Base image choice also interacts directly with the vulnerability-scanning and SBOM work from earlier in this chapter: a base with 280 average CVEs makes every scan noisy and slows down triage, while a near-zero-CVE base keeps scan output meaningful enough that a new finding is actually worth investigating instead of being lost in a wall of pre-existing base-image noise.

Rebuilding on a schedule, not just on code change#

A pinned base-image digest is reproducible, but reproducibility has a downside if nothing ever revisits the pin: a security fix released upstream never reaches a production image until something triggers a rebuild, and "a developer happens to touch that Dockerfile" is not a schedule. Automated dependency-update tooling (Renovate, Dependabot) can open a pull request whenever a pinned base image's digest changes upstream, which turns "someone remembers to bump the base image" into a reviewable, recurring pipeline event instead of a manual chore that quietly stops happening.

{
  "docker": {
    "enabled": true
  },
  "packageRules": [
    { "matchDatasources": ["docker"], "schedule": ["before 6am on monday"] }
  ]
}

A weekly base-image bump PR, gated by the same build-scan-sign pipeline as any other change, keeps the CVE count in the earlier comparison table meaningfully low over time instead of accurate only on the day the image was first built.

From the Trenches: A service pinned its base image digest for reproducibility and then never revisited it for over a year, treating the pin as a one-time decision rather than a maintained dependency. A routine audit found it was running a base image with dozens of unpatched high-severity CVEs, all fixed upstream months earlier — the pin had done exactly what it was designed to do, faithfully reproducing a stale, vulnerable artifact every single build. Reproducibility and staying current are not in tension; they just require an explicit, scheduled process to move the pin forward deliberately instead of leaving it frozen indefinitely.

Dockerfile Instruction Reference Notes#

A few instructions are consistently misused in ways that matter operationally, beyond what a syntax reference documents.

ARG vs ENV#

ARG values exist only at build time and are not present in the running container unless explicitly re-declared as ENV. ENV values persist into the container and are visible via docker inspect and printenv at runtime.

ARG BUILD_VERSION=dev
ENV APP_VERSION=${BUILD_VERSION}

An ARG declared before the first FROM is available to use in FROM lines themselves (for parameterizing a base image tag) but does not automatically carry into any stage — each stage that needs it must redeclare ARG after its own FROM.

ARG NODE_VERSION=20.17
FROM node:${NODE_VERSION}-bookworm-slim AS base
ARG NODE_VERSION
RUN echo "built on Node ${NODE_VERSION}"

USER, WORKDIR, and SHELL#

USER should appear before the container's main process starts, ideally as early as the runtime stage allows once file ownership is set correctly with COPY --chown. Running as root inside the final stage and only "dropping privileges" at docker run time via --user is weaker than baking a non-root user into the image, because any tooling that inspects the image directly (a scanner, an admission policy checking USER metadata) sees the image's own declared identity.

RUN addgroup --system app && adduser --system --ingroup app app
COPY --chown=app:app --from=build /out/api /usr/local/bin/api
USER app

SHELL changes which shell RUN instructions with shell form use — useful for switching to ["/bin/bash", "-o", "pipefail", "-c"] so a piped command's failure (curl ... | tar ...) is not silently swallowed by the default shell form, which only checks the last command's exit code.

ONBUILD — usually the wrong tool now#

ONBUILD registers instructions that run only when another Dockerfile uses this image as its FROM. It made sense for framework base images distributed years ago; multi-stage builds solve the same "parameterize a common build pattern" problem more transparently, because the actual instructions that run are visible in the consuming Dockerfile instead of hidden in a base image's metadata. Avoid introducing new ONBUILD usage; treat existing ONBUILD base images as a legacy pattern to plan away from.

STOPSIGNAL and graceful shutdown#

Part 1 covered SIGTERM/SIGKILL at the runtime level. STOPSIGNAL lets an image declare a different default signal for applications that don't handle SIGTERM for graceful shutdown (some JVM configurations expect SIGINT, for example).

STOPSIGNAL SIGINT

Declaring this in the image is more reliable than expecting every docker run or orchestrator manifest to remember to override the stop signal per service.

Vulnerability Scanning in the Pipeline#

An SBOM (covered earlier in this chapter) is an inventory; a scanner is what turns that inventory into an actionable finding by matching it against known-vulnerability databases.

trivy image --severity CRITICAL,HIGH --exit-code 1 registry.example.com/catalog/api:2.6.0
grype registry.example.com/catalog/api:2.6.0 --fail-on critical
docker scout cves registry.example.com/catalog/api:2.6.0

A practical default: run Trivy (or an equivalent broad scanner) as the primary CI gate with --exit-code 1 on critical/high findings, and treat a second scanner as a periodic cross-check on production-facing images rather than doubling scan time on every build. Feed scan results the same SBOM generated at build time where the tool supports it (trivy sbom accepts a CycloneDX or SPDX document directly) instead of re-deriving the dependency tree from the image a second time.

From the Trenches: A team's CI gate scanned images but only failed the build on findings with a CVSS score above a fixed threshold, with no allowlist mechanism. The first time a base-image CVE had no available fix yet, every single build across every service failed simultaneously, and the team disabled the gate under deadline pressure rather than triage it properly — after which it stayed disabled for months. A workable gate needs a time-boxed exception/allowlist path (a tracked, expiring waiver) for exactly this situation, not just a hard pass/fail threshold.

Linting and Build Reproducibility#

hadolint statically analyzes a Dockerfile against a rule set derived from real-world Dockerfile mistakes — unpinned base image tags, apt-get upgrade inside a build (nondeterministic and usually unintended), missing --no-install-recommends, and instruction ordering that defeats caching.

hadolint Dockerfile

Running it as a pre-commit hook or an early CI stage catches most of this chapter's caching and security mistakes before a build even starts, at effectively zero cost.

Reproducibility beyond caching#

Cache-friendly ordering (covered earlier) makes rebuilds fast; reproducibility is a stronger property — that a rebuild from the same inputs produces an identical digest. Two practical levers:

• Pin base image digests, not just tags, in a pipeline where reproducibility is audited: FROM node:20.17-bookworm-slim@sha256:... rather than the floating tag alone.
• Avoid instructions that embed the current time or non-deterministic ordering — a build script that writes date output into a file that gets copied into the image, or a find invocation whose output ordering depends on filesystem state, both break bit-for-bit reproducibility even when every declared input is pinned.

Full bit-for-bit reproducibility (matching upstream projects like Debian's "reproducible builds" effort) is a deep rabbit hole that few application teams need to fully solve; the pragmatic middle ground most teams should reach is "pinned inputs, deterministic instructions, and a promotion workflow that never rebuilds a promoted artifact" — the last part already covered under tagging and promotion above.

Registry Choices and Image Retention#

The registry is not just storage — it's the enforcement point for retention, access control, and (per the earlier section) attestation storage.

Every option needs an explicit retention policy — an unmanaged registry grows without bound as CI pushes a new tag on every commit. A typical policy: keep all digests referenced by a currently deployed manifest indefinitely, keep the last N commit-tagged builds per branch for a bounded window (for bisecting), and expire untagged/dangling manifests aggressively since nothing can reference them by tag.

aws ecr put-lifecycle-policy --repository-name catalog/api --lifecycle-policy-text file://ecr-lifecycle.json

From the Trenches: A registry with no lifecycle policy grew to tens of thousands of tags over two years of daily CI builds, and a routine "list all images" API call in a deployment tool started timing out. The fix wasn't just adding a retention policy going forward — it required an audited one-time cleanup to confirm no active deployment manifest referenced a digest about to be pruned, since a naive "delete anything older than 90 days" policy can silently break a long-lived environment still pinned to an old digest.

A Complete GitHub Actions Build Pipeline#

Putting every control from this chapter into one concrete, runnable pipeline definition makes the abstract guidance checkable against a real file.

name: build-and-promote
on:
  push:
    branches: [main]
    tags: ["v*"]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write   # required for keyless cosign signing
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-qemu-action@v3
      - uses: docker/setup-buildx-action@v3
        with:
          driver: docker-container

      - uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ github.actor }}
          password: ${{ secrets.REGISTRY_TOKEN }}

      - name: Build, scan-ready, sign-ready image
        uses: docker/build-push-action@v6
        id: build
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: |
            registry.example.com/catalog/api:git-${{ github.sha }}
          labels: |
            org.opencontainers.image.revision=${{ github.sha }}
          sbom: true
          provenance: mode=max
          cache-from: type=registry,ref=registry.example.com/catalog/api:buildcache
          cache-to: type=registry,ref=registry.example.com/catalog/api:buildcache,mode=max

      - name: Scan the pushed digest
        run: |
          trivy image --severity CRITICAL,HIGH --exit-code 1 \
            registry.example.com/catalog/api@${{ steps.build.outputs.digest }}

      - name: Sign the pushed digest
        uses: sigstore/cosign-installer@v3
      - run: |
          cosign sign --yes \
            registry.example.com/catalog/api@${{ steps.build.outputs.digest }}

  promote:
    needs: build
    if: startsWith(github.ref, 'refs/tags/v')
    runs-on: ubuntu-latest
    steps:
      - uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ github.actor }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      - name: Re-tag the verified digest as the release
        run: |
          docker buildx imagetools create \
            --tag registry.example.com/catalog/api:${{ github.ref_name }} \
            registry.example.com/catalog/api@${{ needs.build.outputs.digest }}

Every control introduced earlier in the chapter appears exactly once, at the stage where it belongs: multi-platform build and cache export in the build job, SBOM/provenance attached at build time, scanning and signing gated on the digest the build actually produced, and promotion as a re-tag operation that never rebuilds.

Runner sizing for multi-platform builds#

Emulated cross-architecture builds (the linux/arm64 leg on a standard amd64 GitHub-hosted runner, via QEMU) are meaningfully more CPU- and memory-hungry than a native build of the same Dockerfile, and a runner sized correctly for a single-platform build can silently thrash or time out once a second emulated platform is added.

A team hitting inexplicable timeouts only on the arm64 leg of a multi-platform build should suspect emulation overhead before suspecting the Dockerfile — cross-compiling natively where the toolchain supports it (as this chapter's Go example already does) sidesteps the emulation cost entirely for compiled languages.

Rootless and Daemonless Build Alternatives#

Not every environment should or can run a privileged Docker daemon — a shared CI runner, a Kubernetes-native build pipeline, or a security policy that forbids daemon access are all common reasons to look past docker build itself.

From the Trenches: A platform team exposing /var/run/docker.sock to every CI job to allow container builds discovered, during a security review, that any job — including jobs running untrusted third-party pull-request code — could use that socket to launch a privileged container and read arbitrary host files. Moving CI builds to Kaniko (no daemon socket needed at all) closed the finding without giving up Dockerfile-based builds.

Cache backends beyond the registry#

--cache-from/--cache-to type=registry (used in this chapter's CI pipeline) is portable across any CI system, but GitHub Actions also exposes a native cache backend that avoids pushing cache layers to the image registry at all:

      - uses: docker/build-push-action@v6
        with:
          cache-from: type=gha
          cache-to: type=gha,mode=max

type=gha stores cache in GitHub's own Actions cache service, scoped to the repository, with its own eviction policy (typically a rolling size- and age-based limit). It avoids cluttering the image registry with cache-only manifests, at the cost of being tied to one CI platform — a team running builds across GitHub Actions and a second CI system would still need type=registry for the portable case.

Layer Count, Squashing, and Image Flattening#

Each RUN, COPY, and ADD instruction typically produces its own layer. This is usually good — it's what makes caching and incremental pulls work — but an excessive number of small layers (or one enormous layer from an unbatched sequence of package operations) has real costs: registry push/pull overhead per layer, and, in the unbatched case, a layer that can't be partially cached at all.

# Less efficient: three layers, and an update layer that can go stale
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# More efficient: one layer, cache-safe, and cleanup happens in the same layer
RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

The second form matters for a subtle reason beyond layer count: if rm -rf /var/lib/apt/lists/* runs in a later layer than apt-get install, the apt cache still exists in the earlier layer's content and still contributes to image size — removing a file in a later layer does not shrink an earlier layer, the same lesson from this chapter's secrets section applied to disk space instead of credentials.

--squash and post-build flattening#

BuildKit and some registries support flattening an image's layers into fewer (or one) final layer after the build. This can reduce the layer count exposed downstream, but it destroys the cache-locality benefit the earlier sections of this chapter deliberately engineered — a squashed image cannot share a partial-layer cache hit with a future build the way a well-ordered multi-layer image can. Reach for squashing only for a specific downstream constraint (a registry with a hard layer-count limit, an air-gapped transport format that benefits from fewer files), not as a default optimization.

Measuring layer contribution with dive#

docker history shows per-instruction size but not which specific files live in which layer. dive opens an interactive view of every layer's actual file contents and flags wasted space — files added in one layer and duplicated or deleted in another.

dive registry.example.com/catalog/api:2.6.0
CI=true dive registry.example.com/catalog/api:2.6.0 --lowestEfficiency=0.95

The CI=true mode returns a pass/fail exit code against an efficiency threshold, which makes it usable as a CI gate against image bloat the same way a vulnerability scanner gates security findings — a genuinely oversized layer (an accidentally copied node_modules cache, a build tool left in the final stage) fails the build instead of quietly shipping.

Offline and Air-Gapped Image Delivery#

Some environments — regulated on-premises deployments, disconnected edge sites, classified networks — cannot pull directly from an internet-facing registry. docker save/docker load (or buildx build --output type=oci,dest=... for an OCI-layout tarball) move an image as a file instead of a registry pull.

docker buildx build --output type=oci,dest=catalog-api-2.6.0.tar --tag catalog-api:2.6.0 .
# transport catalog-api-2.6.0.tar across the air gap by an approved offline process
docker load --input catalog-api-2.6.0.tar
docker tag catalog-api:2.6.0 internal-registry.local/catalog/api:2.6.0
docker push internal-registry.local/catalog/api:2.6.0

The attestations and signature discussed earlier in this chapter should travel with the tarball, not be regenerated on the disconnected side — provenance and SBOM data describe the original build, and re-signing on the air-gapped side would only prove the disconnected environment repackaged the file, not that the original build was trustworthy. Verify the signature and SBOM before the tarball crosses the air gap, and carry the verification record across with it.

From the Trenches: An air-gapped deployment process re-built images from source inside the disconnected network "to be safe," reasoning that transporting a tarball felt less auditable than a fresh build. This actually removed the guarantee it was trying to add: the disconnected build used whatever base-image and package-mirror state happened to be available inside the air gap, which had already drifted from what was scanned and approved outside it. Transporting the exact verified artifact (with its attestations) and verifying the signature on arrival is the auditable path; rebuilding blind on the other side of the gap is not.

Build Context Sources and Private Dependencies#

The "build context" is whatever set of files BuildKit can see and COPY/ADD from — usually a local directory, but not always.

# Local directory context (the common case)
docker build -t catalog-api:dev .

# Remote Git repository as the context, no local checkout needed
docker buildx build -t catalog-api:dev https://github.com/example/catalog-api.git#main

# Dockerfile from stdin, context still local
docker build -t catalog-api:dev -f - . <<'EOF'
FROM alpine:3.21
RUN echo "example"
EOF

A Git-URL context is genuinely useful for a build triggered by a webhook that only has a repository reference, not a checked-out working tree — BuildKit clones the reference itself. It has a real limitation worth knowing before relying on it: build secrets and .dockerignore behavior work against the cloned tree, so a private dependency fetched during the build (not present in the repository) still needs the type=secret mechanism covered earlier, not an assumption that the surrounding CI checkout's credentials are somehow available inside the isolated clone.

Private base images and private package registries#

A Dockerfile that pulls FROM registry.example.com/internal/base:1.4 or runs npm ci against a private npm registry needs authentication available to BuildKit itself, which runs isolated from the invoking shell's ambient credentials.

docker login registry.example.com
docker buildx build --tag catalog-api:dev .

For a private base image, a prior docker login on the build host (or CI's registry-login step) is normally sufficient because BuildKit reuses the Docker credential store for FROM pulls. For a private package registry consumed inside a RUN instruction, that ambient login does not automatically propagate into the build's isolated environment — this is exactly the case the type=secret mount (covered earlier for npm ci against a private .npmrc) exists to solve, and it is a common point of confusion: "I'm already logged in, why can't the build reach my private registry?" The answer is that a daemon-level registry login only helps for pulling image layers via FROM, not for arbitrary network calls a RUN instruction makes.

Debugging a Failed or Suspicious Build#

A build that fails deep inside a RUN instruction, or one that "succeeds" but produces a suspicious result, needs different tools than a running container does.

docker build --progress=plain --no-cache -t catalog-api:debug .
BUILDKIT_PROGRESS=plain docker buildx build -t catalog-api:debug .

--progress=plain prints full, unbuffered instruction output instead of the default collapsed TTY view — essential when a RUN instruction's actual error is hidden behind a truncated progress bar. --no-cache rules out a stale or corrupted cache entry as the cause before assuming the Dockerfile itself is wrong; a cache hit against an earlier, subtly different build context is a common source of "it worked yesterday" confusion.

Stopping at an intermediate stage#

Multi-stage Dockerfiles let you target a specific stage instead of building the whole file, which turns a hard-to-reproduce final-stage failure into an inspectable intermediate image.

docker build --target build -t catalog-api:build-stage .
docker run --rm -it catalog-api:build-stage /bin/sh

This drops into the exact filesystem state right after the build stage's instructions ran, before anything was copied into the final runtime stage — often the fastest way to answer "did the compiled artifact even end up where the final COPY --from=build expects it."

From the Trenches: A build that succeeded for months began intermittently failing a COPY step with "no such file or directory" for a file that was clearly present in the repository. The cause was a .dockerignore pattern (*.env) that had been broadened during an unrelated cleanup and started unintentionally matching a same-extension configuration template the build genuinely needed. --progress=plain combined with a deliberate docker build --no-cache isolated it to the .dockerignore change within minutes, versus hours of suspecting the Dockerfile logic itself.

Running Tests as a Build Stage#

A dedicated test stage runs the same dependency-and-source layers already built for the application, so tests execute against exactly the artifact being shipped rather than against a separately maintained CI environment that can drift from the image.

FROM build AS test
RUN --mount=type=cache,target=/root/.npm \
    npm run test -- --ci --coverage

FROM runtime AS final
COPY --from=build /app/dist /app/dist

docker build --target test -t catalog-api:test .
docker buildx build --target test --output type=cacheonly .

--target test builds only up to the test stage — useful for a CI step that should fail the pipeline on a test failure without needing the final runtime image at all. --output type=cacheonly builds a stage purely for its cache and pass/fail exit code, discarding the resulting filesystem entirely, which is the right choice when the stage's only purpose is "did this succeed," not "keep this image."

From the Trenches: A team's unit tests passed reliably in a dedicated CI test job but a production regression shipped anyway, because the CI test environment installed dependencies via a slightly different Node version than the Dockerfile's base image used. Moving unit tests into a test build stage sharing the exact same base image and lockfile-driven install as the shipped artifact eliminated the class of "passes in CI, fails in the image" discrepancy entirely, because there was no longer a second, independently drifting environment.

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. A multi-stage Dockerfile's final image is much larger than expected. How do you find out why?#

Run docker history --no-trunc <image> to see each layer's size and originating instruction, and confirm the final FROM stage is actually a minimal runtime base rather than accidentally reusing a build stage. Check that COPY --from=<stage> only copies the specific artifact needed, not an entire directory that still contains build tooling or source archives. If the base image itself is unexpectedly large, compare it against a distroless or slim equivalent.

2. CI cache hit rate is near zero even though the Dockerfile orders instructions correctly. What's the likely cause?#

Each CI job almost certainly runs on a fresh, ephemeral runner with no local BuildKit cache directory carried over from the previous run. Correct instruction ordering only helps within a single build; across separate runners it does nothing without an explicit shared cache. Add --cache-from/--cache-to type=registry (or the CI platform's native BuildKit cache backend) so the cache persists in a location every runner can reach.

3. A compliance review asks whether a specific CVE-affected library version shipped in a production image built four months ago. How do you answer without pulling the image?#

If the image was built with --sbom=true, query the stored SBOM attestation against the image's digest directly from the registry with docker buildx imagetools inspect --format '{{ json .SBOM }}' and search it for the library and version. This avoids pulling and re-scanning the full image and gives an authoritative answer tied to the exact digest that was deployed, provided the deployment record still identifies that digest.

4. A team wants to switch a service's base image from debian:bookworm to a distroless image, but the on-call runbook relies on docker exec ... /bin/sh for live diagnosis. How do you reconcile this?#

Recognize that distroless images deliberately remove the shell as a security property, not an oversight — reintroducing a shell defeats the point. Replace shell-based interactive diagnosis with structured, externally captured signals: verbose application logging, docker inspect/docker stats for resource and state evidence, a dedicated ephemeral debug container (docker run --rm -it --pid=container:<target> --network=container:<target> <debug-image> sharing the target's namespaces) for cases that genuinely need process-level inspection, and updating the runbook to use those tools instead of an in-container shell.

5. A promoted release tag was rebuilt from source instead of re-tagged from the staging digest, and the two digests differ even though nothing in the Dockerfile changed. What's the most likely explanation, and what's the fix?#

The most likely cause is a non-pinned input: a floating base image tag that received an upstream update between the two builds, a package manager pulling a newer patch version with no version pin, or a build step that embeds a timestamp or non-deterministic file ordering. The fix is twofold — pin the specific inputs that drifted (base image digest, package versions) to restore reproducibility going forward, and change the promotion process itself to re-tag the exact verified digest via docker buildx imagetools create rather than rebuilding for the release tag, so this class of drift can never again separate what was tested from what ships.

6. A docker build --target build intermediate image shows the compiled binary is missing entirely, even though the source file that should produce it is present. Where do you look first?#

Check the exact build command inside the build stage — a common cause is a build command that silently succeeds with exit code 0 while writing its output to a path the Dockerfile doesn't expect, or a working directory mismatch between where WORKDIR places the build and where the compiler was configured to emit output. Run the same build command interactively inside a container from that intermediate stage (docker run --rm -it catalog-api:build-stage /bin/sh, then re-run the compile command manually) to see its actual output path before assuming the Dockerfile's COPY --from=build line is wrong — copying from the wrong path is a symptom, not the root cause, if the artifact was never produced where expected in the first place.

Summary and What's Next#

BuildKit turns a Dockerfile from a sequential script into a cacheable, parallelizable build graph. Ordering instructions around cache locality, isolating build tooling behind multi-stage boundaries, and using cache/bind/secret mounts for their distinct purposes produce faster, smaller, and more auditable images. Base image choice, linting, and vulnerability scanning turn "it built" into "it's safe to run"; tagging and promotion discipline — commit-traceable tags, digest-based promotion, SBOM and provenance attestations, and signature verification — turn the build output into something a security or incident review can trust without re-deriving it from scratch.

The throughline across every section is the same one from Part 1: an image is a declared, auditable artifact, and every practice here exists to keep that declaration trustworthy from the first RUN instruction through a production incident months later. A team that can answer "what's in this image, how was it built, and is this exact digest what's running" without guesswork has already solved most of what makes container delivery hard in practice, and can extend that same discipline confidently to the multi-service, multi-host workflows the rest of this series covers next.

Part 3 moves from building the image to running it as part of a real application: user-defined networks and DNS, published ports, volumes and bind mounts for persistent state, and Docker Compose for coordinating multiple services on one host.