# Automation, CI/CD & GitOps — Part 10: CircleCI
> **Series:** Automation, CI/CD & GitOps (10 of 14)
> **Part 1:** `01-cicd-fundamentals.md` — CI/CD Fundamentals
> **Part 2:** `02-infrastructure-as-code.md` — Infrastructure as Code
> **Part 3:** `03-gitops.md` — GitOps
> **Part 4:** `04-github-actions.md` — GitHub & GitHub Actions
> **Part 5:** `05-github-security-governance.md` — GitHub Security & Governance
> **Part 6:** `06-gitlab-cicd.md` — GitLab & GitLab CI/CD
> **Part 7:** `07-bitbucket-pipelines.md` — Bitbucket & Bitbucket Pipelines
> **Part 8:** `08-azure-devops.md` — Azure DevOps
> **Part 9:** `09-jenkins.md` — Jenkins
> **Part 10:** This file — CircleCI
> **Part 11:** `11-tekton.md` — Tekton
> **Part 12:** `12-monorepo-cicd.md` — Monorepo CI/CD Strategies
> **Part 13:** `13-progressive-delivery.md` — Progressive Delivery with Argo Rollouts & Flagger
> **Part 14:** `14-self-hosted-runner-scaling.md` — Self-Hosted Runner Scaling & Cost Optimization
> **Questions:** `questions.md`
## Table of Contents
1. [Where CircleCI Fits — Cloud-Native, Performance-Focused SaaS](#where-circleci-fits--cloud-native-performance-focused-saas)
2. [CircleCI's Credits Pricing Model](#circlecis-credits-pricing-model)
3. [Anatomy of `.circleci/config.yml`](#anatomy-of-circleciconfigyml)
4. [Jobs, Workflows, and Executors](#jobs-workflows-and-executors)
5. [A Minimal Pipeline, Built Up Step by Step](#a-minimal-pipeline-built-up-step-by-step)
6. [Executors in Depth — Docker, Machine, macOS](#executors-in-depth--docker-machine-macos)
7. [Workflow Orchestration — `requires`, Fan-Out/Fan-In, Approval Jobs](#workflow-orchestration--requires-fan-outfan-in-approval-jobs)
8. [Parallelism and Test Splitting](#parallelism-and-test-splitting)
9. [Caching and Workspaces](#caching-and-workspaces)
10. [Orbs — CircleCI's Reusability Model](#orbs--circlecis-reusability-model)
11. [Contexts — Secrets Shared Across Projects](#contexts--secrets-shared-across-projects)
12. [OIDC — Eliminating Long-Lived Cloud Credentials](#oidc--eliminating-long-lived-cloud-credentials)
13. [A Full Worked OIDC Example: Deploying to AWS](#a-full-worked-oidc-example-deploying-to-aws)
14. [Self-Hosted Runners](#self-hosted-runners)
15. [Docker Layer Caching and Remote Docker](#docker-layer-caching-and-remote-docker)
16. [Pipeline Parameters and Dynamic Config](#pipeline-parameters-and-dynamic-config)
17. [Insights and Test Analytics](#insights-and-test-analytics)
18. [A Full Realistic Multi-Stage Pipeline](#a-full-realistic-multi-stage-pipeline)
19. [CircleCI vs. the Rest — Where It Genuinely Wins](#circleci-vs-the-rest--where-it-genuinely-wins)
20. [Common Mistakes](#common-mistakes)
21. [Worked Practice Problems](#worked-practice-problems)
22. [Summary and What's Next](#summary-and-whats-next)
---
## Where CircleCI Fits — Cloud-Native, Performance-Focused SaaS
Where Part 9's Jenkins sits at the self-hosted, maximum-control end of this series' spectrum, CircleCI sits close to the opposite end: a **cloud-native, SaaS-first CI/CD platform**, git-host-agnostic like Jenkins and Azure Pipelines (it connects to GitHub, GitLab, or Bitbucket repositories rather than hosting its own), but with zero controller infrastructure for the adopting team to run at all — the vendor owns 100% of the execution and scaling layer.
```mermaid
graph TD
CircleCI["CIRCLECI:
fully vendor-hosted
execution (or opt-in
self-hosted runners),
git-host-agnostic like
Jenkins/Azure, but with
ZERO controller ops —
an engineering culture
strongly oriented around
raw build PERFORMANCE
(parallelism, caching,
resource classes)"]
```
**CircleCI's most distinctive positioning, worth stating precisely rather than generically:** among every platform in this series, CircleCI has historically put the most product emphasis specifically on **pipeline execution speed** — fine-grained parallelism and automatic test splitting (covered in depth shortly), Docker layer caching, and a wide selection of tunable "resource classes" (specific vCPU/RAM/architecture combinations, including Arm and GPU options) that a team can pick per job to trade cost against wall-clock time deliberately. Where GitLab's product identity (Part 6) centers on integrated security/compliance breadth and Bitbucket's (Part 7) centers on Atlassian-ecosystem integration, CircleCI's centers on making the pipeline itself as fast as possible — a genuinely different axis of competition worth recognizing as such.
This chapter follows the same comparative structure as Parts 6-9, but reads a little differently in one respect: rather than a broad platform-vs-platform feature comparison at every turn, several sections here zoom into one specific, deep capability (test splitting, Docker Layer Caching, flaky-test analytics) that has no precise equivalent elsewhere in this series — worth reading those sections for what's genuinely new, not just as CircleCI's rename of an already-covered concept.
---
## CircleCI's Credits Pricing Model
CircleCI bills in **credits**, a usage unit that's worth understanding precisely since it differs structurally from every other platform's minutes-based billing covered so far in this series.
| Plan | Included credits/month | Typical fit |
|---|---|---|
| **Free** | 30,000 | Small projects, evaluation |
| **Performance** | 30,000 included, pay-as-you-go beyond | Small-to-mid paid teams |
| **Scale** | Custom, volume-based | Larger organizations |
**The key structural difference from a flat per-minute billing model:** credit consumption varies by **resource class** — a job running on a larger, more powerful resource class (more vCPU/RAM, a GPU, an Arm architecture) consumes credits at a proportionally higher rate per minute than a small default resource class, directly analogous to the OS-based minute multipliers already covered for GitHub (Part 4) and Bitbucket (Part 7), but generalized across an entire menu of machine sizes rather than just a handful of OS choices. This makes resource-class selection a genuine, deliberate cost/speed lever in CircleCI specifically — picking a larger resource class for a slow test suite is a conscious tradeoff a team makes explicitly, in a way that's less directly exposed as a dial on some other platforms.
**Self-hosted runners (covered in depth later in this chapter) consume zero credits at all** — a genuinely important detail for a team running large build volumes, since crossing roughly 50,000 build-minutes a month on hosted executors starts to make self-hosted infrastructure's fixed cost pay for itself compared to continued pay-as-you-go credit consumption, mirroring the same hosted-vs-self-hosted cost crossover logic already covered for GitHub Actions in Part 4.
---
## Anatomy of `.circleci/config.yml`
A single YAML file at `.circleci/config.yml`, structured around four top-level concepts:
```mermaid
graph TD
Config[".circleci/config.yml"] --> Version["version: 2.1 —
REQUIRED, unlocks orbs
and reusable config"]
Config --> Orbs["orbs: — imported
reusable packages"]
Config --> Jobs["jobs: — named units
of work, each with its
OWN executor"]
Config --> Workflows["workflows: — orchestrates
WHICH jobs run, in WHAT
order, with WHAT
dependencies"]
```
- **`version: 2.1`** — required at the top of every modern config; unlocks orbs, reusable executors, and parameterized jobs (version 2.0, still seen in older configs, lacks these).
- **`jobs:`** — a named unit of work, each with its own `executor` (or inline Docker image) and a list of `steps`.
- **`workflows:`** — the orchestration layer, declaring which jobs run, their ordering via `requires`, and any parallelism/fan-out shape — CircleCI's separation of "what a job does" from "how jobs relate to each other" is more explicit and more central to the config's structure than in most other platforms covered so far.
```yaml
version: 2.1
jobs:
build:
docker:
- image: cimg/node:20.11
steps:
- checkout
- run: npm ci
- run: npm run build
test:
docker:
- image: cimg/node:20.11
steps:
- checkout
- run: npm ci
- run: npm test
workflows:
build-and-test:
jobs:
- build
- test
```
**Notice `jobs:` and `workflows:` are genuinely separate top-level sections, unlike every prior platform in this series where a job's ordering (GitHub's `needs:`, GitLab's stages, Azure's `dependsOn`) is declared as a property directly on the job itself.** This separation is a deliberate CircleCI design choice — the same set of `jobs:` can be referenced by *multiple* different `workflows:` blocks in one config (e.g., a `build-and-test` workflow on every push, plus a separate `nightly-full-suite` workflow reusing the same `test` job with different scheduling), without duplicating the job definitions themselves.
---
## Jobs, Workflows, and Executors
Each `job` specifies its own **executor** — the environment its steps run in — independently of every other job:
```yaml
jobs:
build:
docker:
- image: cimg/node:20.11 # a Docker-based executor
steps: [checkout, run: npm ci, run: npm run build]
integration-test:
machine:
image: ubuntu-2204:current # a full VM executor, needed for e.g. Docker-in-Docker
steps: [checkout, run: docker compose up -d, run: npm run test:integration]
```
`steps:` inside a job is an ordered list — `checkout` (a built-in step that clones the repo, closer to GitHub's explicit `actions/checkout` than GitLab's automatic checkout) followed by `run:` (shell commands) or orb-provided steps (covered shortly).
```mermaid
graph TD
Workflow["workflows.build-and-test"] --> Build["job: build
(docker executor)"]
Workflow --> Test["job: integration-test
(machine executor)"]
Build -.independent executor choice per job.-> Test
```
**Each job independently choosing its own executor type is a genuine structural difference worth flagging against GitHub Actions specifically** — a GitHub job's `runs-on:` picks a runner OS, but every job on a given runner type gets the same underlying VM shape; CircleCI's per-job executor choice (Docker vs. a full VM vs. macOS, each independently sized via `resource_class`) is a more explicit, more granular version of the same idea, directly serving the performance-tuning culture covered in the opening section.
---
## A Minimal Pipeline, Built Up Step by Step
**Step 1 — bare minimum:**
```yaml
version: 2.1
jobs:
hello:
docker:
- image: cimg/base:current
steps:
- run: echo "hello"
workflows:
main:
jobs:
- hello
```
**Step 2 — a real Node project:**
```yaml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.11
steps:
- checkout
- run: npm ci
- run: npm test
workflows:
main:
jobs:
- test
```
**Step 3 — splitting build and test into separate, dependency-ordered jobs:**
```yaml
version: 2.1
jobs:
build:
docker: [{ image: cimg/node:20.11 }]
steps:
- checkout
- run: npm ci
- run: npm run build
- persist_to_workspace: { root: ., paths: [dist] } # covered fully in the Caching section
test:
docker: [{ image: cimg/node:20.11 }]
steps:
- checkout
- attach_workspace: { at: . }
- run: npm test
workflows:
main:
jobs:
- build
- test:
requires: [build]
```
**Step 4 — adding a resource class and a manual approval gate before deploy:**
```yaml
version: 2.1
jobs:
build:
docker: [{ image: cimg/node:20.11 }]
resource_class: medium
steps: [checkout, run: npm ci, run: npm run build]
test:
docker: [{ image: cimg/node:20.11 }]
steps: [checkout, run: npm ci, run: npm test]
deploy:
docker: [{ image: cimg/base:current }]
steps: [checkout, run: ./deploy.sh]
workflows:
main:
jobs:
- build
- test:
requires: [build]
- hold-for-approval:
type: approval
requires: [test]
- deploy:
requires: [hold-for-approval]
```
---
## Executors in Depth — Docker, Machine, macOS
| Executor type | What it is | Typical use |
|---|---|---|
| **`docker`** | Job runs inside one or more specified Docker containers | The default choice — fast startup, lightweight |
| **`machine`** | A full VM, with Docker itself available inside it | Needed for Docker-in-Docker, privileged operations, or anything a container can't do |
| **`macos`** | An actual macOS VM | iOS/macOS app builds |
| **`windows`** | A Windows VM/container | .NET or Windows-specific builds |
```yaml
jobs:
build-image:
machine:
image: ubuntu-2204:current
resource_class: large
steps:
- checkout
- run: docker build -t myapp .
```
**`resource_class` is available on every executor type and is the primary performance/cost dial covered in the pricing section** — options range from small (default, cheapest) through multiple large tiers, plus Arm and GPU-enabled variants for genuinely specialized workloads (ML training jobs, Arm-native builds). Picking the right resource class for a given job's actual bottleneck (CPU-bound compile step vs. I/O-bound test suite) is a real, worthwhile tuning exercise in a CircleCI-heavy pipeline, in a way that's less directly exposed as a per-job lever on platforms that default to one fixed runner shape.
---
## Workflow Orchestration — `requires`, Fan-Out/Fan-In, Approval Jobs
`requires:` is CircleCI's dependency-declaration mechanism, functionally equivalent to GitHub's `needs:` and GitLab's `needs:` — a job starts as soon as everything it `requires` finishes, building a genuine DAG rather than assuming strict sequential ordering:
```yaml
workflows:
main:
jobs:
- build
- unit-test:
requires: [build]
- lint:
requires: [build]
- integration-test:
requires: [unit-test, lint] # FAN-IN: waits for BOTH
- deploy-staging:
requires: [integration-test]
- deploy-canary:
requires: [deploy-staging]
- deploy-full:
requires: [deploy-canary]
```
```mermaid
flowchart LR
Build["build"] --> Unit["unit-test"]
Build --> Lint["lint"]
Unit --> Integ["integration-test"]
Lint --> Integ
Integ --> Staging["deploy-staging"]
Staging --> Canary["deploy-canary"]
Canary --> Full["deploy-full"]
```
`unit-test` and `lint` fan out in parallel from `build` (both only depend on it, not on each other), then fan back in at `integration-test` (which needs both to finish) — the exact same DAG-building pattern already covered for GitHub and GitLab, CircleCI's own `requires:` syntax.
**A `type: approval` job (shown in the previous section's Step 4) is CircleCI's manual-gate mechanism** — a pseudo-job with no `steps:` or executor of its own, that simply pauses the workflow until a human clicks approve in the CircleCI UI, the same underlying concept as every prior platform's manual approval gate, expressed here as a first-class workflow-graph node rather than a property attached to an "environment" object.
---
## Parallelism and Test Splitting
Beyond fanning out independent *jobs*, CircleCI has a genuinely distinctive feature for parallelizing **within a single job**: automatic **test splitting** across multiple identical parallel instances of the same job.
```yaml
jobs:
test:
docker: [{ image: cimg/node:20.11 }]
parallelism: 4 # run 4 IDENTICAL copies of this job, in parallel
steps:
- checkout
- run: npm ci
- run:
command: |
TESTFILES=$(circleci tests glob "test/**/*.spec.js" | circleci tests split --split-by=timings)
npx jest $TESTFILES
```
```mermaid
graph TD
AllTests["Full test suite:
400 test files"] --> Split["circleci tests split
--split-by=timings"]
Split --> P1["Container 1:
~100 files
(balanced by
PAST runtime)"]
Split --> P2["Container 2:
~100 files"]
Split --> P3["Container 3:
~100 files"]
Split --> P4["Container 4:
~100 files"]
```
**`--split-by=timings` is the detail worth understanding precisely, since it's what makes this genuinely smarter than a naive even split.** CircleCI tracks how long each individual test file took to run in *previous* builds, and uses that historical timing data to balance the 4 parallel containers by expected *runtime*, not just file count — a naive "100 files each" split can leave one container with several unusually slow files while others finish early and sit idle; timing-based splitting actively balances for wall-clock finish time instead. **This is a genuinely distinctive capability among the platforms covered in this series** — GitHub/GitLab/Bitbucket/Azure/Jenkins can all achieve similar results via a matrix or manually-sharded test command, but none ship this specific "split by historical per-file timing data, automatically" mechanism as a first-party, built-in feature the way CircleCI does.
---
## Caching and Workspaces
CircleCI distinguishes three related-but-distinct data-sharing mechanisms, worth being precise about since the terms are easy to conflate:
| Mechanism | Scope | Purpose |
|---|---|---|
| **`save_cache`/`restore_cache`** | Across different *builds* (over time) | Speed — dependency caching, matching the cache concept from every prior platform |
| **`persist_to_workspace`/`attach_workspace`** | Across different *jobs*, within one workflow run | Passing build output from one job to a dependent job (already shown in Step 3 earlier) |
| **`store_artifacts`** | Attached to a specific build, browsable afterward | Human-facing output — test reports, coverage HTML, build logs kept for inspection |
```yaml
jobs:
build:
docker: [{ image: cimg/node:20.11 }]
steps:
- checkout
- restore_cache:
keys:
- npm-deps-{{ checksum "package-lock.json" }}
- npm-deps- # fallback prefix match if an exact match isn't found
- run: npm ci
- save_cache:
key: npm-deps-{{ checksum "package-lock.json" }}
paths: [node_modules]
- run: npm run build
- persist_to_workspace:
root: .
paths: [dist]
- store_artifacts:
path: coverage/
destination: coverage-report
```
**The three-way split maps cleanly onto concepts already covered for other platforms, worth stating the correspondence explicitly:** `save_cache`/`restore_cache` is the same idea as GitHub's `actions/cache` or GitLab's `cache:`; `persist_to_workspace`/`attach_workspace` is the same idea as GitHub's `upload-artifact`/`download-artifact` or Azure's `PublishPipelineArtifact`/`DownloadPipelineArtifact`; `store_artifacts` has no precise single-word equivalent elsewhere in this series but is closest to what every platform's "build artifacts" browsing UI does with anything explicitly published for human inspection, as distinct from data one job passes to another programmatically.
---
## Orbs — CircleCI's Reusability Model
An **Orb** is CircleCI's reusable, versioned, shareable package — bundling jobs, commands, and executors into a single importable unit, closest in spirit to a GitHub Action combined with a GitLab CI/CD Component, but with a more centralized, curated registry (the CircleCI Orb Registry) than either.
```yaml
version: 2.1
orbs:
node: circleci/node@5.1.0 # official, CircleCI-maintained orb, pinned to an exact version
aws-cli: circleci/aws-cli@4.1.3
jobs:
build-and-test:
docker: [{ image: cimg/node:20.11 }]
steps:
- checkout
- node/install-packages: # a STEP provided by the node orb — no need to hand-write npm ci + caching
pkg-manager: npm
- run: npm test
workflows:
main:
jobs:
- build-and-test
```
**Orbs can also define entire reusable jobs, not just individual steps**, directly parameterizable — the closest CircleCI equivalent to a GitHub reusable workflow:
```yaml
orbs:
deploy-orb: my-org/deploy-orb@2.0.0
workflows:
main:
jobs:
- deploy-orb/deploy:
environment: production
context: aws-prod-creds
```
**Orb registration/publishing has a namespace and certification model worth knowing about, since it directly shapes trust:** orbs published under a **certified** or **partner** namespace (CircleCI-vetted, or an official vendor like AWS/Slack) carry a stronger trust signal than an arbitrary community-published orb — the same "check the maintenance/trust signal before adopting a third-party reusable unit" discipline already established for GitHub Actions (Part 5) and Jenkins plugins (Part 9), here backed by CircleCI's own registry-level certification tiers rather than left entirely to the adopting team's own due diligence.
An organization can also publish **private orbs**, visible only within its own account — the direct analogue of a GitHub reusable workflow hosted in a private shared repo, or a GitLab CI/CD Component published to a private instance of the Catalog, for centralizing an org's own golden-path logic without exposing it publicly.
---
## Contexts — Secrets Shared Across Projects
A **Context** is CircleCI's mechanism for sharing environment variables/secrets across *multiple projects*, with access restricted to specific named security groups — CircleCI's equivalent of GitHub's Organization secrets or GitLab's Group-level variables (Parts 5 and 6), but with a more explicit, security-group-based access model.
```yaml
workflows:
main:
jobs:
- deploy:
context: aws-prod-deploy # the job gets every secret defined in this context
```
```mermaid
graph TD
Context["Context: aws-prod-deploy
(AWS_ACCESS_KEY_ID, etc.)"] --> Group["Restricted to security group:
'platform-team' — configured
in Organization Settings"]
Group --> Job1["Project A's deploy job"]
Group --> Job2["Project B's deploy job"]
```
**Restricting a context to a specific security group, rather than leaving it available to every project in the organization by default, is the least-privilege-scoping practice here** — directly analogous to scoping a GitHub Environment secret or a GitLab Protected variable, except the access boundary in CircleCI is an explicit, named group of people/projects configured centrally, rather than a property attached to a branch or deployment target. A context with no group restriction is available to any job in any project within the organization that references it by name — a genuinely broad default worth actively tightening for anything sensitive.
---
## OIDC — Eliminating Long-Lived Cloud Credentials
CircleCI's OIDC implementation follows the same structural pattern established repeatedly across this series: a short-lived, signed identity token, exchanged with a cloud provider for temporary credentials.
```mermaid
sequenceDiagram
participant Job as CircleCI job
participant CircleCI as CircleCI OIDC provider
participant AWS as AWS STS
Job->>CircleCI: $CIRCLE_OIDC_TOKEN automatically
populated (requires a context)
Job->>AWS: AssumeRoleWithWebIdentity -
present the token
AWS->>AWS: Verify signature - check
IAM role trust policy
AWS-->>Job: Short-lived AWS credentials
```
```yaml
jobs:
deploy:
docker: [{ image: cimg/aws:2024.03 }]
steps:
- checkout
- run:
command: |
aws configure set web_identity_token_file /tmp/oidc_token
echo "$CIRCLE_OIDC_TOKEN" > /tmp/oidc_token
aws sts assume-role-with-web-identity \
--role-arn arn:aws:iam::123456789012:role/circleci-deploy \
--role-session-name circleci \
--web-identity-token file:///tmp/oidc_token
workflows:
main:
jobs:
- deploy:
context: aws-oidc-config # a context is REQUIRED for $CIRCLE_OIDC_TOKEN to populate at all
```
**The one CircleCI-specific gotcha worth memorizing precisely, since it's an easy first-time trap:** `$CIRCLE_OIDC_TOKEN` is only populated in a job that references **at least one context** — a job with no `context:` at all, even one that otherwise looks correctly configured for OIDC, silently has no token available. This is a different failure mode from GitHub's explicit `permissions: id-token: write` (Part 5) or GitLab's explicit `id_tokens:` block (Part 6) — CircleCI's requirement is implicit, tied to context usage rather than a dedicated, self-documenting declaration, making it worth double-checking explicitly the first time OIDC is wired up on this specific platform.
---
## A Full Worked OIDC Example: Deploying to AWS
**Step 1 — one-time AWS setup**, structurally identical to every prior platform's OIDC trust-policy pattern, trusting CircleCI's own OIDC issuer:
```json
{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.circleci.com/org/ORGANIZATION_ID" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.circleci.com/org/ORGANIZATION_ID:aud": "ORGANIZATION_ID"
},
"StringLike": {
"oidc.circleci.com/org/ORGANIZATION_ID:sub": "org/ORGANIZATION_ID/project/PROJECT_ID/*"
}
}
}
```
**Step 2 — the pipeline job**, using the official `circleci/aws-cli` orb to simplify the exchange:
```yaml
version: 2.1
orbs:
aws-cli: circleci/aws-cli@4.1.3
jobs:
deploy-production:
docker: [{ image: cimg/base:current }]
steps:
- checkout
- aws-cli/setup:
role_arn: arn:aws:iam::123456789012:role/circleci-deploy
region: us-east-1
- run: aws s3 sync ./dist s3://my-production-bucket
workflows:
main:
jobs:
- deploy-production:
context: aws-oidc-config
```
The trust policy's `sub` condition scopes the role to a specific CircleCI organization and project — the same defense-in-depth pattern as every prior platform's OIDC example in this series, restricting which project's pipeline can actually assume production-level cloud access, independent of whatever the CircleCI-side pipeline config itself allows.
---
## Self-Hosted Runners
CircleCI's self-hosted execution option — a machine or Kubernetes-hosted agent a team registers to run jobs against, instead of CircleCI's own hosted infrastructure — functionally parallel to every other platform's equivalent covered in this series.
```yaml
jobs:
build:
machine: true
resource_class: my-namespace/my-self-hosted-runner # references a registered self-hosted runner resource class
steps: [checkout, run: ./build-with-internal-access.sh]
```
**The economics covered in the pricing section — zero credit consumption on self-hosted runners — is the single most CircleCI-specific reason to reach for this option**, beyond the usual specialized-hardware/private-network-access motivations already covered for every prior platform's self-hosted runner section. The security tradeoffs are identical to what's already been established repeatedly in this series: real hardware/network access at the cost of real security responsibility, and the same caution around running untrusted, externally-triggered pipelines against self-hosted infrastructure.
Runners can be registered individually or as part of a resource class shared by a whole team, and (like every other platform's self-hosted option) commonly run on Kubernetes for the same autoscaling and per-job pod isolation benefits already covered for GitHub, GitLab, and Jenkins — a topic Part 14 returns to in full depth.
---
## Docker Layer Caching and Remote Docker
A CircleCI-specific performance feature worth knowing by name: **Docker Layer Caching (DLC)**, available on the `machine` executor, caches individual Docker image *layers* between builds — not just the final built image, but each intermediate step of the `docker build` process itself.
```yaml
jobs:
build-image:
machine:
image: ubuntu-2204:current
docker_layer_caching: true # reuse unchanged layers from the previous build
steps:
- checkout
- run: docker build -t myapp .
```
```mermaid
flowchart LR
Dockerfile["Dockerfile:
5 layers"] --> Check{"Has this EXACT
layer been built
before, unchanged?"}
Check -->|"Yes - layers 1-3
unchanged"| Reuse["Reuse cached layers
1-3 instantly"]
Check -->|"No - layer 4
changed"| Rebuild["Rebuild layer 4
onward only"]
```
**Why this matters beyond ordinary dependency caching:** a typical multi-stage Dockerfile might have unchanged base-image and dependency-installation layers on most builds (only application code, the last layer or two, actually changes) — without DLC, every build re-executes every layer from scratch regardless of what actually changed; with DLC, only the layers whose inputs genuinely changed since the last build are rebuilt, which can turn a multi-minute image build into a matter of seconds for a small code-only change. This is a paid-tier feature on CircleCI's pricing structure, reflecting how much wall-clock time (and therefore compute cost) it genuinely saves for any team building container images frequently.
---
## Pipeline Parameters and Dynamic Config
**Pipeline parameters** make a workflow configurable at trigger time — CircleCI's equivalent of GitHub's `workflow_dispatch: inputs:` (Part 4) and Azure's runtime parameters (Part 8), but usable both for manually-triggered runs and for values passed in via the API:
```yaml
version: 2.1
parameters:
environment:
type: string
default: "staging"
run-integration-tests:
type: boolean
default: false
workflows:
main:
jobs:
- deploy:
environment: << pipeline.parameters.environment >>
- integration-test:
requires: [deploy]
filters: { branches: { only: main } }
# only meaningfully runs when triggered with run-integration-tests: true
```
**Dynamic config** goes a step further, addressing a genuinely different problem than parameters alone: a **setup workflow** runs first, executes arbitrary logic (commonly a script that inspects which files actually changed), and then dynamically *generates and triggers* a second-stage config file — rather than the entire pipeline structure being fixed at commit time.
```yaml
# .circleci/config.yml — the SETUP config, minimal, runs first
version: 2.1
setup: true
orbs:
path-filtering: circleci/path-filtering@1.1.0
workflows:
setup-workflow:
jobs:
- path-filtering/filter:
base-revision: main
config-path: .circleci/continue-config.yml
mapping: |
frontend/.* run-frontend true
backend/.* run-backend true
```
```mermaid
flowchart LR
Push["Push to a monorepo"] --> Setup["Setup workflow runs FIRST
(minimal, fast)"]
Setup --> Detect["Detects WHICH paths
actually changed"]
Detect --> Continue["Triggers continue-config.yml
dynamically, with ONLY the
relevant jobs included"]
```
**This is a direct, first-party answer to the exact "don't run the frontend suite for a backend-only change" monorepo problem this series returns to in Part 12** — rather than encoding path-based conditionals into every job's `filters:` by hand (workable, but verbose at real monorepo scale), dynamic config lets the *set of jobs that even exist in the triggered pipeline* be computed programmatically, based on what genuinely changed. Worth flagging as a preview of a recurring theme: every CI/CD platform in this series has its own answer to "how do we avoid running everything on every change in a large monorepo," and CircleCI's dynamic config is one of the more structurally flexible ones, since it can compute the *entire* pipeline shape, not just gate individual jobs.
---
## Insights and Test Analytics
CircleCI ships **Insights**, a built-in analytics dashboard tracking pipeline health metrics over time — directly implementing several of the DORA metrics this series established tool-agnostically in Part 1, as an out-of-the-box product feature rather than something a team has to assemble from raw CI logs themselves.
```mermaid
graph TD
Insights["CircleCI Insights"] --> Duration["Workflow duration
trends over time"]
Insights --> Success["Success rate
(directly: Change
Failure Rate from
Part 1's DORA metrics)"]
Insights --> Flaky["Flaky test detection —
tests that pass/fail
INCONSISTENTLY on the
SAME code"]
Insights --> Credit["Credit usage per
job/workflow — cost
attribution"]
```
**Flaky test detection deserves particular attention as a genuinely high-value, not-always-obvious feature.** A flaky test — one that passes and fails inconsistently against the *same* underlying code, usually due to a timing race, shared test-state pollution, or an external dependency the test doesn't properly mock — is uniquely corrosive to a team's trust in CI: developers who've seen a test fail "for no reason" a few times start reflexively re-running failed builds instead of investigating, which quietly erodes the entire "if CI is red, something is actually broken" guarantee this series' Part 1 built the whole concept of CI around. CircleCI's Insights specifically identifies tests with inconsistent pass/fail history *on unchanged code* and surfaces them separately from genuine regressions, giving a team an actionable, ranked list of exactly which tests are undermining confidence in the pipeline, rather than an undifferentiated pile of "sometimes red" builds with no way to tell a flaky test from a real bug at a glance.
---
## A Full Realistic Multi-Stage Pipeline
The same build → test (parallelized) → security scan → deploy-staging → approval → deploy-production shape from every prior platform chapter, in CircleCI's job/workflow/orb model:
```yaml
version: 2.1
orbs:
node: circleci/node@5.1.0
aws-cli: circleci/aws-cli@4.1.3
jobs:
build:
docker: [{ image: cimg/node:20.11 }]
resource_class: medium
steps:
- checkout
- node/install-packages
- run: npm run build
- persist_to_workspace: { root: ., paths: [dist] }
test:
docker: [{ image: cimg/node:20.11 }]
parallelism: 4
steps:
- checkout
- node/install-packages
- run:
command: |
TESTFILES=$(circleci tests glob "test/**/*.spec.js" | circleci tests split --split-by=timings)
npx jest $TESTFILES
security-scan:
docker: [{ image: cimg/node:20.11 }]
steps:
- checkout
- run: npm audit --audit-level=high
deploy-staging:
docker: [{ image: cimg/base:current }]
steps:
- attach_workspace: { at: . }
- aws-cli/setup: { role_arn: "arn:aws:iam::123456789012:role/circleci-staging" }
- run: aws s3 sync ./dist s3://staging-bucket
deploy-production:
docker: [{ image: cimg/base:current }]
steps:
- attach_workspace: { at: . }
- aws-cli/setup: { role_arn: "arn:aws:iam::123456789012:role/circleci-prod" }
- run: aws s3 sync ./dist s3://production-bucket
workflows:
main:
jobs:
- build
- test:
requires: [build]
- security-scan:
requires: [build]
- deploy-staging:
context: aws-staging
requires: [test, security-scan]
- hold-for-approval:
type: approval
requires: [deploy-staging]
- deploy-production:
context: aws-prod
requires: [hold-for-approval]
```
```mermaid
flowchart LR
Build["build"] --> Test["test
(4-way parallel)"]
Build --> Scan["security-scan"]
Test --> Staging["deploy-staging"]
Scan --> Staging
Staging --> Gate{"hold-for-approval"}
Gate --> Prod["deploy-production"]
```
---
## CircleCI vs. the Rest — Where It Genuinely Wins
An honest comparison, extending this series' running platform table:
| Dimension | CircleCI's position |
|---|---|
| **Raw pipeline speed tuning** | Strongest in this series — per-job resource classes, timing-based test splitting, Docker Layer Caching all as first-party features |
| **Reusability** | Orbs — centrally registry-curated, with certification tiers, stronger trust signal than an open marketplace |
| **Built-in security scanning** | Thin — no first-party SAST/DAST equivalent to GitLab's; relies on orbs/third-party integration |
| **Git host coupling** | None — works with GitHub, GitLab, or Bitbucket repos |
| **Cost model transparency** | Credits scale with resource class choice — a genuine, deliberate cost/speed dial, not just a flat per-minute rate |
**The honest recommendation, stated plainly:** a team whose actual pain point is pipeline *speed* at scale — a large test suite, frequent container builds, a need for fine-grained per-job hardware tuning — gets more concrete, immediate value from CircleCI's specific feature set than from any other platform covered in this series on that specific axis. A team whose priority is deep built-in security scanning, tight Jira integration, or avoiding any SaaS billing entirely should weigh CircleCI against GitLab, Bitbucket, or Jenkins respectively, per the comparisons already built up across this series — CircleCI's strength is genuinely narrow and genuinely deep, not a broad "best at everything" claim.
---
## Common Mistakes
| Mistake | Why it's a problem | Fix |
|---|---|---|
| Forgetting `version: 2.1` (or leaving it at 2.0) | Orbs, reusable executors, and parameterized jobs are unavailable | Always use `2.1` for any new config |
| No `context:` on a job that needs `$CIRCLE_OIDC_TOKEN` | The token silently never populates — no explicit error pointing at the real cause | Always attach a context, even a near-empty one, to any job using OIDC |
| An unrestricted context available to every project in the org | Broad default access to sensitive secrets, beyond what any single project actually needs | Restrict every context to a specific, named security group |
| Using a naive equal-count test split instead of `--split-by=timings` | Some parallel containers finish early and sit idle while others carry disproportionately slow files | Use `circleci tests split --split-by=timings` for genuinely balanced parallel test runs |
| Choosing an oversized `resource_class` for every job by default "to be safe" | Consumes credits at a proportionally higher rate for jobs that don't actually need the extra power | Size `resource_class` to each job's actual bottleneck — profile before over-provisioning |
| Treating an uncertified, low-activity community orb the same as an official one | Same trust/maintenance risk as an unvetted GitHub Action or Jenkins plugin | Check an orb's certification tier and maintenance signal before adopting it, especially for anything security-sensitive |
| Rebuilding a Docker image from scratch on every build with no Docker Layer Caching enabled | Wastes real build time and cost re-executing unchanged layers | Enable `docker_layer_caching: true` on the `machine` executor for any frequently-rebuilt image |
| Running every job in a large monorepo on every push, with no path filtering | Wastes significant CI time/credits testing services nothing actually changed | Use dynamic config (setup workflows + `path-filtering`) to include only genuinely affected jobs |
| Treating a stable aggregate CI success rate as proof there's no flakiness problem | A small number of chronically flaky tests can hide behind a healthy-looking aggregate number while eroding developer trust | Use Insights' flaky-test detection to surface specific problem tests, not just the overall pass rate |
---
## Worked Practice Problems
**Problem 1:** A team's test suite takes 18 minutes on a single CircleCI job. They add `parallelism: 6` but see almost no improvement — one container still takes nearly the full 18 minutes while the other five finish in under 5. What's the most likely cause, and the fix?
*Answer:* The team is almost certainly splitting test files with a naive method (e.g. an even file-count split, or CircleCI's default filename-based split with no historical timing data yet) rather than `circleci tests split --split-by=timings` — if a handful of unusually slow test files all land in the same container by chance, that container becomes the bottleneck regardless of how many parallel containers exist, since the workflow can't finish faster than its slowest single container. The fix is explicitly using `circleci tests split --split-by=timings`, which balances containers by each test file's actual historical runtime rather than raw file count — after a few builds accumulate timing data, the slow files get distributed roughly evenly across containers instead of clustering in one.
**Problem 2:** A security audit finds a CircleCI Context named `all-cloud-creds` containing both staging and production AWS credentials, with no security-group restriction, referenced by workflows across 15 different projects in the organization. What's the exploitable risk, and the fix?
*Answer:* Every one of those 15 projects' pipelines — including ones with no legitimate business need for production credentials — has access to both staging and production AWS secrets, meaning a compromised dependency or a malicious PR in *any* one of those 15 projects (the same supply-chain risk pattern established repeatedly across this series) could exfiltrate production credentials, not just that project's own. The fix: split into separate `staging-creds` and `production-creds` contexts, each restricted to a specific named security group containing only the people/projects that genuinely need that specific level of access — directly mirroring the least-privilege secrets-scoping principle already established for GitHub Environment secrets (Part 5) and GitLab Protected variables (Part 6).
**Problem 3:** An engineering team is deciding between CircleCI and GitHub Actions for a new project already hosted on GitHub, where the primary stated pain point from their current Jenkins setup is "our test suite takes 25 minutes and blocks every PR." Which platform's specific feature set most directly addresses this stated pain point, and why?
*Answer:* CircleCI's timing-based automatic test splitting most directly addresses this specific complaint — GitHub Actions can achieve similar wall-clock improvement via a manually configured `strategy.matrix` sharding the test suite across parallel jobs, but that requires the team to manually determine and maintain a reasonable shard count and split strategy themselves, with no built-in mechanism for balancing shards by actual historical per-test timing data the way CircleCI's `--split-by=timings` does automatically. That said, the project is already on GitHub, and CircleCI's git-host-agnostic design means adopting it doesn't require moving off GitHub — the two platforms aren't mutually exclusive with the existing repo host, and a fair recommendation would weigh CircleCI's speed-tuning advantage specifically for this stated problem against the operational simplicity of staying within GitHub's own native, already-adopted platform (Parts 4-5) and manually sharding via `strategy.matrix` as a lower-effort first attempt before concluding a platform switch is actually warranted.
**Problem 4:** A monorepo hosts three independent services. A backend-only change currently triggers all three services' full test suites, wasting significant CI time and credits. Which CircleCI feature addresses this specifically, and how does it differ from just adding `filters:` to each job?
*Answer:* Dynamic config, via a setup workflow using the `path-filtering` orb (or equivalent custom logic) to detect which paths actually changed and generate a second-stage config containing only the relevant jobs. This differs meaningfully from per-job `filters:` (which can gate *whether* a fixed set of already-defined jobs runs, based on branch name or similar static conditions) because dynamic config can change the actual *set* of jobs that exist in the pipeline at all, computed from real changed-file data at trigger time — `filters:` alone can't express "only include the backend test job's definition in this specific run because backend/ changed," it can only turn already-defined jobs on or off based on simpler, static criteria.
**Problem 5:** A team notices their CI success rate looks stable at 92% month over month, but developers increasingly complain about "flaky, unreliable CI." Reconcile these two observations, and explain what CircleCI feature would clarify what's actually happening.
*Answer:* An aggregate 92% success rate is fully consistent with a small number of specific, chronically flaky tests causing most of the failures — if the same 3-4 tests intermittently fail across many otherwise-unrelated builds, the overall pass-rate percentage can look stable even as developer trust in CI erodes, because from a developer's perspective, "I keep having to re-run CI for no reason" is a qualitatively different, more corrosive experience than the aggregate statistic conveys. CircleCI's Insights flaky-test detection would directly clarify this — it identifies which specific tests have an inconsistent pass/fail history against unchanged code, surfacing the actual small set of problem tests separately from genuine regressions, turning a vague "CI feels unreliable" complaint into a concrete, actionable list of exactly what to fix first.
---
## Summary and What's Next
CircleCI occupies the cloud-native, performance-tuning-focused end of this series' platform spectrum — fully vendor-hosted execution (or opt-in, zero-credit self-hosted runners), git-host-agnostic like Jenkins and Azure Pipelines, with `jobs:` and `workflows:` deliberately separated so the same job definitions can be reused across multiple orchestration graphs. Its most genuinely distinctive capabilities among every platform covered in this series are timing-based automatic test splitting and Docker Layer Caching, both first-party features addressing pipeline *speed* specifically, backed by a credits-based billing model where resource-class selection is an explicit, deliberate cost/speed dial. Orbs provide GitHub-Action-like reusability with a more centrally curated, certification-tiered registry; Contexts provide GitHub-Environment-like secret scoping via named security groups; and OIDC follows the same credential-free cloud-authentication pattern established throughout this series, with one CircleCI-specific gotcha (the token requires an attached context to populate at all).
**Part 11** moves to Tekton — a genuinely different category of tool from everything covered so far in this series: not a hosted CI/CD *platform* at all, but a set of Kubernetes-native building-block CRDs that a platform team assembles into its own custom CI/CD system, running entirely inside a Kubernetes cluster.
The contrast is worth anticipating: everything from GitHub Actions through CircleCI in this series ships as a complete, opinionated product with its own UI, its own YAML schema, and its own hosted (or self-hosted) execution model. Tekton ships none of that by default — it's closer to a set of primitives a team builds *with* than a platform a team simply adopts, and Part 11 covers exactly what that tradeoff buys and costs.