# Linux & Networking Fundamentals — Part 6: systemd Deep Dive

> **Series:** Linux & Networking Fundamentals (6 of 7)
> **Part 1:** `01-linux-process-and-memory-internals.md` — Process & Memory Internals
> **Part 2:** `02-tcp-ip-and-dns.md` — TCP/IP & DNS
> **Part 3:** `03-linux-troubleshooting-toolkit.md` — The Linux Troubleshooting Toolkit
> **Part 4:** `04-netfilter-iptables-nftables.md` — Netfilter, iptables & nftables
> **Part 5:** `05-namespaces-and-virtual-networking.md` — Network Namespaces & Virtual Networking
> **Part 6:** This file — systemd Deep Dive
> **Part 7:** `07-ebpf-observability-and-networking.md` — eBPF for Observability & Networking
> **Questions:** `questions.md`

## Table of Contents

1. [Why This Part Exists](#why-this-part-exists)
2. [What systemd Actually Is — PID 1 and Beyond](#what-systemd-actually-is-pid-1-and-beyond)
3. [Unit Types — More Than Just Services](#unit-types-more-than-just-services)
4. [Anatomy of a `.service` Unit File](#anatomy-of-a-service-unit-file)
5. [A Minimal Service Unit, Built Up Step by Step](#a-minimal-service-unit-built-up-step-by-step)
6. [Dependency Ordering — `Wants`, `Requires`, `After`, `Before`](#dependency-ordering-wants-requires-after-before)
7. [Why `Wants`+`After` Is the Default Recommended Pattern](#why-wantsafter-is-the-default-recommended-pattern)
8. [Targets — systemd's Replacement for Runlevels](#targets-systemds-replacement-for-runlevels)
9. [The Boot Sequence, Concretely](#the-boot-sequence-concretely)
10. [Socket Activation — Starting a Service On Demand](#socket-activation-starting-a-service-on-demand)
11. [A Worked Example: Socket-Activated Service](#a-worked-example-socket-activated-service)
12. [Restart Policies and Service Supervision](#restart-policies-and-service-supervision)
13. [journald — Structured, Centralized Logging](#journald-structured-centralized-logging)
14. [Querying journald With `journalctl`](#querying-journald-with-journalctl)
15. [journald's Own Storage and Retention](#journalds-own-storage-and-retention)
16. [cgroup v2 Integration — systemd as the Cgroup Manager](#cgroup-v2-integration-systemd-as-the-cgroup-manager)
17. [Resource Control in a Unit File](#resource-control-in-a-unit-file)
18. [Cgroup Delegation — Why Container Runtimes Need It](#cgroup-delegation-why-container-runtimes-need-it)
19. [systemd-networkd — Declarative, Persistent Network Configuration](#systemd-networkd-declarative-persistent-network-configuration)
20. [A Worked Example: systemd-networkd Configuring an Interface](#a-worked-example-systemd-networkd-configuring-an-interface)
21. [Timers — systemd's Replacement for Cron](#timers-systemds-replacement-for-cron)
22. [Sandboxing a Service — Hardening Directives](#sandboxing-a-service-hardening-directives)
23. [Drop-In Overrides — Modifying a Unit Without Editing It](#drop-in-overrides-modifying-a-unit-without-editing-it)
24. [systemd in Containers — a Genuine Nuance](#systemd-in-containers-a-genuine-nuance)
25. [Debugging a Failing Service — a Practical Workflow](#debugging-a-failing-service-a-practical-workflow)
26. [systemd-resolved — DNS Resolution Revisited](#systemd-resolved-dns-resolution-revisited)
27. [systemd-logind and Session Management](#systemd-logind-and-session-management)
28. [A Full Realistic Example: a Production-Hardened, Socket-Activated Service](#a-full-realistic-example-a-production-hardened-socket-activated-service)
29. [Watchdogs — Detecting a Hung, Not Just a Crashed, Process](#watchdogs-detecting-a-hung-not-just-a-crashed-process)
30. [A systemd Change Checklist for Production Hosts](#a-systemd-change-checklist-for-production-hosts)
31. [Analyzing Boot Performance](#analyzing-boot-performance)
32. [Common Mistakes](#common-mistakes)
33. [Worked Practice Problems](#worked-practice-problems)
34. [Key Terms Glossary — This Chapter's Vocabulary in One Place](#key-terms-glossary-this-chapters-vocabulary-in-one-place)
35. [Summary and What's Next](#summary-and-whats-next)

---

## Why This Part Exists

Parts 4 and 5 covered, in hands-on depth, exactly how packets get filtered, NAT'd, routed, and how virtual network topology gets built. Neither chapter addressed a genuinely foundational question sitting underneath all of it: **what actually starts the processes, brings up the network interfaces, and keeps everything running on a real, booted Linux host in the first place?** This chapter answers that question directly — systemd is the init system and service manager responsible for exactly this, on the overwhelming majority of production Linux distributions in 2026.

```mermaid
graph TD
    Boot["Machine powers on"] --> Kernel["Kernel initializes hardware,\nmounts the root filesystem"]
    Kernel --> PID1["Kernel starts PID 1 —\non nearly every modern\ndistro, this is systemd"]
    PID1 --> Everything["systemd starts EVERYTHING\nelse: services, network\ninterfaces (Part 5's own\nsubject), mount points,\nlogging (journald)"]
```

Worth stating directly why this belongs in a networking-focused series specifically: `systemd-networkd` is one of the standard mechanisms for declaring persistent network interface configuration — the exact detail Part 5 flagged as its own closing forward-reference, since every `ip link`/`ip addr` command demonstrated there configures state that vanishes on reboot without something like this chapter's own material making it persistent.

---

## What systemd Actually Is — PID 1 and Beyond

systemd is not just "the init system" in the narrow, historical sense (the very first userspace process the kernel starts) — it's a much broader suite of integrated system-management components, worth naming precisely rather than treating as one monolithic, undifferentiated thing.

```mermaid
graph TD
    Systemd["systemd (the suite)"] --> InitCore["PID 1 core: unit\nmanagement, dependency\nresolution, service\nsupervision"]
    Systemd --> Journald["systemd-journald:\nstructured logging"]
    Systemd --> Networkd["systemd-networkd:\nnetwork configuration"]
    Systemd --> Resolved["systemd-resolved:\nDNS resolution (Part 2's\nown subject, revisited)"]
    Systemd --> Logind["systemd-logind: user\nsession/login management"]
    Systemd --> Timers["systemd timers:\ncron replacement"]
```

**Worth being precise about the "PID 1" property specifically, since it carries real operational consequences: PID 1 is a genuinely special process from the kernel's own point of view — it's the parent (directly or via reparenting) of every other process on the system, and if it crashes, the kernel panics, since there's no other process left to manage the entire system.** This is exactly why systemd's own reliability and correctness matters disproportionately compared to an ordinary service — a bug in PID 1 itself, unlike a bug in any individual managed service, has no graceful failure mode; the whole system goes down with it.

---

## Unit Types — More Than Just Services

Everything systemd manages is expressed as a **unit** — worth surveying the full range of unit types, since "systemd manages services" undersells its actual scope considerably.

| Unit type | Manages |
|---|---|
| `.service` | A long-running process or daemon — this chapter's primary focus |
| `.socket` | A network or IPC socket, enabling socket activation (covered later in this chapter) |
| `.target` | A synchronization point/grouping of other units — systemd's replacement for old-style runlevels |
| `.timer` | A scheduled trigger — systemd's replacement for cron |
| `.mount` / `.automount` | A filesystem mount point |
| `.device` | A kernel-recognized device, exposed to the dependency graph |
| `.swap` | A swap device or file, managed with the same dependency semantics as a `.mount` unit |
| `.path` | A trigger based on filesystem path changes (a file appearing, being modified) |
| `.slice` | A cgroup grouping node (e.g. `system.slice`, `user.slice`) — a container for other units' cgroups |
| `.scope` | An externally-created process group systemd tracks and manages the cgroup for, without owning its lifecycle |

**The breadth of this table is worth internalizing directly: nearly every "system-level thing that needs to happen in a coordinated, dependency-aware order" — starting a service, mounting a filesystem, waiting for a device, running a scheduled job — is unified under the exact same dependency-graph, unit-based model, rather than each having its own separate, bespoke mechanism the way older init systems typically required.** This chapter focuses primarily on `.service`, `.socket`, `.target`, and `.timer` units, since they cover the large majority of what a platform engineer actually authors directly.

---

## Anatomy of a `.service` Unit File

```ini
[Unit]
Description=A simple example web application
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my-app --port 8080
Restart=on-failure
RestartSec=5
User=myapp
Group=myapp

[Install]
WantedBy=multi-user.target
```

**Every unit file's three-section structure is worth memorizing directly, since it's genuinely universal across nearly every unit type this chapter covers: `[Unit]` holds metadata and dependency directives common to any unit type; the type-specific section (`[Service]`, here) holds directives specific to that unit type; `[Install]` defines what happens when the unit is `enable`d — which target(s) should pull this unit in automatically at boot.** `Type=simple` (the default) tells systemd the process specified by `ExecStart` IS the main service process itself, immediately — other `Type=` values (covered implicitly through this chapter's own worked examples) change this assumption for processes with more complex startup behavior.

---

## A Minimal Service Unit, Built Up Step by Step

```bash
# 1. Write the unit file
sudo tee /etc/systemd/system/my-app.service <<'EOF'
[Unit]
Description=My Application

[Service]
ExecStart=/usr/local/bin/my-app
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd's own unit cache — REQUIRED after any unit file change
sudo systemctl daemon-reload

# 3. Start it immediately
sudo systemctl start my-app

# 4. Enable it to start automatically on future boots
sudo systemctl enable my-app

# 5. Check status
sudo systemctl status my-app
```

**Step 2 deserves the strongest emphasis of this sequence, since forgetting it is the single most common source of "I edited the unit file but nothing changed" confusion: systemd caches unit file contents in memory, and `daemon-reload` is what tells it to re-read the actual files on disk** — editing a unit file directly has zero effect on a running or subsequently-started service until this command runs. This is worth treating as a standing habit after any unit file edit, not an occasionally-remembered extra step.

---

## Dependency Ordering — `Wants`, `Requires`, `After`, `Before`

Worth a precise, careful distinction between two genuinely separate concepts this chapter's own directive names can easily blur together: **ordering** (which unit starts before/after another) and **dependency strength** (whether a failure in one unit should affect the other at all).

```mermaid
graph TD
    Ordering["ORDERING directives:\nAfter=, Before= —\ncontrol WHEN, relative\nto another unit"] --> NoStrength["Say NOTHING about whether\nthe unit should even start\nif the other one fails"]
    Strength["STRENGTH directives:\nWants=, Requires= —\ncontrol WHETHER a unit\nshould start at all"] --> NoOrdering["Say NOTHING about WHEN,\nrelative to the other unit"]
```

**This is the single most important, most commonly conflated distinction in this entire chapter's dependency material, worth stating as plainly as possible: `After=foo.service` alone does NOT mean "start after foo, and only if foo succeeds" — it only affects timing, and a unit with only an `After=` directive (no `Wants=`/`Requires=`) will still attempt to start even if `foo.service` never starts at all.** `Wants=foo.service` (a soft dependency — a failure of `foo` doesn't prevent this unit from starting) and `Requires=foo.service` (a hard dependency — this unit fails to start if `foo` fails) each need to be paired with the corresponding `After=` directive to actually control both "whether" and "when" together — specifying only one or the other is a genuinely common source of subtle boot-ordering bugs.

---

## Why `Wants`+`After` Is the Default Recommended Pattern

```ini
[Unit]
After=network-online.target
Wants=network-online.target
```

**`Wants=`+`After=`, used together, is the systemd-recommended default pairing for the large majority of real service dependencies — worth understanding precisely why `Requires=` is NOT the default recommendation, since this is a genuinely counter-intuitive point on first encounter.** `Requires=` creates a hard failure dependency — if the required unit fails to start for ANY reason, the dependent unit fails too, cascading failures across the boot sequence in a way that's often more brittle than actually desired. `Wants=` expresses "start this too, and prefer it to have started first (via the paired `After=`)," without turning a transient failure in one unit into a hard failure of every unit that merely wanted it — the correct default posture for most real-world service relationships, reserving `Requires=` specifically for cases where a genuine, non-negotiable hard dependency exists (a service that is architecturally meaningless without its database connection, for instance).

---

## Targets — systemd's Replacement for Runlevels

A **target** is a synchronization point — a named grouping other units can depend on, replacing the old, numbered runlevel concept (runlevel 3, runlevel 5) with named, more expressive groupings.

```mermaid
graph TD
    MultiUser["multi-user.target\n(a full, network-enabled,\nnon-graphical system —\nthe typical SERVER default)"] --> Graphical["graphical.target\n(multi-user.target PLUS\na display manager/GUI —\nthe typical DESKTOP default)"]
    NetworkOnline["network-online.target\n(network interfaces are\nUP and genuinely usable —\nNOT just 'network service\nhas started')"] --> MultiUser
```

**The `network-online.target` distinction deserves specific, careful emphasis, since it resolves a genuinely common source of boot-order bugs: `network.target` merely means the network management SERVICE has started, not that any interface actually has a usable IP address or connectivity yet — a service that genuinely needs real network connectivity at startup (this chapter's own `my-app.service` example, reaching an external database) should depend on `network-online.target` specifically, not the more commonly (and often incorrectly) referenced plain `network.target`.** This distinction is a real, frequently-encountered production gotcha — a service depending only on `network.target` can start successfully before the network interface it actually needs has finished acquiring a DHCP lease, failing intermittently in a way that looks like a flaky application bug rather than the actual, systemic ordering mistake it is.

---

## The Boot Sequence, Concretely

```mermaid
sequenceDiagram
    participant Kernel
    participant Systemd as systemd (PID 1)
    participant Default as default.target
    participant Services as Individual services

    Kernel->>Systemd: Start PID 1
    Systemd->>Systemd: Read unit files, build dependency graph
    Systemd->>Default: Begin activating default.target\n(usually symlinked to multi-user.target)
    Default->>Services: Activate every unit default.target\ntransitively Wants/Requires
    Services->>Services: Each unit starts, respecting\nAfter/Before ordering
    Services-->>Systemd: Report started/failed status
```

**`systemctl get-default` reveals which target a given host actually boots into — worth checking directly rather than assuming, since a misconfigured default target (booting into `rescue.target` unexpectedly, for instance) is a genuinely real, if uncommon, production incident category.** Every unit ultimately activated during a normal boot traces back to being a transitive `Wants=`/`Requires=` dependency of this one default target — the entire boot sequence is, concretely, systemd resolving one large dependency graph rooted at that single target.

---

## Socket Activation — Starting a Service On Demand

**Socket activation** is a genuinely distinctive systemd capability worth understanding in depth: rather than starting a service eagerly at boot, systemd can instead create and listen on a socket immediately, and only actually start the corresponding service process the first time a connection actually arrives.

```mermaid
graph TD
    Boot["At boot: systemd creates\nand listens on the SOCKET\nitself — the actual service\nprocess is NOT started yet"] --> FirstConn["First connection arrives"]
    FirstConn --> StartService["systemd starts the actual\nservice process NOW,\nhanding it the already-\nlistening socket"]
    StartService --> Subsequent["Subsequent connections go\ndirectly to the now-running\nservice process"]
```

**The concrete, practical benefit worth stating precisely: a rarely-used service consumes zero memory/CPU until its first actual connection, while still being immediately available (no cold-start delay perceptible to the FIRST connecting client, since the socket itself is already listening even before the process starts) — a genuine resource-efficiency win for services that are needed occasionally but not constantly.** This is worth connecting directly to this course's own CI/CD & GitOps series' self-hosted runner scaling chapter and its own "scale-to-zero" framing — socket activation is, conceptually, the exact same scale-to-zero idea applied at the level of one systemd-managed service on one host, rather than an entire fleet of Kubernetes runner Pods.

---

## A Worked Example: Socket-Activated Service

```ini
# /etc/systemd/system/my-app.socket
[Unit]
Description=Socket for my-app

[Socket]
ListenStream=8080

[Install]
WantedBy=sockets.target
```

```ini
# /etc/systemd/system/my-app.service
[Unit]
Description=My Application (socket-activated)
Requires=my-app.socket

[Service]
ExecStart=/usr/local/bin/my-app
# No explicit port binding needed in the app itself —
# systemd passes the already-open socket via file descriptor
```

```bash
sudo systemctl enable --now my-app.socket
# my-app.service itself is NOT started yet — check:
systemctl status my-app.service   # inactive (dead)

# First connection triggers activation:
curl http://localhost:8080
systemctl status my-app.service   # now active (running)
```

**Worth noticing directly what's absent from `my-app.service`'s own `[Service]` section: no explicit port-binding configuration at all, because the application itself doesn't open the listening socket — systemd already did, and hands the already-bound file descriptor to the service process on startup.** This requires the application itself to support systemd's socket-passing convention (reading the pre-opened file descriptor rather than calling its own `bind()`/`listen()`), a real integration requirement worth checking before assuming socket activation is a drop-in capability for any arbitrary existing service.

---

## Restart Policies and Service Supervision

```ini
[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3
```

| `Restart=` value | Behavior |
|---|---|
| `no` (default) | Never automatically restart |
| `on-failure` | Restart only on a non-zero exit code, or a signal/timeout/watchdog failure |
| `always` | Restart regardless of exit code, even after a clean, intentional exit |
| `on-abnormal` | Restart on a signal, timeout, or watchdog failure — NOT on a plain non-zero exit |

**`StartLimitIntervalSec`+`StartLimitBurst` deserve specific emphasis as a genuinely important safety valve worth including alongside any `Restart=` policy other than `no`: without them, a service crash-looping due to a persistent underlying bug restarts indefinitely, potentially generating an unbounded flood of restart attempts, log noise, and resource churn.** With the values shown above, systemd gives up after 3 restart attempts within any 60-second window, transitioning the unit to a `failed` state instead of restarting forever — a real, concrete circuit-breaker pattern directly analogous to the retry-budget concepts this course's own Reliability & Architecture series covers for application-level retries, now applied at the process-supervision layer.

---

## journald — Structured, Centralized Logging

`systemd-journald` is systemd's own logging daemon — worth understanding as a genuine architectural departure from older, plain-text syslog: journald stores logs in a structured, indexed **binary** format, capturing every log entry with rich, queryable metadata automatically.

```mermaid
graph TD
    Sources["Every systemd-managed\nservice's stdout/stderr,\nPLUS the kernel's own\nlog ring buffer"] --> Journald["systemd-journald —\nstructured, indexed BINARY\nstorage (NOT plain text)"]
    Journald --> Metadata["Automatically captured per\nentry: PID, UID, unit name,\ntimestamp, boot ID, and\nmore — with ZERO extra\napplication instrumentation"]
```

**The "zero extra application instrumentation" property deserves the strongest emphasis, since it's a genuine, practical operational win worth connecting directly to this course's own Observability series: any application that simply writes to stdout/stderr — no logging library integration, no structured-logging discipline required from the application itself — automatically gets rich, per-entry metadata (which unit, which PID, which boot) captured by journald, purely because systemd started and supervises that process.** This is a meaningfully lower floor than the application-level structured-logging discipline this course's Observability series otherwise recommends — journald's own metadata capture happens regardless of whether the application itself does anything special at all.

---

## Querying journald With `journalctl`

```bash
# Logs for a specific unit
journalctl -u my-app.service

# Follow logs in real time (like tail -f)
journalctl -u my-app.service -f

# Logs since a specific time
journalctl -u my-app.service --since "1 hour ago"

# Only this boot's logs
journalctl -b

# Kernel-only messages (dmesg equivalent — Part 4's own debugging workflow used this)
journalctl -k

# JSON output, for piping into another tool
journalctl -u my-app.service -o json
```

**The `-o json` output format deserves specific mention as a genuinely important integration point with this course's own Observability series' log-shipping material: journald's structured, machine-parseable JSON export is exactly what a log-forwarding agent (Fluentd, Vector, or an equivalent) consumes to ship these logs into a centralized aggregation pipeline (Loki, Elasticsearch) without needing to parse fragile, free-text log lines — the structured metadata journald captures automatically becomes structured fields in the shipped log entry, directly, with no additional parsing logic required on the shipping side.**

---

## journald's Own Storage and Retention

```ini
# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=30day
```

**`Storage=persistent` deserves specific mention as a setting worth checking explicitly on any production host, since journald's own default behavior varies by distribution: without persistent storage configured, journal logs may live only in a `tmpfs`-backed, volatile location, meaning every log entry is lost on a reboot** — a real, easy-to-miss gap for any host relying purely on local journald storage as its log retention strategy without also shipping logs to a centralized, durable pipeline. `SystemMaxUse` and `MaxRetentionSec` are the practical disk-space and time-based retention controls, worth setting explicitly on any host rather than trusting the distribution's own default, which can consume more local disk than a specific host's own storage budget allows for.

---

## cgroup v2 Integration — systemd as the Cgroup Manager

Worth a direct, important architectural fact: on essentially every modern Linux distribution, **systemd is the exclusive, authoritative manager of the cgroup v2 hierarchy** — it creates a sub-tree for every unit it manages, and enforces a real kernel-level rule that only one userspace agent may manage a given cgroup subtree at a time.

```mermaid
graph TD
    CgroupRoot["cgroup v2 root hierarchy"] --> SystemdSlice["system.slice\n(systemd's OWN top-level\ncgroup for system services)"]
    SystemdSlice --> ServiceCgroup["my-app.service's OWN\ncgroup sub-tree — CPU/\nmemory limits enforced HERE"]
    CgroupRoot --> UserSlice["user.slice\n(per-user session cgroups)"]
```

**This "exclusive ownership" property is worth stating precisely, since it directly explains a real, sometimes-confusing operational detail: a tool attempting to manage cgroups directly, bypassing systemd, on a system where systemd already owns the hierarchy will either fail outright or produce inconsistent, conflicting state** — the kernel's own single-writer rule for a given cgroup subtree means systemd's ownership isn't a mere convention, it's an enforced constraint. Every `[Service]`-section resource-control directive covered in this chapter's next section is, concretely, systemd translating that directive into real cgroup v2 controller files on the unit's own dedicated subtree.

---

## Resource Control in a Unit File

```ini
[Service]
ExecStart=/usr/local/bin/my-app
CPUQuota=50%
MemoryMax=512M
MemoryHigh=400M
TasksMax=100
```

| Directive | Effect |
|---|---|
| `CPUQuota=` | Hard CPU usage ceiling, as a percentage of one core |
| `MemoryMax=` | Hard memory ceiling — the kernel's OOM killer targets this cgroup if exceeded |
| `MemoryHigh=` | A SOFTER memory ceiling — throttles the cgroup's memory allocation rate before the hard `MemoryMax` is reached |
| `TasksMax=` | Maximum number of tasks (threads/processes) the unit's cgroup may spawn |
| `IOWeight=` | Relative I/O bandwidth priority for this unit's cgroup, on a 1-10000 scale |

**`MemoryHigh=` deserves specific emphasis as a genuinely more graceful mechanism than `MemoryMax=` alone, worth connecting directly to this course's own Kubernetes Deep Dive series' own requests-vs-limits framing: `MemoryHigh` is conceptually closer to a soft throttle (the kernel slows the cgroup's own memory allocation, applying backpressure) while `MemoryMax` is a hard kill boundary (the OOM killer is invoked once crossed) — the same two-tier "soft pressure, then hard limit" pattern Kubernetes itself expresses via requests and limits, here expressed directly through systemd's own resource-control directives on a bare host with no Kubernetes involved at all.**

This is worth internalizing as more than a passing analogy: a Kubernetes node's own `kubelet` is itself just another systemd-managed service, and every Pod's cgroup ultimately nests underneath that node's own systemd-managed cgroup hierarchy — the resource-control mechanics covered in this section aren't a separate system from what a Kubernetes cluster does, they're the literal foundation every Kubernetes node builds its own Pod resource enforcement on top of.

---

## Cgroup Delegation — Why Container Runtimes Need It

Worth a direct, important connection to Part 5's own container-networking material: a container runtime (Docker, containerd) needs to create and manage its OWN cgroups for each container it starts — but systemd's exclusive-ownership rule (covered in this chapter's own cgroup section) would normally forbid exactly that.

```mermaid
graph TD
    SystemdOwns["systemd owns the cgroup\nhierarchy EXCLUSIVELY\n(this chapter's own rule)"] --> Conflict["A container runtime ALSO\nneeds to manage cgroups\nfor each container it starts"]
    Conflict --> Delegation["CGROUP DELEGATION: systemd\nexplicitly hands a specific\nsub-tree to a delegate\nunit (e.g. the container\nruntime's own systemd\nservice), which may then\nmanage everything BENEATH\nthat point itself"]
```

**Cgroup delegation is worth understanding as the explicit, sanctioned mechanism resolving this exact conflict — a unit marked `Delegate=yes` in its own `[Service]` section is given genuine, systemd-authorized management authority over its own cgroup sub-tree, letting a container runtime running as a systemd-managed service (`docker.service` or `containerd.service`, typically already configured this way by the distribution's own packaging) create and manage its own per-container cgroups beneath that delegated point without violating systemd's own single-writer rule.** This is worth recognizing directly as the real, concrete mechanism underneath every container's own resource limits (`docker run --memory=512m`, or a Kubernetes Pod's own resource limits) — ultimately expressed as cgroup v2 controller files, within a subtree systemd explicitly delegated to the container runtime for exactly this purpose.

---

## systemd-networkd — Declarative, Persistent Network Configuration

Worth the direct payoff of this chapter's own opening forward-reference from Part 5: `systemd-networkd` declares network interface configuration in version-controllable, plain-text `.network`/`.link`/`.netdev` files, applied automatically and persistently at boot — the exact mechanism Part 5's own `ip link`/`ip addr` commands lacked.

```mermaid
graph TD
    NetworkFiles["/etc/systemd/network/*.network\n(.link, .netdev files)"] --> NetworkdDaemon["systemd-networkd daemon"]
    NetworkdDaemon -->|"applies automatically\nEVERY boot"| RealInterfaces["Real network interfaces —\nusing the SAME underlying\nkernel mechanisms Part 5\ncovered directly (ip link/\nip addr, under the hood)"]
```

**This is worth stating precisely as the resolution to the exact gap Part 5 flagged explicitly: every `ip link`/`ip addr` command Part 5 demonstrated configures state that vanishes on reboot, while a `.network` file declares the identical configuration persistently — systemd-networkd re-applies it automatically on every boot, using the same underlying kernel mechanisms Part 5 covered by hand, just declared once and enforced continuously rather than requiring manual re-application.** This is a real, concrete example of the "declarative configuration, continuously reconciled" pattern this entire course returns to repeatedly (Kubernetes' own reconciliation loops, GitOps's own continuous reconciliation) — now demonstrated at the level of a single host's own network interface configuration.

---

## A Worked Example: systemd-networkd Configuring an Interface

```ini
# /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes

[DHCP]
UseDNS=yes
UseRoutes=yes
```

```ini
# /etc/systemd/network/20-vlan100.netdev — a VLAN interface, declaratively
[NetDev]
Name=vlan100
Kind=vlan

[VLAN]
Id=100
```

```bash
sudo systemctl enable --now systemd-networkd
sudo systemctl status systemd-networkd
networkctl status eth0
```

**The `[Match]` section deserves specific mention, since it's the mechanism that makes these files genuinely portable and reusable across similar hosts: rather than a script hard-coding a specific interface's exact configuration inline, a `.network` file MATCHES against interface properties (by name, by MAC address, by driver) and applies its `[Network]` configuration to whatever real interface happens to match — the same declarative-selector pattern this course's Kubernetes series covers for label selectors, here applied to network interface configuration on a bare host.**

---

## Timers — systemd's Replacement for Cron

```ini
# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup.service daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/backup.service
[Unit]
Description=Backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/run-backup.sh
```

**`Persistent=true` deserves specific emphasis as a genuine, practical improvement over plain cron worth highlighting directly: if the host was OFF (shut down, or the timer service wasn't running) at the exact moment a scheduled run should have fired, `Persistent=true` causes systemd to run the missed job as soon as the system is next available — a real gap plain cron has no native answer for at all, since a cron job simply never runs if the system was down at its scheduled time.** Beyond this specific reliability improvement, systemd timers integrate directly with journald (this chapter's own logging material) for a scheduled job's own output and exit status, and with systemd's own dependency-ordering directives — genuinely more capable than plain cron's isolated, largely unintegrated scheduling model.

---

## Sandboxing a Service — Hardening Directives

Worth a direct connection to this course's own DevSecOps series: systemd provides genuinely powerful, built-in service-sandboxing directives, applying real security hardening to a service with no application-level code changes required at all.

```ini
[Service]
ExecStart=/usr/local/bin/my-app
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/my-app
```

| Directive | Effect |
|---|---|
| `NoNewPrivileges=true` | The process (and any child it spawns) can never gain more privileges than it started with — closes a real privilege-escalation path |
| `ProtectSystem=strict` | The entire filesystem, except explicitly listed `ReadWritePaths=`, becomes read-only to this service |
| `ProtectHome=true` | User home directories become completely inaccessible to this service |
| `RestrictAddressFamilies=` | Limits which socket address families (e.g. only `AF_INET`) the service may even attempt to use |
| `SystemCallFilter=` | A seccomp-based allowlist/denylist restricting which syscalls the service process may invoke at all |
| `systemd-analyze security` | A scored, actionable report of which hardening directives a given unit is still missing |
| `@system-service` | A pre-built syscall filter group covering a typical, well-behaved service's genuine needs |
| `PrivateTmp=true` | The service gets its own private `/tmp`, invisible to (and isolated from) every other process on the host |

**This is worth connecting directly to Part 5's own namespace material, since the mechanism underneath several of these directives is genuinely the same: `PrivateTmp=true` and `ProtectHome=true` work by placing the service inside its own mount namespace (Part 5's own "other namespace types" section, specifically the `mnt` namespace) with a deliberately restricted view of the filesystem — the exact same kernel primitive underlying container isolation, applied here to a single systemd-managed service on a bare host, with zero container runtime involved at all.** A platform engineer hardening a production service with these directives is, concretely, applying container-grade isolation primitives to a plain systemd unit — worth recognizing directly rather than treating systemd sandboxing and container isolation as two unrelated security mechanisms.

`SystemCallFilter=` deserves its own brief, separate note, since it operates through a genuinely different kernel mechanism than the namespace-based directives above: it configures a seccomp-BPF filter — an early preview of the eBPF-adjacent kernel filtering technology Part 7 covers in full depth — restricting the specific set of syscalls a process may invoke at all, closing off an entire class of kernel attack surface (invoking syscalls the service was never designed to need) regardless of what the application code itself does or doesn't validate internally.

systemd ships several pre-built syscall filter groups (`@network-io`, `@file-system`, `@process`) specifically so a platform engineer doesn't need to hand-enumerate individual syscall names — `SystemCallFilter=@system-service` is a commonly-used, broadly-applicable starting point covering the syscalls a typical, well-behaved service actually needs, worth reaching for as a sensible default before hand-tuning a more restrictive, service-specific filter. `systemd-analyze security my-app.service` reports an actual, scored assessment of which hardening directives a given unit is missing — worth running directly against any production service as a concrete, actionable checklist rather than manually cross-referencing this chapter's own directive table by hand.

---

## Drop-In Overrides — Modifying a Unit Without Editing It

Worth a genuinely important, often-overlooked capability: rather than editing a unit file directly (risking the edit being overwritten by a future package upgrade), systemd supports **drop-in override files** — a targeted way to change or add specific directives without touching the original.

```bash
sudo systemctl edit my-app.service
# Opens an editor for a NEW file at:
# /etc/systemd/system/my-app.service.d/override.conf
```

```ini
# The resulting override.conf — ONLY the changed directive, not a full copy
[Service]
MemoryMax=1G
```

**This is worth treating as the correct, standard practice for modifying any package-managed or distribution-provided unit file, worth stating the reasoning precisely: editing `/etc/systemd/system/my-app.service` directly (if it originated from a package) risks that edit being silently overwritten the next time the package updates and reinstalls its own unit file, while a drop-in override in the separate `.service.d/` directory survives package upgrades untouched, since the package only manages the original file, never the override directory.** `systemctl edit --full my-app.service` is the alternative for genuinely replacing an entire unit file rather than layering a targeted override — worth knowing both options exist, and defaulting to the targeted drop-in override for anything short of a complete rewrite.

---

## systemd in Containers — a Genuine Nuance

Worth a direct, honest connection back to Part 5's own container-networking material: **running systemd itself INSIDE a container is a genuinely different, more nuanced topic than running systemd on a bare host**, worth understanding precisely rather than assuming it works identically.

```mermaid
graph TD
    BareHost["systemd on a bare host:\nfull PID 1, full cgroup\nownership, full device access"] --> Normal["Works exactly as this\nchapter has described\nthroughout"]
    InContainer["systemd running AS PID 1\nINSIDE a container"] --> Nuance["Needs SPECIAL container\nconfiguration — cgroup\naccess, specific mount\npoints (/sys/fs/cgroup),\nand often a privileged or\nspecially-configured\ncontainer runtime setup"]
```

**Most container images deliberately do NOT run systemd as their own PID 1 — a typical application container runs the application process directly as PID 1, precisely because a full systemd instance inside a container is genuinely heavier and more complex than most containerized workloads need, and because systemd's own cgroup-management expectations (this chapter's own "exclusive ownership" rule) can conflict with the outer host's own systemd instance without careful, explicit configuration.** Running systemd inside a container is a real, legitimate pattern specifically for certain use cases (testing full-system behavior inside a container, or certain legacy application migration scenarios), but worth knowing as the deliberate exception requiring special container configuration, not the default assumption for how containerized applications should be built.

---

## Debugging a Failing Service — a Practical Workflow

```bash
# 1. Check the unit's own current status
systemctl status my-app.service

# 2. Read recent logs for this specific unit
journalctl -u my-app.service -n 50 --no-pager

# 3. Check for a failed dependency
systemctl list-dependencies my-app.service

# 4. Verify the unit file itself has no syntax errors
systemd-analyze verify my-app.service

# 5. Check WHY a unit failed to start (exit code, signal)
systemctl show my-app.service -p ExecMainStatus -p Result
```

**`systemd-analyze verify` deserves specific mention as a genuinely underused, valuable check: it validates a unit file's own syntax and catches common configuration mistakes (a typo in a directive name, a reference to a non-existent dependency unit) BEFORE attempting to actually start the service** — the same "validate before applying" discipline this series' own netfilter chapter recommended for `nft -c -f`, worth applying here as a standing habit before troubleshooting a failing service by trial and error, since a genuinely common cause of a mysteriously-failing unit is a simple, easily-caught syntax mistake in the unit file itself.

---

## systemd-resolved — DNS Resolution Revisited

Worth a direct callback to Part 2's own DNS material: `systemd-resolved` is systemd's own DNS resolution component, worth understanding as a genuine architectural change from the traditional model Part 2 covered.

```mermaid
graph TD
    Traditional["Traditional model:\n/etc/resolv.conf directly\nlists real, external DNS\nservers. EVERY application\nqueries them directly"] --> ResolvedModel["systemd-resolved model:\n/etc/resolv.conf points to\nresolved's OWN local stub\nresolver (127.0.0.53).\nresolved itself handles\nactual upstream queries,\nCACHING, and per-interface\nDNS configuration"]
```

**The concrete, practical benefit worth stating precisely: `systemd-resolved` provides a local DNS cache (reducing real upstream query volume and latency for repeated lookups) and genuinely per-INTERFACE DNS configuration — a real, practical need for a host with multiple network interfaces (a VPN interface needing different DNS servers than a general internet-facing interface, for instance), which a single, flat, traditional `/etc/resolv.conf` has no clean way to express at all.** `resolvectl status` (or the older `systemd-resolve --status`) shows the actual per-interface DNS configuration currently in effect — worth using directly when debugging a DNS resolution issue on a `systemd-resolved`-managed host, since a plain `cat /etc/resolv.conf` only shows the stub resolver address, not the real, currently-active upstream DNS servers resolved is actually using.

---

## systemd-logind and Session Management

Worth a brief, honest mention of a systemd component this chapter hasn't yet covered: `systemd-logind` manages user login sessions and their associated resource accounting — worth knowing exists specifically because of one common, confusing production interaction.

```mermaid
graph TD
    SSHLogin["An admin SSHes in and\nstarts a long-running\nbackground process with\nnohup or & (NOT via\nsystemd)"] --> LogindQuestion{"Does logind kill this\nprocess when the SSH\nsession ends?"}
    LogindQuestion -->|"Depends on\nKillUserProcesses\nsetting"| Answer["DEFAULT on many distros:\nYES — logind cleans up\nALL of a user's processes\nwhen their LAST session\nends, including background\njobs started via nohup"]
```

**This is worth stating directly as a genuinely common source of confusion for anyone used to older init systems: on a host with logind's `KillUserProcesses=yes` (the default on several distributions), a background process started via `nohup command &` during an SSH session can still be killed when that SSH session ends, surprising an admin who assumed `nohup` alone was sufficient protection.** The correct, robust fix for anything genuinely needing to survive a session end is not relying on `nohup` at all, but instead running it as a proper systemd unit (this chapter's entire subject) — or, at minimum, explicitly using `systemd-run --scope` to launch it outside the user's own login session scope entirely, avoiding this cleanup behavior by construction rather than hoping `nohup` alone is enough on a given distribution's specific logind configuration.

---

## A Full Realistic Example: a Production-Hardened, Socket-Activated Service

Tying nearly every mechanism from this chapter together into one complete, production-realistic unit definition — a socket-activated API service with resource limits, sandboxing, and a disciplined restart policy.

```ini
# /etc/systemd/system/api.socket
[Unit]
Description=Socket for the API service

[Socket]
ListenStream=8443

[Install]
WantedBy=sockets.target
```

```ini
# /etc/systemd/system/api.service
[Unit]
Description=Production API service
Requires=api.socket
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/api-server
User=api
Group=api

# Resource control (this chapter's own cgroup material)
CPUQuota=75%
MemoryMax=1G
MemoryHigh=800M

# Sandboxing (this chapter's own hardening material)
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/api

# Restart policy, with a circuit breaker
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

[Install]
WantedBy=multi-user.target
```

**Worth reading this pair of files as the concrete, cumulative synthesis of this entire chapter: socket activation (zero idle resource cost), `network-online.target` dependency ordering (correct, not the common `network.target` mistake), cgroup-backed resource limits (a real ceiling and a softer throttle point), full sandboxing (container-grade isolation with zero container runtime involved), and a disciplined restart policy with an actual circuit breaker — every one of these mechanisms this chapter covered individually, now composed together into a single, genuinely production-realistic service definition.** `Type=notify` (used here rather than the earlier examples' `Type=simple`) is worth a brief additional note: it requires the application itself to send an explicit "I'm actually ready" signal back to systemd via the `sd_notify()` protocol, letting systemd know precisely when startup has genuinely completed — more accurate than `Type=simple`'s assumption that the process starting IS the same as the service being ready, useful for services with a real, non-trivial startup/warm-up sequence.

---

## Watchdogs — Detecting a Hung, Not Just a Crashed, Process

Worth a direct extension of this chapter's own restart-policy material: `Restart=on-failure` only catches a process that has actually exited — it does nothing for a process that's still technically running but has genuinely hung, deadlocked, or stopped making progress. systemd's watchdog mechanism closes exactly this gap.

```ini
[Service]
ExecStart=/usr/local/bin/my-app
WatchdogSec=30
Restart=on-watchdog
```

```mermaid
graph TD
    AppRunning["Application process,\nstill technically alive"] --> Ping["App must call sd_notify\nWATCHDOG=1 periodically\n(the app's OWN responsibility\nto integrate this)"]
    Ping --> SystemdCheck{"Has systemd received\na ping within\nWatchdogSec?"}
    SystemdCheck -->|"No — the process\nhas HUNG, not exited"| Restart["systemd treats this as\na FAILURE and restarts\nthe service (Restart=\non-watchdog)"]
```

**This is worth stating precisely, since it's a genuinely distinct failure mode from anything `Restart=on-failure` alone catches: a process that's deadlocked, stuck in an infinite loop, or otherwise unresponsive is still technically "running" from the kernel's own point of view — its exit code is irrelevant because it never exits — and `Restart=on-failure` has no signal to act on at all in this scenario.** The watchdog mechanism requires the application itself to periodically call `sd_notify(WATCHDOG=1)` as an active "I'm still genuinely making progress" heartbeat — a real integration requirement on the application's own side, worth checking for before assuming watchdog-based hang detection is available for an arbitrary existing service, directly analogous to a Kubernetes liveness probe's own "prove you're actually healthy, not just alive" distinction covered in this course's Kubernetes series.

---

## A systemd Change Checklist for Production Hosts

Worth closing this chapter's practical guidance with one final, walkable checklist, directly extending the change-management discipline this series has built up across Part 4 and Part 5's own closing checklists.

| Step | Why |
|---|---|
| 1. Validate the unit file syntax with `systemd-analyze verify` | Catches typos and bad references before attempting a real start |
| 2. Run `daemon-reload` after any unit file change | Required — systemd caches unit files in memory |
| 3. Test with `systemctl start` before `enable`-ing | Confirms the unit actually works before committing it to every future boot |
| 4. Confirm dependency ordering with `systemctl list-dependencies` | Verifies the actual resolved dependency graph, not just the directives as written |
| 5. Check resource limits took effect via `systemctl status` (shows current cgroup values) | Confirms `MemoryMax=`/`CPUQuota=` are genuinely enforced, not just declared |
| 6. Prefer a drop-in override over editing a package-managed unit directly | Survives the next package upgrade |
| 7. Confirm journald is actually capturing and persisting output | `Storage=persistent` and a working log-shipping pipeline, not just local, volatile storage |
| 8. Re-run `systemd-analyze critical-chain` if the change touches boot-time ordering | Confirms the change didn't inadvertently lengthen the critical boot path |

**This checklist deliberately mirrors the same shape as this series' own Part 4 and Part 5 closing checklists — worth treating all three as one continuous production-change discipline spanning firewall rules, virtual networking, and now service/unit management, rather than three unrelated processes.**

---

## Analyzing Boot Performance

Worth a closing, practical observability connection: systemd provides genuine, built-in tooling for diagnosing slow boots — a real, if infrequent, production concern for anything sensitive to boot time (an autoscaled fleet where node startup latency directly affects scale-up responsiveness, per this course's own CI/CD & GitOps series' self-hosted runner chapter).

```bash
# Overall boot time breakdown
systemd-analyze

# Per-unit startup time, slowest first
systemd-analyze blame

# A visual, dependency-aware breakdown of the critical boot path
systemd-analyze critical-chain
```

**`critical-chain` deserves specific emphasis over `blame` alone, since it answers a genuinely different, more actionable question: `blame` lists every unit's own individual startup time in isolation, while `critical-chain` shows the actual DEPENDENCY CHAIN that determined the overall boot time — a unit that took a long time to start but wasn't actually on the critical path (nothing else was blocked waiting on it) doesn't matter nearly as much for total boot latency as a much faster unit that happened to be sitting directly on the chain everything else depended on.** This directly parallels this course's own CI/CD pipeline material — the same "slowest stage" versus "critical path" distinction already covered for pipeline stage ordering — worth recognizing as the identical performance-analysis principle, now applied to systemd's own boot sequence instead of a CI/CD pipeline's stages.

---

## Common Mistakes

| Mistake | Why it's a problem | Fix |
|---|---|---|
| Editing a unit file without running `daemon-reload` afterward | systemd caches unit file contents; the edit has no effect until reloaded | Always run `systemctl daemon-reload` after any unit file change |
| Using only `After=` without a paired `Wants=`/`Requires=` | Controls timing only — the unit still starts even if the "after" unit never starts at all | Pair ordering directives with the corresponding strength directive |
| Depending on `network.target` instead of `network-online.target` for a service needing real connectivity | `network.target` only means the network SERVICE started, not that an interface has a usable IP yet | Use `network-online.target` (with `Wants=`) for any service genuinely needing network connectivity at startup |
| Setting `Restart=always` (or `on-failure`) with no `StartLimitBurst`/`StartLimitIntervalSec` | A persistently crashing service restarts indefinitely, generating unbounded log noise and churn | Always pair a restart policy with start-limit directives as a circuit breaker |
| Editing a distribution-provided unit file directly instead of using a drop-in override | The edit risks being silently overwritten on the next package upgrade | Use `systemctl edit` to create a `.service.d/override.conf` drop-in instead |
| Assuming a container should run full systemd as PID 1 by default | Adds real, usually unnecessary complexity and cgroup-ownership conflicts with the host | Run the application process directly as PID 1 in a container; reserve in-container systemd for the specific, deliberate cases that need it |
| Relying purely on local journald storage with no centralized log shipping | Local storage may be volatile (`tmpfs`-backed) by default, and is lost entirely if the host itself is lost | Set `Storage=persistent` explicitly, and ship logs to a centralized, durable pipeline for anything that matters |
| Optimizing boot time based on `systemd-analyze blame` alone | Shows per-unit duration in isolation, not whether that unit is actually on the critical dependency path | Use `systemd-analyze critical-chain` to identify the real, boot-time-determining dependency chain |
| Assuming `Restart=on-failure` catches a hung, non-exiting process | It only reacts to an actual process exit — a deadlocked-but-alive process produces no signal it can act on | Add a watchdog (`WatchdogSec=` + `Restart=on-watchdog`) with application-side `sd_notify()` heartbeat integration |

---

## Worked Practice Problems

**Problem 1:** A team writes a `.service` unit with `After=postgresql.service` but no `Wants=` or `Requires=` directive, expecting their application to fail cleanly if PostgreSQL isn't running. Instead, their application starts anyway (and then crashes trying to connect to a database that was never started). What went wrong?

*Answer:* `After=` alone controls only ordering (WHEN the unit starts relative to another), not whether it starts AT ALL if the other unit fails or is absent. Without a paired `Wants=postgresql.service` (soft dependency) or `Requires=postgresql.service` (hard dependency), the application's own unit has no directive at all preventing it from starting even if `postgresql.service` never runs — the fix is adding the appropriate strength directive alongside the existing `After=`.

**Problem 2:** A service depends on `network.target` and works fine most of the time, but occasionally fails to connect to an external API immediately after boot, succeeding on a subsequent manual restart. Diagnose the likely cause.

*Answer:* `network.target` only guarantees the network management service itself has started, not that any actual interface has a usable IP address and real connectivity — the service can start before DHCP has actually completed, producing an intermittent, boot-timing-dependent failure. The fix is depending on `network-online.target` (paired with `Wants=`) instead, which specifically waits for interfaces to be genuinely up and usable before the dependent unit starts.

**Problem 3:** A platform engineer needs to increase a production service's memory limit but the unit file is owned and managed by the distribution's own package. What's the correct way to make this change so it survives the next package upgrade?

*Answer:* `systemctl edit my-app.service`, creating a drop-in override file at `/etc/systemd/system/my-app.service.d/override.conf` containing only the changed `MemoryMax=` directive — not editing the original package-managed unit file directly, which risks being silently overwritten the next time the package updates. The drop-in directory is never touched by the package's own files, so the override survives upgrades untouched.

**Problem 4:** A container runtime (containerd) running as a systemd-managed service needs to create and manage its own per-container cgroups, but systemd is supposed to have exclusive ownership of the cgroup v2 hierarchy. How is this conflict actually resolved?

*Answer:* Cgroup delegation — the `containerd.service` unit is marked `Delegate=yes`, which systemd-authorizes as an explicit hand-off of management authority over that unit's own cgroup sub-tree. This lets containerd create and manage its own per-container cgroups beneath that delegated point without violating the kernel's single-writer-per-subtree rule or systemd's own general exclusive-ownership policy — the conflict is resolved by explicit, sanctioned delegation rather than either side bypassing the other.

**Problem 5:** A team wants a rarely-used internal admin API to consume zero resources when idle, but still respond immediately to the very first request without any noticeable cold-start delay. Which systemd capability directly provides this, and how?

*Answer:* Socket activation — systemd creates and listens on the service's socket immediately at boot (so the very first connection is accepted immediately, with no delay), but doesn't actually start the service's own process until that first connection genuinely arrives. This gives exactly the described property: zero resource consumption while idle, and immediate responsiveness on first use, since the listening socket itself is already active before the process starts.

**Problem 6:** A service is genuinely still running (its process hasn't exited) but has deadlocked and stopped responding entirely. The team has `Restart=on-failure` configured, but systemd never restarts it. Why not, and what should they add?

*Answer:* `Restart=on-failure` only reacts to a process actually exiting with a failure condition — a deadlocked process that never exits produces no signal `Restart=on-failure` can act on at all, since from the kernel's own point of view the process is still technically alive. The fix is adding a watchdog (`WatchdogSec=` plus `Restart=on-watchdog`), which requires the application to periodically call `sd_notify(WATCHDOG=1)` as an active heartbeat — if that heartbeat stops arriving within the configured interval, systemd treats the hang itself as a failure and restarts the service, catching exactly this failure mode that exit-code-based restart policies cannot.

**Problem 7:** An admin SSHes into a production host, starts a long-running data migration script with `nohup ./migrate.sh &`, and disconnects, expecting it to keep running. They later discover it was killed when their SSH session ended. What's the likely cause, and what's the more robust fix?

*Answer:* `KillUserProcesses=yes` (systemd-logind's default on several distributions) kills all of a user's processes — including `nohup`-backgrounded ones — when their last login session ends, a real behavior `nohup` alone doesn't protect against on such a system. The more robust fix is running the script as a genuine systemd unit (a `oneshot` service, or via `systemd-run --scope`) rather than relying on `nohup` and a background shell job, since a proper systemd-managed process is not tied to any particular login session's lifecycle at all.

**Problem 8:** A platform team notices their fleet's average boot time increased significantly after a recent change, and `systemd-analyze blame` shows one particular unit taking the longest individual startup time. They fix that unit, but total boot time barely improves. What did they likely misdiagnose?

*Answer:* They optimized based on `blame` alone, which shows each unit's individual startup duration in isolation, not whether that unit was actually on the critical dependency path determining total boot time. A slow unit that nothing else was waiting on doesn't meaningfully affect overall boot time even if it's individually the slowest; the correct diagnostic tool is `systemd-analyze critical-chain`, which shows the actual chain of dependencies that determined the real, total boot duration — the fix should target whichever unit sits on that critical chain, not necessarily the single slowest unit in isolation.

---

## Key Terms Glossary — This Chapter's Vocabulary in One Place

| Term | Meaning in this chapter's context |
|---|---|
| Unit | The general term for anything systemd manages — services, sockets, targets, timers, and more |
| `Wants=` / `Requires=` | Dependency strength — soft vs. hard failure propagation between units |
| `After=` / `Before=` | Dependency ordering — timing only, independent of strength |
| Target | A named synchronization point, replacing old-style numbered runlevels |
| Socket activation | Deferring a service's actual start until its first real connection arrives |
| journald | systemd's structured, binary logging daemon — automatic per-entry metadata |
| cgroup delegation | The sanctioned mechanism letting a service (e.g. a container runtime) manage its own cgroup sub-tree |
| `systemd-networkd` | Declarative, persistent network interface configuration, reapplied automatically every boot |
| Watchdog (`WatchdogSec=`) | Detects a hung-but-still-running process via a required periodic application heartbeat |
| Drop-in override | A `.service.d/override.conf` file modifying a unit without editing (and risking overwriting) the original |
| `systemd-resolved` | systemd's local DNS stub resolver and cache, with per-interface DNS configuration |
| `Type=notify` | A service type requiring the app to explicitly signal readiness via `sd_notify()` |
| `systemd-analyze critical-chain` | Shows the actual dependency chain determining total boot time, distinct from per-unit blame |
| `KillUserProcesses` | logind's setting controlling whether a user's background processes die with their last login session |
| `MemoryHigh=` vs. `MemoryMax=` | Soft throttle vs. hard OOM-kill ceiling — the same two-tier model Kubernetes expresses via requests/limits |

---

## Summary and What's Next

systemd is PID 1 on nearly every modern Linux distribution — a broader suite (init core, journald, networkd, resolved, timers) rather than a single monolithic init process, unifying nearly every "coordinated, dependency-aware system task" (starting services, mounting filesystems, running scheduled jobs) under one consistent unit-based model. Dependency ordering (`After=`/`Before=`) and dependency strength (`Wants=`/`Requires=`) are genuinely separate concerns that need to be paired deliberately — `Wants=`+`After=` is the recommended default for most real service relationships, reserving the harder-failing `Requires=` for genuinely non-negotiable dependencies. Targets replace old-style runlevels as named synchronization points, with `network-online.target` specifically resolving a common, real boot-ordering gotcha that plain `network.target` doesn't address. Socket activation provides genuine scale-to-zero behavior for a single host-level service, directly parallel to the Kubernetes-scale scale-to-zero concepts this course's CI/CD & GitOps series covers at fleet scale. journald's structured, automatically-metadata-rich logging requires zero application instrumentation to gain real value, and integrates directly with this course's own centralized log-shipping material via its JSON export. systemd's exclusive, kernel-enforced ownership of the cgroup v2 hierarchy is what every `[Service]`-section resource-control directive (`MemoryMax=`, `CPUQuota=`) ultimately compiles down to, with cgroup delegation as the explicit, sanctioned mechanism letting a container runtime manage its own per-container cgroups without violating that ownership. `systemd-networkd` directly resolves Part 5's own closing gap — declarative, version-controllable, persistent network interface configuration, applied automatically on every boot using the identical underlying kernel mechanisms Part 5 covered by hand. Service sandboxing directives (`ProtectSystem=`, `PrivateTmp=`) apply real, container-grade isolation to a plain systemd unit using the same namespace primitives Part 5 covered directly, with zero container runtime required at all.

**Part 7, immediately following, closes this series with eBPF** — a genuinely different, increasingly significant kernel-level mechanism this series has already previewed twice (Part 4's own XDP preview, and Cilium's eBPF-based CNI dataplane referenced across both Parts 4 and 5) — covering eBPF's own programming model, `bpftrace` for ad-hoc kernel-level tracing, and exactly how Cilium builds a complete CNI dataplane on top of these same primitives.

Every mechanism this chapter has covered — units, dependencies, targets, cgroups, sandboxing — is worth recognizing as the same underlying discipline running throughout this entire series: explicit, declarative configuration, continuously enforced by the kernel and its userspace managers, rather than implicit, manually-maintained state a human has to remember to keep consistent by hand.