systemd Deep Dive

Table of Contents#

1. Why This Part Exists
2. What systemd Actually Is — PID 1 and Beyond
3. Unit Types — More Than Just Services
4. Anatomy of a .service Unit File
5. A Minimal Service Unit, Built Up Step by Step
6. Dependency Ordering — Wants, Requires, After, Before
7. Why Wants+After Is the Default Recommended Pattern
8. Targets — systemd's Replacement for Runlevels
9. The Boot Sequence, Concretely
10. Socket Activation — Starting a Service On Demand
11. A Worked Example: Socket-Activated Service
12. Restart Policies and Service Supervision
13. journald — Structured, Centralized Logging
14. Querying journald With journalctl
15. journald's Own Storage and Retention
16. cgroup v2 Integration — systemd as the Cgroup Manager
17. Resource Control in a Unit File
18. Cgroup Delegation — Why Container Runtimes Need It
19. systemd-networkd — Declarative, Persistent Network Configuration
20. A Worked Example: systemd-networkd Configuring an Interface
21. Timers — systemd's Replacement for Cron
22. Sandboxing a Service — Hardening Directives
23. Drop-In Overrides — Modifying a Unit Without Editing It
24. systemd in Containers — a Genuine Nuance
25. Debugging a Failing Service — a Practical Workflow
26. systemd-resolved — DNS Resolution Revisited
27. systemd-logind and Session Management
28. A Full Realistic Example: a Production-Hardened, Socket-Activated Service
29. Watchdogs — Detecting a Hung, Not Just a Crashed, Process
30. A systemd Change Checklist for Production Hosts
31. Analyzing Boot Performance
32. Common Mistakes
33. Worked Practice Problems
34. Key Terms Glossary — This Chapter's Vocabulary in One Place
35. Summary and What's Next

Why This Part Exists#

Parts 4 and 5 covered, in hands-on depth, exactly how packets get filtered, NAT'd, routed, and how virtual network topology gets built. Neither chapter addressed a genuinely foundational question sitting underneath all of it: what actually starts the processes, brings up the network interfaces, and keeps everything running on a real, booted Linux host in the first place? This chapter answers that question directly — systemd is the init system and service manager responsible for exactly this, on the overwhelming majority of production Linux distributions in 2026.

Worth stating directly why this belongs in a networking-focused series specifically: systemd-networkd is one of the standard mechanisms for declaring persistent network interface configuration — the exact detail Part 5 flagged as its own closing forward-reference, since every ip link/ip addr command demonstrated there configures state that vanishes on reboot without something like this chapter's own material making it persistent.

What systemd Actually Is — PID 1 and Beyond#

systemd is not just "the init system" in the narrow, historical sense (the very first userspace process the kernel starts) — it's a much broader suite of integrated system-management components, worth naming precisely rather than treating as one monolithic, undifferentiated thing.

Worth being precise about the "PID 1" property specifically, since it carries real operational consequences: PID 1 is a genuinely special process from the kernel's own point of view — it's the parent (directly or via reparenting) of every other process on the system, and if it crashes, the kernel panics, since there's no other process left to manage the entire system. This is exactly why systemd's own reliability and correctness matters disproportionately compared to an ordinary service — a bug in PID 1 itself, unlike a bug in any individual managed service, has no graceful failure mode; the whole system goes down with it.

Unit Types — More Than Just Services#

Everything systemd manages is expressed as a unit — worth surveying the full range of unit types, since "systemd manages services" undersells its actual scope considerably.

The breadth of this table is worth internalizing directly: nearly every "system-level thing that needs to happen in a coordinated, dependency-aware order" — starting a service, mounting a filesystem, waiting for a device, running a scheduled job — is unified under the exact same dependency-graph, unit-based model, rather than each having its own separate, bespoke mechanism the way older init systems typically required. This chapter focuses primarily on .service, .socket, .target, and .timer units, since they cover the large majority of what a platform engineer actually authors directly.

Anatomy of a .service Unit File#

[Unit]
Description=A simple example web application
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my-app --port 8080
Restart=on-failure
RestartSec=5
User=myapp
Group=myapp

[Install]
WantedBy=multi-user.target

Every unit file's three-section structure is worth memorizing directly, since it's genuinely universal across nearly every unit type this chapter covers: [Unit] holds metadata and dependency directives common to any unit type; the type-specific section ([Service], here) holds directives specific to that unit type; [Install] defines what happens when the unit is enabled — which target(s) should pull this unit in automatically at boot. Type=simple (the default) tells systemd the process specified by ExecStart IS the main service process itself, immediately — other Type= values (covered implicitly through this chapter's own worked examples) change this assumption for processes with more complex startup behavior.

A Minimal Service Unit, Built Up Step by Step#

# 1. Write the unit file
sudo tee /etc/systemd/system/my-app.service <<'EOF'
[Unit]
Description=My Application

[Service]
ExecStart=/usr/local/bin/my-app
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd's own unit cache — REQUIRED after any unit file change
sudo systemctl daemon-reload

# 3. Start it immediately
sudo systemctl start my-app

# 4. Enable it to start automatically on future boots
sudo systemctl enable my-app

# 5. Check status
sudo systemctl status my-app

Step 2 deserves the strongest emphasis of this sequence, since forgetting it is the single most common source of "I edited the unit file but nothing changed" confusion: systemd caches unit file contents in memory, and daemon-reload is what tells it to re-read the actual files on disk — editing a unit file directly has zero effect on a running or subsequently-started service until this command runs. This is worth treating as a standing habit after any unit file edit, not an occasionally-remembered extra step.

Dependency Ordering — Wants, Requires, After, Before#

Worth a precise, careful distinction between two genuinely separate concepts this chapter's own directive names can easily blur together: ordering (which unit starts before/after another) and dependency strength (whether a failure in one unit should affect the other at all).

This is the single most important, most commonly conflated distinction in this entire chapter's dependency material, worth stating as plainly as possible: After=foo.service alone does NOT mean "start after foo, and only if foo succeeds" — it only affects timing, and a unit with only an After= directive (no Wants=/Requires=) will still attempt to start even if foo.service never starts at all. Wants=foo.service (a soft dependency — a failure of foo doesn't prevent this unit from starting) and Requires=foo.service (a hard dependency — this unit fails to start if foo fails) each need to be paired with the corresponding After= directive to actually control both "whether" and "when" together — specifying only one or the other is a genuinely common source of subtle boot-ordering bugs.

Why Wants+After Is the Default Recommended Pattern#

[Unit]
After=network-online.target
Wants=network-online.target

Wants=+After=, used together, is the systemd-recommended default pairing for the large majority of real service dependencies — worth understanding precisely why Requires= is NOT the default recommendation, since this is a genuinely counter-intuitive point on first encounter. Requires= creates a hard failure dependency — if the required unit fails to start for ANY reason, the dependent unit fails too, cascading failures across the boot sequence in a way that's often more brittle than actually desired. Wants= expresses "start this too, and prefer it to have started first (via the paired After=)," without turning a transient failure in one unit into a hard failure of every unit that merely wanted it — the correct default posture for most real-world service relationships, reserving Requires= specifically for cases where a genuine, non-negotiable hard dependency exists (a service that is architecturally meaningless without its database connection, for instance).

Targets — systemd's Replacement for Runlevels#

A target is a synchronization point — a named grouping other units can depend on, replacing the old, numbered runlevel concept (runlevel 3, runlevel 5) with named, more expressive groupings.

The network-online.target distinction deserves specific, careful emphasis, since it resolves a genuinely common source of boot-order bugs: network.target merely means the network management SERVICE has started, not that any interface actually has a usable IP address or connectivity yet — a service that genuinely needs real network connectivity at startup (this chapter's own my-app.service example, reaching an external database) should depend on network-online.target specifically, not the more commonly (and often incorrectly) referenced plain network.target. This distinction is a real, frequently-encountered production gotcha — a service depending only on network.target can start successfully before the network interface it actually needs has finished acquiring a DHCP lease, failing intermittently in a way that looks like a flaky application bug rather than the actual, systemic ordering mistake it is.

The Boot Sequence, Concretely#

systemctl get-default reveals which target a given host actually boots into — worth checking directly rather than assuming, since a misconfigured default target (booting into rescue.target unexpectedly, for instance) is a genuinely real, if uncommon, production incident category. Every unit ultimately activated during a normal boot traces back to being a transitive Wants=/Requires= dependency of this one default target — the entire boot sequence is, concretely, systemd resolving one large dependency graph rooted at that single target.

Socket Activation — Starting a Service On Demand#

Socket activation is a genuinely distinctive systemd capability worth understanding in depth: rather than starting a service eagerly at boot, systemd can instead create and listen on a socket immediately, and only actually start the corresponding service process the first time a connection actually arrives.

The concrete, practical benefit worth stating precisely: a rarely-used service consumes zero memory/CPU until its first actual connection, while still being immediately available (no cold-start delay perceptible to the FIRST connecting client, since the socket itself is already listening even before the process starts) — a genuine resource-efficiency win for services that are needed occasionally but not constantly. This is worth connecting directly to this course's own CI/CD & GitOps series' self-hosted runner scaling chapter and its own "scale-to-zero" framing — socket activation is, conceptually, the exact same scale-to-zero idea applied at the level of one systemd-managed service on one host, rather than an entire fleet of Kubernetes runner Pods.

A Worked Example: Socket-Activated Service#

# /etc/systemd/system/my-app.socket
[Unit]
Description=Socket for my-app

[Socket]
ListenStream=8080

[Install]
WantedBy=sockets.target

# /etc/systemd/system/my-app.service
[Unit]
Description=My Application (socket-activated)
Requires=my-app.socket

[Service]
ExecStart=/usr/local/bin/my-app
# No explicit port binding needed in the app itself —
# systemd passes the already-open socket via file descriptor

sudo systemctl enable --now my-app.socket
# my-app.service itself is NOT started yet — check:
systemctl status my-app.service   # inactive (dead)

# First connection triggers activation:
curl http://localhost:8080
systemctl status my-app.service   # now active (running)

Worth noticing directly what's absent from my-app.service's own [Service] section: no explicit port-binding configuration at all, because the application itself doesn't open the listening socket — systemd already did, and hands the already-bound file descriptor to the service process on startup. This requires the application itself to support systemd's socket-passing convention (reading the pre-opened file descriptor rather than calling its own bind()/listen()), a real integration requirement worth checking before assuming socket activation is a drop-in capability for any arbitrary existing service.

Restart Policies and Service Supervision#

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

StartLimitIntervalSec+StartLimitBurst deserve specific emphasis as a genuinely important safety valve worth including alongside any Restart= policy other than no: without them, a service crash-looping due to a persistent underlying bug restarts indefinitely, potentially generating an unbounded flood of restart attempts, log noise, and resource churn. With the values shown above, systemd gives up after 3 restart attempts within any 60-second window, transitioning the unit to a failed state instead of restarting forever — a real, concrete circuit-breaker pattern directly analogous to the retry-budget concepts this course's own Reliability & Architecture series covers for application-level retries, now applied at the process-supervision layer.

journald — Structured, Centralized Logging#

systemd-journald is systemd's own logging daemon — worth understanding as a genuine architectural departure from older, plain-text syslog: journald stores logs in a structured, indexed binary format, capturing every log entry with rich, queryable metadata automatically.

The "zero extra application instrumentation" property deserves the strongest emphasis, since it's a genuine, practical operational win worth connecting directly to this course's own Observability series: any application that simply writes to stdout/stderr — no logging library integration, no structured-logging discipline required from the application itself — automatically gets rich, per-entry metadata (which unit, which PID, which boot) captured by journald, purely because systemd started and supervises that process. This is a meaningfully lower floor than the application-level structured-logging discipline this course's Observability series otherwise recommends — journald's own metadata capture happens regardless of whether the application itself does anything special at all.

Querying journald With journalctl#

# Logs for a specific unit
journalctl -u my-app.service

# Follow logs in real time (like tail -f)
journalctl -u my-app.service -f

# Logs since a specific time
journalctl -u my-app.service --since "1 hour ago"

# Only this boot's logs
journalctl -b

# Kernel-only messages (dmesg equivalent — Part 4's own debugging workflow used this)
journalctl -k

# JSON output, for piping into another tool
journalctl -u my-app.service -o json

The -o json output format deserves specific mention as a genuinely important integration point with this course's own Observability series' log-shipping material: journald's structured, machine-parseable JSON export is exactly what a log-forwarding agent (Fluentd, Vector, or an equivalent) consumes to ship these logs into a centralized aggregation pipeline (Loki, Elasticsearch) without needing to parse fragile, free-text log lines — the structured metadata journald captures automatically becomes structured fields in the shipped log entry, directly, with no additional parsing logic required on the shipping side.

journald's Own Storage and Retention#

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=30day

Storage=persistent deserves specific mention as a setting worth checking explicitly on any production host, since journald's own default behavior varies by distribution: without persistent storage configured, journal logs may live only in a tmpfs-backed, volatile location, meaning every log entry is lost on a reboot — a real, easy-to-miss gap for any host relying purely on local journald storage as its log retention strategy without also shipping logs to a centralized, durable pipeline. SystemMaxUse and MaxRetentionSec are the practical disk-space and time-based retention controls, worth setting explicitly on any host rather than trusting the distribution's own default, which can consume more local disk than a specific host's own storage budget allows for.

cgroup v2 Integration — systemd as the Cgroup Manager#

Worth a direct, important architectural fact: on essentially every modern Linux distribution, systemd is the exclusive, authoritative manager of the cgroup v2 hierarchy — it creates a sub-tree for every unit it manages, and enforces a real kernel-level rule that only one userspace agent may manage a given cgroup subtree at a time.

This "exclusive ownership" property is worth stating precisely, since it directly explains a real, sometimes-confusing operational detail: a tool attempting to manage cgroups directly, bypassing systemd, on a system where systemd already owns the hierarchy will either fail outright or produce inconsistent, conflicting state — the kernel's own single-writer rule for a given cgroup subtree means systemd's ownership isn't a mere convention, it's an enforced constraint. Every [Service]-section resource-control directive covered in this chapter's next section is, concretely, systemd translating that directive into real cgroup v2 controller files on the unit's own dedicated subtree.

Resource Control in a Unit File#

[Service]
ExecStart=/usr/local/bin/my-app
CPUQuota=50%
MemoryMax=512M
MemoryHigh=400M
TasksMax=100

MemoryHigh= deserves specific emphasis as a genuinely more graceful mechanism than MemoryMax= alone, worth connecting directly to this course's own Kubernetes Deep Dive series' own requests-vs-limits framing: MemoryHigh is conceptually closer to a soft throttle (the kernel slows the cgroup's own memory allocation, applying backpressure) while MemoryMax is a hard kill boundary (the OOM killer is invoked once crossed) — the same two-tier "soft pressure, then hard limit" pattern Kubernetes itself expresses via requests and limits, here expressed directly through systemd's own resource-control directives on a bare host with no Kubernetes involved at all.

This is worth internalizing as more than a passing analogy: a Kubernetes node's own kubelet is itself just another systemd-managed service, and every Pod's cgroup ultimately nests underneath that node's own systemd-managed cgroup hierarchy — the resource-control mechanics covered in this section aren't a separate system from what a Kubernetes cluster does, they're the literal foundation every Kubernetes node builds its own Pod resource enforcement on top of.

Cgroup Delegation — Why Container Runtimes Need It#

Worth a direct, important connection to Part 5's own container-networking material: a container runtime (Docker, containerd) needs to create and manage its OWN cgroups for each container it starts — but systemd's exclusive-ownership rule (covered in this chapter's own cgroup section) would normally forbid exactly that.

Cgroup delegation is worth understanding as the explicit, sanctioned mechanism resolving this exact conflict — a unit marked Delegate=yes in its own [Service] section is given genuine, systemd-authorized management authority over its own cgroup sub-tree, letting a container runtime running as a systemd-managed service (docker.service or containerd.service, typically already configured this way by the distribution's own packaging) create and manage its own per-container cgroups beneath that delegated point without violating systemd's own single-writer rule. This is worth recognizing directly as the real, concrete mechanism underneath every container's own resource limits (docker run --memory=512m, or a Kubernetes Pod's own resource limits) — ultimately expressed as cgroup v2 controller files, within a subtree systemd explicitly delegated to the container runtime for exactly this purpose.

systemd-networkd — Declarative, Persistent Network Configuration#

Worth the direct payoff of this chapter's own opening forward-reference from Part 5: systemd-networkd declares network interface configuration in version-controllable, plain-text .network/.link/.netdev files, applied automatically and persistently at boot — the exact mechanism Part 5's own ip link/ip addr commands lacked.

This is worth stating precisely as the resolution to the exact gap Part 5 flagged explicitly: every ip link/ip addr command Part 5 demonstrated configures state that vanishes on reboot, while a .network file declares the identical configuration persistently — systemd-networkd re-applies it automatically on every boot, using the same underlying kernel mechanisms Part 5 covered by hand, just declared once and enforced continuously rather than requiring manual re-application. This is a real, concrete example of the "declarative configuration, continuously reconciled" pattern this entire course returns to repeatedly (Kubernetes' own reconciliation loops, GitOps's own continuous reconciliation) — now demonstrated at the level of a single host's own network interface configuration.

A Worked Example: systemd-networkd Configuring an Interface#

# /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes

[DHCP]
UseDNS=yes
UseRoutes=yes

# /etc/systemd/network/20-vlan100.netdev — a VLAN interface, declaratively
[NetDev]
Name=vlan100
Kind=vlan

[VLAN]
Id=100

sudo systemctl enable --now systemd-networkd
sudo systemctl status systemd-networkd
networkctl status eth0

The [Match] section deserves specific mention, since it's the mechanism that makes these files genuinely portable and reusable across similar hosts: rather than a script hard-coding a specific interface's exact configuration inline, a .network file MATCHES against interface properties (by name, by MAC address, by driver) and applies its [Network] configuration to whatever real interface happens to match — the same declarative-selector pattern this course's Kubernetes series covers for label selectors, here applied to network interface configuration on a bare host.

Timers — systemd's Replacement for Cron#

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup.service daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/run-backup.sh

Persistent=true deserves specific emphasis as a genuine, practical improvement over plain cron worth highlighting directly: if the host was OFF (shut down, or the timer service wasn't running) at the exact moment a scheduled run should have fired, Persistent=true causes systemd to run the missed job as soon as the system is next available — a real gap plain cron has no native answer for at all, since a cron job simply never runs if the system was down at its scheduled time. Beyond this specific reliability improvement, systemd timers integrate directly with journald (this chapter's own logging material) for a scheduled job's own output and exit status, and with systemd's own dependency-ordering directives — genuinely more capable than plain cron's isolated, largely unintegrated scheduling model.

Sandboxing a Service — Hardening Directives#

Worth a direct connection to this course's own DevSecOps series: systemd provides genuinely powerful, built-in service-sandboxing directives, applying real security hardening to a service with no application-level code changes required at all.

[Service]
ExecStart=/usr/local/bin/my-app
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/my-app

This is worth connecting directly to Part 5's own namespace material, since the mechanism underneath several of these directives is genuinely the same: PrivateTmp=true and ProtectHome=true work by placing the service inside its own mount namespace (Part 5's own "other namespace types" section, specifically the mnt namespace) with a deliberately restricted view of the filesystem — the exact same kernel primitive underlying container isolation, applied here to a single systemd-managed service on a bare host, with zero container runtime involved at all. A platform engineer hardening a production service with these directives is, concretely, applying container-grade isolation primitives to a plain systemd unit — worth recognizing directly rather than treating systemd sandboxing and container isolation as two unrelated security mechanisms.

SystemCallFilter= deserves its own brief, separate note, since it operates through a genuinely different kernel mechanism than the namespace-based directives above: it configures a seccomp-BPF filter — an early preview of the eBPF-adjacent kernel filtering technology Part 7 covers in full depth — restricting the specific set of syscalls a process may invoke at all, closing off an entire class of kernel attack surface (invoking syscalls the service was never designed to need) regardless of what the application code itself does or doesn't validate internally.

systemd ships several pre-built syscall filter groups (@network-io, @file-system, @process) specifically so a platform engineer doesn't need to hand-enumerate individual syscall names — SystemCallFilter=@system-service is a commonly-used, broadly-applicable starting point covering the syscalls a typical, well-behaved service actually needs, worth reaching for as a sensible default before hand-tuning a more restrictive, service-specific filter. systemd-analyze security my-app.service reports an actual, scored assessment of which hardening directives a given unit is missing — worth running directly against any production service as a concrete, actionable checklist rather than manually cross-referencing this chapter's own directive table by hand.

Drop-In Overrides — Modifying a Unit Without Editing It#

Worth a genuinely important, often-overlooked capability: rather than editing a unit file directly (risking the edit being overwritten by a future package upgrade), systemd supports drop-in override files — a targeted way to change or add specific directives without touching the original.

sudo systemctl edit my-app.service
# Opens an editor for a NEW file at:
# /etc/systemd/system/my-app.service.d/override.conf

# The resulting override.conf — ONLY the changed directive, not a full copy
[Service]
MemoryMax=1G

This is worth treating as the correct, standard practice for modifying any package-managed or distribution-provided unit file, worth stating the reasoning precisely: editing /etc/systemd/system/my-app.service directly (if it originated from a package) risks that edit being silently overwritten the next time the package updates and reinstalls its own unit file, while a drop-in override in the separate .service.d/ directory survives package upgrades untouched, since the package only manages the original file, never the override directory. systemctl edit --full my-app.service is the alternative for genuinely replacing an entire unit file rather than layering a targeted override — worth knowing both options exist, and defaulting to the targeted drop-in override for anything short of a complete rewrite.

systemd in Containers — a Genuine Nuance#

Worth a direct, honest connection back to Part 5's own container-networking material: running systemd itself INSIDE a container is a genuinely different, more nuanced topic than running systemd on a bare host, worth understanding precisely rather than assuming it works identically.

Most container images deliberately do NOT run systemd as their own PID 1 — a typical application container runs the application process directly as PID 1, precisely because a full systemd instance inside a container is genuinely heavier and more complex than most containerized workloads need, and because systemd's own cgroup-management expectations (this chapter's own "exclusive ownership" rule) can conflict with the outer host's own systemd instance without careful, explicit configuration. Running systemd inside a container is a real, legitimate pattern specifically for certain use cases (testing full-system behavior inside a container, or certain legacy application migration scenarios), but worth knowing as the deliberate exception requiring special container configuration, not the default assumption for how containerized applications should be built.

Debugging a Failing Service — a Practical Workflow#

# 1. Check the unit's own current status
systemctl status my-app.service

# 2. Read recent logs for this specific unit
journalctl -u my-app.service -n 50 --no-pager

# 3. Check for a failed dependency
systemctl list-dependencies my-app.service

# 4. Verify the unit file itself has no syntax errors
systemd-analyze verify my-app.service

# 5. Check WHY a unit failed to start (exit code, signal)
systemctl show my-app.service -p ExecMainStatus -p Result

systemd-analyze verify deserves specific mention as a genuinely underused, valuable check: it validates a unit file's own syntax and catches common configuration mistakes (a typo in a directive name, a reference to a non-existent dependency unit) BEFORE attempting to actually start the service — the same "validate before applying" discipline this series' own netfilter chapter recommended for nft -c -f, worth applying here as a standing habit before troubleshooting a failing service by trial and error, since a genuinely common cause of a mysteriously-failing unit is a simple, easily-caught syntax mistake in the unit file itself.

systemd-resolved — DNS Resolution Revisited#

Worth a direct callback to Part 2's own DNS material: systemd-resolved is systemd's own DNS resolution component, worth understanding as a genuine architectural change from the traditional model Part 2 covered.

The concrete, practical benefit worth stating precisely: systemd-resolved provides a local DNS cache (reducing real upstream query volume and latency for repeated lookups) and genuinely per-INTERFACE DNS configuration — a real, practical need for a host with multiple network interfaces (a VPN interface needing different DNS servers than a general internet-facing interface, for instance), which a single, flat, traditional /etc/resolv.conf has no clean way to express at all. resolvectl status (or the older systemd-resolve --status) shows the actual per-interface DNS configuration currently in effect — worth using directly when debugging a DNS resolution issue on a systemd-resolved-managed host, since a plain cat /etc/resolv.conf only shows the stub resolver address, not the real, currently-active upstream DNS servers resolved is actually using.

systemd-logind and Session Management#

Worth a brief, honest mention of a systemd component this chapter hasn't yet covered: systemd-logind manages user login sessions and their associated resource accounting — worth knowing exists specifically because of one common, confusing production interaction.

This is worth stating directly as a genuinely common source of confusion for anyone used to older init systems: on a host with logind's KillUserProcesses=yes (the default on several distributions), a background process started via nohup command & during an SSH session can still be killed when that SSH session ends, surprising an admin who assumed nohup alone was sufficient protection. The correct, robust fix for anything genuinely needing to survive a session end is not relying on nohup at all, but instead running it as a proper systemd unit (this chapter's entire subject) — or, at minimum, explicitly using systemd-run --scope to launch it outside the user's own login session scope entirely, avoiding this cleanup behavior by construction rather than hoping nohup alone is enough on a given distribution's specific logind configuration.

A Full Realistic Example: a Production-Hardened, Socket-Activated Service#

Tying nearly every mechanism from this chapter together into one complete, production-realistic unit definition — a socket-activated API service with resource limits, sandboxing, and a disciplined restart policy.

# /etc/systemd/system/api.socket
[Unit]
Description=Socket for the API service

[Socket]
ListenStream=8443

[Install]
WantedBy=sockets.target

# /etc/systemd/system/api.service
[Unit]
Description=Production API service
Requires=api.socket
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/api-server
User=api
Group=api

# Resource control (this chapter's own cgroup material)
CPUQuota=75%
MemoryMax=1G
MemoryHigh=800M

# Sandboxing (this chapter's own hardening material)
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/api

# Restart policy, with a circuit breaker
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

Worth reading this pair of files as the concrete, cumulative synthesis of this entire chapter: socket activation (zero idle resource cost), network-online.target dependency ordering (correct, not the common network.target mistake), cgroup-backed resource limits (a real ceiling and a softer throttle point), full sandboxing (container-grade isolation with zero container runtime involved), and a disciplined restart policy with an actual circuit breaker — every one of these mechanisms this chapter covered individually, now composed together into a single, genuinely production-realistic service definition. Type=notify (used here rather than the earlier examples' Type=simple) is worth a brief additional note: it requires the application itself to send an explicit "I'm actually ready" signal back to systemd via the sd_notify() protocol, letting systemd know precisely when startup has genuinely completed — more accurate than Type=simple's assumption that the process starting IS the same as the service being ready, useful for services with a real, non-trivial startup/warm-up sequence.

Watchdogs — Detecting a Hung, Not Just a Crashed, Process#

Worth a direct extension of this chapter's own restart-policy material: Restart=on-failure only catches a process that has actually exited — it does nothing for a process that's still technically running but has genuinely hung, deadlocked, or stopped making progress. systemd's watchdog mechanism closes exactly this gap.

[Service]
ExecStart=/usr/local/bin/my-app
WatchdogSec=30
Restart=on-watchdog

This is worth stating precisely, since it's a genuinely distinct failure mode from anything Restart=on-failure alone catches: a process that's deadlocked, stuck in an infinite loop, or otherwise unresponsive is still technically "running" from the kernel's own point of view — its exit code is irrelevant because it never exits — and Restart=on-failure has no signal to act on at all in this scenario. The watchdog mechanism requires the application itself to periodically call sd_notify(WATCHDOG=1) as an active "I'm still genuinely making progress" heartbeat — a real integration requirement on the application's own side, worth checking for before assuming watchdog-based hang detection is available for an arbitrary existing service, directly analogous to a Kubernetes liveness probe's own "prove you're actually healthy, not just alive" distinction covered in this course's Kubernetes series.

A systemd Change Checklist for Production Hosts#

Worth closing this chapter's practical guidance with one final, walkable checklist, directly extending the change-management discipline this series has built up across Part 4 and Part 5's own closing checklists.

This checklist deliberately mirrors the same shape as this series' own Part 4 and Part 5 closing checklists — worth treating all three as one continuous production-change discipline spanning firewall rules, virtual networking, and now service/unit management, rather than three unrelated processes.

Analyzing Boot Performance#

Worth a closing, practical observability connection: systemd provides genuine, built-in tooling for diagnosing slow boots — a real, if infrequent, production concern for anything sensitive to boot time (an autoscaled fleet where node startup latency directly affects scale-up responsiveness, per this course's own CI/CD & GitOps series' self-hosted runner chapter).

# Overall boot time breakdown
systemd-analyze

# Per-unit startup time, slowest first
systemd-analyze blame

# A visual, dependency-aware breakdown of the critical boot path
systemd-analyze critical-chain

critical-chain deserves specific emphasis over blame alone, since it answers a genuinely different, more actionable question: blame lists every unit's own individual startup time in isolation, while critical-chain shows the actual DEPENDENCY CHAIN that determined the overall boot time — a unit that took a long time to start but wasn't actually on the critical path (nothing else was blocked waiting on it) doesn't matter nearly as much for total boot latency as a much faster unit that happened to be sitting directly on the chain everything else depended on. This directly parallels this course's own CI/CD pipeline material — the same "slowest stage" versus "critical path" distinction already covered for pipeline stage ordering — worth recognizing as the identical performance-analysis principle, now applied to systemd's own boot sequence instead of a CI/CD pipeline's stages.

Common Mistakes#

Worked Practice Problems#

Problem 1: A team writes a .service unit with After=postgresql.service but no Wants= or Requires= directive, expecting their application to fail cleanly if PostgreSQL isn't running. Instead, their application starts anyway (and then crashes trying to connect to a database that was never started). What went wrong?

Answer: After= alone controls only ordering (WHEN the unit starts relative to another), not whether it starts AT ALL if the other unit fails or is absent. Without a paired Wants=postgresql.service (soft dependency) or Requires=postgresql.service (hard dependency), the application's own unit has no directive at all preventing it from starting even if postgresql.service never runs — the fix is adding the appropriate strength directive alongside the existing After=.

Problem 2: A service depends on network.target and works fine most of the time, but occasionally fails to connect to an external API immediately after boot, succeeding on a subsequent manual restart. Diagnose the likely cause.

Answer: network.target only guarantees the network management service itself has started, not that any actual interface has a usable IP address and real connectivity — the service can start before DHCP has actually completed, producing an intermittent, boot-timing-dependent failure. The fix is depending on network-online.target (paired with Wants=) instead, which specifically waits for interfaces to be genuinely up and usable before the dependent unit starts.

Problem 3: A platform engineer needs to increase a production service's memory limit but the unit file is owned and managed by the distribution's own package. What's the correct way to make this change so it survives the next package upgrade?

Answer: systemctl edit my-app.service, creating a drop-in override file at /etc/systemd/system/my-app.service.d/override.conf containing only the changed MemoryMax= directive — not editing the original package-managed unit file directly, which risks being silently overwritten the next time the package updates. The drop-in directory is never touched by the package's own files, so the override survives upgrades untouched.

Problem 4: A container runtime (containerd) running as a systemd-managed service needs to create and manage its own per-container cgroups, but systemd is supposed to have exclusive ownership of the cgroup v2 hierarchy. How is this conflict actually resolved?

Answer: Cgroup delegation — the containerd.service unit is marked Delegate=yes, which systemd-authorizes as an explicit hand-off of management authority over that unit's own cgroup sub-tree. This lets containerd create and manage its own per-container cgroups beneath that delegated point without violating the kernel's single-writer-per-subtree rule or systemd's own general exclusive-ownership policy — the conflict is resolved by explicit, sanctioned delegation rather than either side bypassing the other.

Problem 5: A team wants a rarely-used internal admin API to consume zero resources when idle, but still respond immediately to the very first request without any noticeable cold-start delay. Which systemd capability directly provides this, and how?

Answer: Socket activation — systemd creates and listens on the service's socket immediately at boot (so the very first connection is accepted immediately, with no delay), but doesn't actually start the service's own process until that first connection genuinely arrives. This gives exactly the described property: zero resource consumption while idle, and immediate responsiveness on first use, since the listening socket itself is already active before the process starts.

Problem 6: A service is genuinely still running (its process hasn't exited) but has deadlocked and stopped responding entirely. The team has Restart=on-failure configured, but systemd never restarts it. Why not, and what should they add?

Answer: Restart=on-failure only reacts to a process actually exiting with a failure condition — a deadlocked process that never exits produces no signal Restart=on-failure can act on at all, since from the kernel's own point of view the process is still technically alive. The fix is adding a watchdog (WatchdogSec= plus Restart=on-watchdog), which requires the application to periodically call sd_notify(WATCHDOG=1) as an active heartbeat — if that heartbeat stops arriving within the configured interval, systemd treats the hang itself as a failure and restarts the service, catching exactly this failure mode that exit-code-based restart policies cannot.

Problem 7: An admin SSHes into a production host, starts a long-running data migration script with nohup ./migrate.sh &, and disconnects, expecting it to keep running. They later discover it was killed when their SSH session ended. What's the likely cause, and what's the more robust fix?

Answer: KillUserProcesses=yes (systemd-logind's default on several distributions) kills all of a user's processes — including nohup-backgrounded ones — when their last login session ends, a real behavior nohup alone doesn't protect against on such a system. The more robust fix is running the script as a genuine systemd unit (a oneshot service, or via systemd-run --scope) rather than relying on nohup and a background shell job, since a proper systemd-managed process is not tied to any particular login session's lifecycle at all.

Problem 8: A platform team notices their fleet's average boot time increased significantly after a recent change, and systemd-analyze blame shows one particular unit taking the longest individual startup time. They fix that unit, but total boot time barely improves. What did they likely misdiagnose?

Answer: They optimized based on blame alone, which shows each unit's individual startup duration in isolation, not whether that unit was actually on the critical dependency path determining total boot time. A slow unit that nothing else was waiting on doesn't meaningfully affect overall boot time even if it's individually the slowest; the correct diagnostic tool is systemd-analyze critical-chain, which shows the actual chain of dependencies that determined the real, total boot duration — the fix should target whichever unit sits on that critical chain, not necessarily the single slowest unit in isolation.

Key Terms Glossary — This Chapter's Vocabulary in One Place#

Summary and What's Next#

systemd is PID 1 on nearly every modern Linux distribution — a broader suite (init core, journald, networkd, resolved, timers) rather than a single monolithic init process, unifying nearly every "coordinated, dependency-aware system task" (starting services, mounting filesystems, running scheduled jobs) under one consistent unit-based model. Dependency ordering (After=/Before=) and dependency strength (Wants=/Requires=) are genuinely separate concerns that need to be paired deliberately — Wants=+After= is the recommended default for most real service relationships, reserving the harder-failing Requires= for genuinely non-negotiable dependencies. Targets replace old-style runlevels as named synchronization points, with network-online.target specifically resolving a common, real boot-ordering gotcha that plain network.target doesn't address. Socket activation provides genuine scale-to-zero behavior for a single host-level service, directly parallel to the Kubernetes-scale scale-to-zero concepts this course's CI/CD & GitOps series covers at fleet scale. journald's structured, automatically-metadata-rich logging requires zero application instrumentation to gain real value, and integrates directly with this course's own centralized log-shipping material via its JSON export. systemd's exclusive, kernel-enforced ownership of the cgroup v2 hierarchy is what every [Service]-section resource-control directive (MemoryMax=, CPUQuota=) ultimately compiles down to, with cgroup delegation as the explicit, sanctioned mechanism letting a container runtime manage its own per-container cgroups without violating that ownership. systemd-networkd directly resolves Part 5's own closing gap — declarative, version-controllable, persistent network interface configuration, applied automatically on every boot using the identical underlying kernel mechanisms Part 5 covered by hand. Service sandboxing directives (ProtectSystem=, PrivateTmp=) apply real, container-grade isolation to a plain systemd unit using the same namespace primitives Part 5 covered directly, with zero container runtime required at all.

Part 7, immediately following, closes this series with eBPF — a genuinely different, increasingly significant kernel-level mechanism this series has already previewed twice (Part 4's own XDP preview, and Cilium's eBPF-based CNI dataplane referenced across both Parts 4 and 5) — covering eBPF's own programming model, bpftrace for ad-hoc kernel-level tracing, and exactly how Cilium builds a complete CNI dataplane on top of these same primitives.

Every mechanism this chapter has covered — units, dependencies, targets, cgroups, sandboxing — is worth recognizing as the same underlying discipline running throughout this entire series: explicit, declarative configuration, continuously enforced by the kernel and its userspace managers, rather than implicit, manually-maintained state a human has to remember to keep consistent by hand.