# Linux Troubleshooting Cheat Sheet — systemd & journalctl

> **Tool:** systemd (systemctl, journalctl)
> **Category:** Linux & Troubleshooting
> **Verified against:** systemd 255 (255.4-1ubuntu8.17), flags verified via `systemctl --help`, `journalctl --help`, and `man systemctl`, 2026-08-21
> **Official docs:** https://www.man7.org/linux/man-pages/man1/systemctl.1.html, https://www.man7.org/linux/man-pages/man1/journalctl.1.html

Managing services and reading their logs — the first two tools for "why did this service stop / why won't it start."

## Service status and control

```bash
systemctl status nginx
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx                        # re-read config without a full restart, if the service supports it
```

`reload` only works if the service's unit defines an `ExecReload` — many services fall back to a full restart if it doesn't, so check `systemctl status` output for whether reload actually did what you expected.

## Enabling and disabling on boot

```bash
systemctl enable nginx                         # start automatically on boot
systemctl disable nginx
systemctl is-enabled nginx
systemctl enable --now nginx                    # enable AND start in one command
```

`enable`/`disable` control boot-time startup; `start`/`stop` control the running state right now — they're independent. A service can be enabled but currently stopped, or running but not enabled (won't survive a reboot). A common on-call mistake is `systemctl stop` during an incident, without also `disable`, then being surprised it's back after the next reboot/deploy.

## Listing and filtering units

```bash
systemctl list-units --type=service --state=running
systemctl list-units --state=failed             # everything currently in a failed state
systemctl is-active nginx
systemctl is-failed nginx
```

## Masking a unit (stronger than disable)

```bash
systemctl mask nginx      # prevents starting even manually or as a dependency of another unit
systemctl unmask nginx
```

## Editing a unit with a drop-in

`systemctl edit` creates an override on top of the vendor-supplied unit instead of modifying it directly, so a package update never silently clobbers your customization.

```bash
systemctl edit nginx                              # opens $EDITOR on a drop-in override.conf for the unit
systemctl edit --full nginx                        # edit a full copy of the unit instead of a drop-in
systemctl edit --drop-in=timeout.conf nginx         # use a named drop-in file instead of the default override.conf
systemctl edit --force nginx-custom.service          # create the unit if it doesn't exist yet
systemctl cat nginx                                   # show the effective unit file plus every drop-in applied to it
```

Saving in `systemctl edit` automatically reloads systemd config (equivalent to `daemon-reload`), but it does **not** restart the unit — a running service keeps running on its old config until you `systemctl restart` it.

## Working with systemd timers

`.timer` units are the systemd-native alternative to cron entries — they can fire on a fixed schedule (`OnCalendar=`) or relative to boot/last-run (`OnBootSec=`/`OnUnitActiveSec=`), and each run is captured in `systemctl status`/`journalctl` like any other unit.

```bash
systemctl list-timers                              # every active timer: next/last run time and its target unit
systemctl list-timers --all                          # include inactive/disabled timers too
systemctl status backup.timer                          # a timer's own status (separate from the service it triggers)
systemctl start backup.service                            # manually fire the timer's target service now, bypassing the schedule
```

A timer unit (`backup.timer`) and the unit it triggers (`backup.service`) are two separate units — `systemctl status backup.timer` only shows the schedule; check `backup.service`'s own status/logs to see what happened during the actual run.

## Reloading systemd after editing a unit file

```bash
systemctl daemon-reload
```

Required any time you hand-edit a `.service` file — without it, systemd keeps using its in-memory copy of the old unit definition, and a `restart` will silently apply the *old* config.

## Reading logs for a specific service

```bash
journalctl -u nginx
journalctl -u nginx -f                          # follow/stream live
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2026-08-19 09:00" --until "2026-08-19 10:00"
journalctl -u nginx -n 100                       # last 100 lines
```

## Filtering and searching logs

```bash
journalctl -p err                                # priority filter: emerg/alert/crit/err/warning/notice/info/debug
journalctl -g "connection refused"                # grep the message field
journalctl -b                                     # logs since the current boot only
journalctl -k                                      # kernel messages (dmesg equivalent, from the journal)
journalctl -o json-pretty -u nginx -n 5            # structured output for piping into another tool
```

## Managing journal disk usage

```bash
journalctl --disk-usage
journalctl --vacuum-time=7d          # delete journal entries older than 7 days
journalctl --vacuum-size=500M         # shrink the journal to under 500MB
```

An unbounded journal is a real, recurring cause of a host quietly running out of disk — `--vacuum-time`/`--vacuum-size` are the direct fix; the durable fix is setting `SystemMaxUse=` in `/etc/systemd/journald.conf` so it never grows unbounded in the first place.