systemd · Software Engineering Wiki

Cheatsheet #

Task	Command
Is it running, why not	`systemctl status myapp`
Everything that failed	`systemctl --failed`
Start, stop, restart	`systemctl restart myapp`
Reload config, keep the process	`systemctl reload myapp`
Enable at boot	`systemctl enable --now myapp`
Follow logs	`journalctl -u myapp -f`
Logs since boot	`journalctl -u myapp -b`
Errors only, last hour	`journalctl -p err --since -1h`
Effective unit file	`systemctl cat myapp`
All properties	`systemctl show myapp`
Edit safely	`systemctl edit myapp`
After editing a unit	`systemctl daemon-reload`
What blocks boot	`systemd-analyze blame`
Dependency tree	`systemctl list-dependencies myapp`
Run a one-off sandboxed	`systemd-run --scope --unit=t1 -p MemoryMax=1G ./cmd`
Timer schedule	`systemctl list-timers --all`

A failing service #

status shows the last lines of the journal, the exit code and the main PID. The exit code and Result= line say what happened; the journal says why.

systemctl status myapp --no-pager -l
journalctl -u myapp -b --no-pager | tail -50
systemctl show myapp -p ExecMainStatus -p Result -p NRestarts
systemd-analyze verify /etc/systemd/system/myapp.service    # catches syntax and dependency errors

`Result=`	Meaning
`exit-code`	Process returned non-zero; read its logs
`signal`	Killed — `SIGKILL` here usually means a timeout or OOM
`timeout`	Exceeded `TimeoutStartSec` or `TimeoutStopSec`
`oom-kill`	Hit `MemoryMax` or the system ran out
`protocol`	`Type=notify` unit never called `sd_notify(READY=1)`
`exec-condition`	`ExecCondition=` returned a skip status

Active: activating (start) forever is almost always a Type= mismatch: a forking daemon declared Type=simple, or a simple daemon declared Type=notify without the notification.

Inspecting units #

systemctl list-units --type=service --state=running
systemctl list-unit-files --state=enabled
systemctl cat myapp                      # unit file plus every drop-in, in order
systemctl show myapp -p ExecStart -p User -p MemoryMax
systemctl list-dependencies myapp --reverse   # what depends on this
systemctl is-enabled myapp; systemctl is-active myapp
systemd-delta                            # overridden or masked units across the system

systemctl cat is the only reliable way to see what is actually in effect — a drop-in in /etc/systemd/system/myapp.service.d/ silently changes behaviour defined in /usr/lib.

Journal #

The journal is a structured, indexed binary store: every field is queryable, not just the message text.

journalctl -u myapp -f                       # follow
journalctl -u myapp --since '2 hours ago' --until '10 min ago'
journalctl -u myapp -b -1                    # previous boot
journalctl -p warning..err -b                # by priority range
journalctl _PID=1234 ; journalctl _UID=1000
journalctl -u myapp -o json-pretty | head -40   # all structured fields
journalctl -u myapp -g 'timeout|refused'     # grep, with the unit still filtered
journalctl --disk-usage; journalctl --vacuum-time=7d
journalctl -k                                # kernel ring buffer

Persistence requires /var/log/journal to exist (Storage=persistent in journald.conf); otherwise the journal is memory-backed and -b -1 returns nothing after a reboot.

Writing a unit #

[Unit]
Description=Ingest worker
Documentation=https://wiki.example.internal/ingest
After=network-online.target postgresql.service
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=notify
User=ingest
Group=ingest
WorkingDirectory=/opt/ingest
EnvironmentFile=-/etc/ingest/env
ExecStart=/opt/ingest/bin/worker --config /etc/ingest/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=30

# sandboxing: each line removes capability the service does not need
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/ingest
ProtectKernelTunables=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
CapabilityBoundingSet=

[Install]
WantedBy=multi-user.target

Setting	Effect
`Type=simple`	The process is the service; ready immediately (default)
`Type=exec`	Like simple, but start completes only after `execve` succeeds
`Type=forking`	Daemonises; needs `PIDFile=` to be tracked correctly
`Type=notify`	Waits for `sd_notify(READY=1)`; accurate ordering for dependants
`Type=oneshot`	Runs to completion; pair with `RemainAfterExit=yes` for state
`Restart=on-failure`	Restart on non-zero exit or signal, not on clean exit
`StartLimitBurst`/`IntervalSec`	Give up after N restarts in a window, leaving it `failed`
`After=` vs `Requires=`	Ordering versus dependency — you almost always need both
`EnvironmentFile=-`	Leading `-` makes a missing file non-fatal

After=network-online.target needs Wants=network-online.target as well, or nothing pulls the target in and the unit starts before addresses exist.

Check the sandbox you have actually achieved:

systemd-analyze security myapp        # scored exposure report, per setting

Overriding a packaged unit #

Never edit files in /usr/lib/systemd/system — a package update replaces them.

systemctl edit myapp                  # creates /etc/systemd/system/myapp.service.d/override.conf
systemctl edit --full myapp           # copy the whole unit to /etc for heavier changes
systemctl revert myapp                # discard overrides

# override.conf
[Service]
ExecStart=
ExecStart=/opt/ingest/bin/worker --config /etc/ingest/other.yaml
MemoryMax=2G

List-valued settings such as ExecStart= accumulate; clearing with an empty assignment first is required, and forgetting it produces “only one ExecStart= is allowed”.

systemctl daemon-reload && systemctl restart myapp

Timers #

A timer unit triggers a service unit of the same name. Timers survive downtime with Persistent=true, report failures through the normal unit machinery and inherit all the sandboxing — which is why they replace cron for anything that matters.

# backup.timer
[Unit]
Description=Nightly backup

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=300
Persistent=true
Unit=backup.service

[Install]
WantedBy=timers.target

systemctl enable --now backup.timer
systemctl list-timers --all                      # next and last run for each
systemd-analyze calendar 'Mon *-*-* 06:00:00'    # validate and show the next elapse
systemctl start backup.service                   # run it now, independent of the timer
journalctl -u backup.service --since today

OnCalendar=daily is midnight exactly, which is when every other job on the fleet also runs; RandomizedDelaySec spreads the load.

Resource control #

Every service runs in a cgroup, so limits are enforced by the kernel rather than by the process behaving.

[Service]
MemoryMax=2G              # hard limit: OOM kill above this
MemoryHigh=1.5G           # soft: reclaim pressure before the hard limit
CPUQuota=150%             # 1.5 cores
CPUWeight=50              # relative share under contention
IOWeight=50
TasksMax=512

systemd-cgtop                              # live resource use by unit
systemctl show myapp -p MemoryCurrent -p CPUUsageNSec
systemd-run --scope -p MemoryMax=1G -p CPUQuota=50% ./heavy-job   # limit an ad-hoc command

Boot problems #

systemd-analyze                     # total boot time by phase
systemd-analyze blame               # slowest units
systemd-analyze critical-chain      # what actually delayed the boot target
systemctl --failed
journalctl -b -p err
systemctl list-jobs                 # jobs still waiting, when boot hangs

A unit stuck in activating blocks anything ordered After= it. systemctl list-jobs names the one that is waiting, which is faster than reading the whole journal.

Oneliners #

# Services that failed, with the reason
systemctl --failed --no-legend | awk '{print $1}' | xargs -r -n1 -I{} sh -c 'echo "== {}"; systemctl show {} -p Result -p ExecMainStatus'

# Top memory consumers among services
systemctl show '*.service' -p Id -p MemoryCurrent --value | paste - - | sort -k2 -nr | head

# Everything a unit logged during its last start attempt
journalctl -u myapp --since "$(systemctl show myapp -p ActiveEnterTimestamp --value)"

# Units enabled but not running
comm -13 <(systemctl list-units --type=service --state=running --no-legend | awk '{print $1}' | sort) <(systemctl list-unit-files --state=enabled --no-legend | awk '{print $1}' | sort)

# Which unit owns a process
systemctl status $(pgrep -f worker | head -1)

# Restart count since boot
systemctl show myapp -p NRestarts --value

# Watch a unit's cgroup live
systemd-cgls /system.slice/myapp.service

# Run something now with the service's own environment
systemd-run --uid=ingest --same-dir --wait --pty /opt/ingest/bin/worker --check

# Mask a unit so nothing can start it
systemctl mask --now myapp

# Reload every changed unit file and restart the ones that changed
systemctl daemon-reload && systemctl reset-failed

# Journal fields available for a unit
journalctl -u myapp -o verbose -n1

# Find units with no sandboxing at all
systemd-analyze security --no-pager | sort -k2 -nr | head