How to manage systemd units at start-up

954

Systemd is an init system and system manager in Linux systems and is compatible with LSB and SysV. You can use the systemd suite to manage and optimize system start-up services and resources in a Linux system. It is a practical tool for sysadmins to get their system up and running, optimize processes, debug, and troubleshoot system services.

This article will build on our systemd series guide and illustrate how to manage system Units, file systems mounts, troubleshoot, and give you tips and tricks when working with systems.

Our first systemd guide highlighted why systemd is a practical tool for Linux sysadmins. The second illustrates how to schedule system tasks with systemd timers and automate the tedious system boot tasks.

Linux start-up

As a retro Linux user, I always got existed (still do) when the Fedora system boots and pages upon pages of diagnostic messages scroll by before the login prompt. These pages contain information on start-up processes being, file systems mounts, and much more. To understand how to manage and optimize start-up services, let us have a quick look at what transpires between when you hit the power-on button and the login prompt.

A boot process begins with hardware boot which initializes the system hardware, OS boot which loads the Kernel, then systemd, and Linux start-up, where systemd prepares the system’s processes. The start-up process begins when the Kernel transfers control of the host to the systemd. At this point, a sysadmin can manage services, Units, sockets, D-Bus activation, tracks processes, daemons, and file system mount points.

Systemd components

The following are some building blocks of the systemd software suite you can use to manage Linux start-up.

• systemd-boot – a UEFI boot manager.
• systemd-firstboot – manages basic system setting initialization before first boot.
• systemd-logind – a session management tool.
• systemd-networkd – manage network configurations.
• systemd-sysusers  – a tool to create system user groups and add users to groups at boot time.
• systemd/Journal — manage system logging.
• systemd/Timers — timers for controlling .service files or events.

Systemctl

To manage systemd, use the systemctl command, which utilizes both the functionality of SysVinit’s service and chkconfig. You can use it to manage system units which are representations of system services and resources.

# systemctl daemon-reload

Systemd Unit management

Units in systemd can be services (.service), mount points (.mount), devices (.device) or sockets (.socket). Systemctl provides different commands to manage units.

Use the following command to analyze the system state:

$ systemctl status #Show system status 
$ systemctl or #List running units
$ systemctl list-units #List running units
$ systemctl --failed #List failed units
$ systemctl list-unit-files #List installed unit files1
$ systemctl status pid #Show process status for a PID

Use the following commands to check unit status:

$ systemctl help unit #Show a manual page associated with a unit
$ systemctl status unit #Status of a unit
$ systemctl is-enabled unit #Check whether a Unit is enabled

Use the following commands to start and reload a unit:

systemctl start unit  #start a unit immediately
systemctl stop unit   #stop a unit immediately
systemctl restart unit   #restart a unit
systemctl reload unit  #reload a unit and configurations
systemctl daemon-reload   #reload systemd manager configuration

Use the following commands to mask a Unit:

systemctl mask unit     #mask a Unit to make it impossible to start
systemctl unmask unit     #Unmask a unit

Use the following commands to enable a unit:

systemctl enable Unit   #enable a Unit to start automatically at boot
systemctl enable --now unit   #enable a Unit to start automatically and immediately at boot
systemctl disable unit   #disable a Unit to no longer start at boot
systemctl reenable unit     #disable and enable anew

Editing Unit Files

Linux loads Unit files from multiple locations in your system. You can run the [systemctl show –property=UnitPath] command to see the complete list.

•  /usr/lib/systemd/system/: units provided by installed packages.
•  /etc/systemd/system/: units installed by the sysadmin.

Example unit file:

/etc/systemd/system/unit.d/example_unit.conf
[Unit]
Requires=new dependency
After=new dependency

Handling dependencies

You can resolve Unit dependencies by designing the unit files correctly.
For example, if unit A requires unit B to be running before A is started. Then, add Requires=B and After=B to the [Unit] section of A.

# /etc/systemd/system/unit.d/example_unit.conf
[Unit]
Requires=B
After=B

If the dependency is optional, add Wants=B and After=B.

/etc/systemd/system/unit.d/example_unit.conf
[Unit]
Requires=B
After=B
Wants=B

Note: Dependencies are placed on services and not on targets.

Service types

You can set different start-up service types in a custom service file with the Type= parameter in the [Service] section:

/etc/systemd/system/unit.d/example_unit.conf
[Unit]
Requires=B
After=B
Wants=B
[Service]
Type=simple

• Type=simple (default): systemd considers the service to start up immediately.
• Type=forking: systemd considers the service started up once the process forks and the parent has exited.
• Type=oneshot: You can use it for scripts that do a single job and then exit. You can set RemainAfterExit=yes so that systemd can still consider the service active after the process has exited.
• Type=idle: systemd will delay the execution of the service binary until all jobs are dispatched.
• Type=notify: You can use it like Type=simple, but the daemon will signal the systemd when it is ready.
• Type=dbus: the service is considered ready when the specified BusName appears on DBus’s system bus.

Replacement unit files

You can replace a unit file in [/usr/lib/systemd/system/] by creating a new unit file with a similar name and reenable the Unit to update the symlinks.

# systemctl reenable Unit

Alternatively, you can run the [# systemctl edit –full Unit] command that opens the unit file in your editor and automatically reloads it when you finish editing.

# systemctl edit --full Unit

Drop-in files

You can create a drop-in unit file by creating the directory /etc/systemd/system/unit.d/ and place your new .conf file. The file will override or add new configuration options. systemd will parse and apply these files on top of the original Unit file.

Alternatively, run the following command [# systemctl edit unit] to open the file /etc/systemd/system/unit.d/new_override.conf in a text editor and automatically reloads the Unit file.

Revert changes of a unit file

Use the following command to revert any changes you made to a unit using the systemctl edit command.

# systemctl revert unit

Add additional dependency to a Unit

/etc/systemd/system/unit.d/newcustomdependency.conf
[Unit]
Requires=new customdependency
After=new customdependency

Power management

Systemd provides different commands to reboot or power off your system.

systemctl reboot     #shut down and reboot the system
systemctl poweroff   #shut down and power-off the system
systemctl suspend     #suspend the system
systemctl hibernate    #put the system into hibernation
systemctl hybrid-sleep   #put the system into hybrid-sleep state

Mount file systems and partitions

systemd is in charge of mounting the partitions and filesystems specified in /etc/fstab. You can manage or mount a file system by defining all the required parameters in a unit file. Include details of the filesystem and mount point. Systemd gives you more flexibility while working with mount units. It uses the /etc/fstab file for filesystem configuration and mounting. The process involves using the systemd-fstab-generator tool to create mount units from the data in the fstab file.

Create a systemd mount unit

The illustration is on Fedora 33 running a btrfs filesystem.
Verify that you have free space on the volume group.

# lsblk

List available mount unit files:

[root@foss]# systemctl list-unit-files -t mount
OR
[root@foss]# systemctl status *mount

Create a systemd .mount unit file:

Check the filesystem UUID using the blkid command.

[root@foss]# blkid /dev/sda2
/dev/sda2: LABEL="fedoraworkstation33" UUID="688a6af2-xxx-4da4-xxx-878c5b0f063b" UUID_SUB="690a86e7-yyyy-9a92-4bc6-49cca4yyyy8df" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="0byyyb88-02"

Create a new file [var-lib-docker.mount] in the etc/systemd/system directory. Add configuration data below. Note that the unit file name and mount point must be identical.

# vi /etc/systemd/system/var-lib-docker.mount
[Unit]
Description=docker mount
[Mount]
What=/dev/disk/by-uuid/688a6af2-77e1-4da4-bc63-878c5b0f063b
Where=/var/lib/docker
Type=btrfs
Options=defaults
[Install]
WantedBy=multi-user.target

Note that the “what” argument can take UUID, LABEL, and path to the disk.

Description in the [Unit] section provides the mount name, displaying with the systemctl -t mount. The configuration data in the [Mount] section contains the same data in the fstab file.

Enable the Mount Unit to start after boot:

[root@foss]# systemctl enable var-lib-docker.mount
Created symlink /etc/systemd/system/multi-user.target.wants/var-lib-docker.mount → /etc/systemd/system/var-lib-docker.mount.

The command creates a symlink in the /etc/systemd/system directory, enabling the mount Unit to be mounted on all subsequent boots.

Start and mount the filesystem:

# systemctl start var-lib-docker.mount

Verify that the filesystem has been mounted:

# systemctl status var-lib-docker.mount
● var-lib-docker.mount - Docker mount
Loaded: loaded (/etc/systemd/system/var-lib-docker.mount; enabled; vendor preset: disabled)
Active: inactive (dead)
Where: /var/lib/docker
What: /dev/disk/by-uuid/688a6af2-77e1-4da4-bc63-878c5b0f063b

You can also modify the docker service unit file to ensure the service only starts after the mount service is called.

# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=docker service unit file
After=network.target var-lib-docker.mount # Added mount unit for docker service to wait
Requires=docker.socket

Reboot and check the status of the mount service unit.

# systemctl reboot
# systemctl status var-lib-docker.mount

Check for the [var-lib-docker] mount point:

# mount | grep var-lib-docker

Tips and tricks

Run a service after the network is up

You can delay a service from starting until after the network is up and running by adding the following dependencies to your .service file.

/etc/systemd/system/test_foo.service
[Unit]
Wants=network-online.target
After=network-online.target

Moreover, you can add the nss-lookup.target if a service needs to perform DNS queries.

/etc/systemd/system/test_foo.service
[Unit]
Wants=network-online.target
After=network-online.target nss-lookup.target
...

Use the following command to check which service is pulling in nss-lookup.target.

# systemctl list-dependencies --reverse nss-lookup.target

Install systemd GUI configuration tools

You can also work with systemd using the following GUI tools.

• SystemdGenie – is a systemd management tool based on KDE.
• Systemadm – is a graphical browser for systemd units.

systemd optimizations

Systemd offers fast boot times of <2s for up-to-date desktop environments. However, we could optimize it further without having to write any code with the following steps:

•  First, consider bypassing the initrd if you use one in your system.
• Consider disabling SELinux and auditing by adding selinux=0 on the kernel command-line. However, note that sysadmins recommend leaving SElinux on for security reasons.
• Consider uninstalling Syslog and use journal instead. Journal is the default logging tool in newer systemd systems.
• If your console output is slow, use the quiet flag on the command line and disable system debug logging.
• Consider removing cron and use systemd timers instead.
• Use a modern desktop environment like GNOME 40 that does not pull in ConsoleKit.
• Check and disable any unnecessary boot processes or services. A system boot gets faster if you start fewer processes at boot-up.
• Please get rid of shell-based services like SysV init scripts and replace them with unit files.
• Avoid using Type=forking and ordering dependencies. Instead, replace them with socket activation and Type=simple whenever possible. It will allow a better-parallelized start-up for services.

Troubleshooting

Investigate failed services

Use the following command to find systemd services that failed to start:

# systemctl --state=failed

Diagnosing a service

You can also get more troubleshooting information about a service by setting the SYSTEMD_LOG_LEVEL environment variable to debug.
For example, to run the systemd-networkd daemon in debug mode, add a drop-in file for the service and the following additional configurations.

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

Alternatively, you can set the environment variable manually with the following command:

# SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd

After completing your setting, restart the service and monitor the service journal with the -f/–follow option to view all the logs.

Systemd logs

System logs stores a history of system activities such as boot-up time, when services launched, system jobs, background services, failed activities, and much more. Systemd maintains a “catalog” of errors, messages, possible solutions and highlights critical contexts in the logs messages which could go unnoticed. You can review systemd logs with the journalctl command.

$ journalctl --pager-end

The flag –pager-end starts your log review at the end of the journalctl output.

Additional resources

1. Systemd.unit manual page.
2. Fedora project’s guide to systems.
3. Description of systemd from Freedesktop.org.
4. Systemd ArchWiki.

Wrapping up

systemd provides a robust way of managing Linux start-up with systemd units. The article has highlighted different ways to use the systemctl command to edit unit files, manage systemd units. It has highlighted how to create a new systemd mount unit to mount a new filesystem and enable it to initiate during start-up. Finally, I have shared some systemd optimization tips and how to troubleshoot failed services.