Systemd is an init system and system manager in Linux systems and is compatible with LSB and SysV. You can use the systemd suite to manage and optimize system start-up services and resources in a Linux system. It is a practical tool for sysadmins to get their system up and running, optimize processes, debug, and troubleshoot system services.
This article will build on our systemd series guide and illustrate how to manage system Units, file systems mounts, troubleshoot, and give you tips and tricks when working with systems.
Our first systemd guide highlighted why systemd is a practical tool for Linux sysadmins. The second illustrates how to schedule system tasks with systemd timers and automate the tedious system boot tasks.
As a retro Linux user, I always got existed (still do) when the Fedora system boots and pages upon pages of diagnostic messages scroll by before the login prompt. These pages contain information on start-up processes being, file systems mounts, and much more. To understand how to manage and optimize start-up services, let us have a quick look at what transpires between when you hit the power-on button and the login prompt.
A boot process begins with hardware boot which initializes the system hardware, OS boot which loads the Kernel, then systemd, and Linux start-up, where systemd prepares the system’s processes. The start-up process begins when the Kernel transfers control of the host to the systemd. At this point, a sysadmin can manage services, Units, sockets, D-Bus activation, tracks processes, daemons, and file system mount points.
The following are some building blocks of the systemd software suite you can use to manage Linux start-up.
- systemd-boot – a UEFI boot manager.
- systemd-firstboot – manages basic system setting initialization before first boot.
- systemd-logind – a session management tool.
- systemd-networkd – manage network configurations.
- systemd-sysusers – a tool to create system user groups and add users to groups at boot time.
- systemd/Journal — manage system logging.
- systemd/Timers — timers for controlling .service files or events.
To manage systemd, use the systemctl command, which utilizes both the functionality of SysVinit’s service and chkconfig. You can use it to manage system units which are representations of system services and resources.
# systemctl daemon-reload
Systemd Unit management
Units in systemd can be services (.service), mount points (.mount), devices (.device) or sockets (.socket). Systemctl provides different commands to manage units.
Use the following command to analyze the system state:
$ systemctl status #Show system status $ systemctl or #List running units $ systemctl list-units #List running units $ systemctl --failed #List failed units $ systemctl list-unit-files #List installed unit files1 $ systemctl status pid #Show process status for a PID
Use the following commands to check unit status:
$ systemctl help unit #Show a manual page associated with a unit $ systemctl status unit #Status of a unit $ systemctl is-enabled unit #Check whether a Unit is enabled
Use the following commands to start and reload a unit:
systemctl start unit #start a unit immediately systemctl stop unit #stop a unit immediately systemctl restart unit #restart a unit systemctl reload unit #reload a unit and configurations systemctl daemon-reload #reload systemd manager configuration
Use the following commands to mask a Unit:
systemctl mask unit #mask a Unit to make it impossible to start systemctl unmask unit #Unmask a unit
Use the following commands to enable a unit:
systemctl enable Unit #enable a Unit to start automatically at boot systemctl enable --now unit #enable a Unit to start automatically and immediately at boot systemctl disable unit #disable a Unit to no longer start at boot systemctl reenable unit #disable and enable anew
Editing Unit Files
Linux loads Unit files from multiple locations in your system. You can run the [systemctl show –property=UnitPath] command to see the complete list.
- /usr/lib/systemd/system/: units provided by installed packages.
- /etc/systemd/system/: units installed by the sysadmin.
Example unit file:
/etc/systemd/system/unit.d/example_unit.conf [Unit] Requires=new dependency After=new dependency
You can resolve Unit dependencies by designing the unit files correctly.
For example, if unit A requires unit B to be running before A is started. Then, add Requires=B and After=B to the [Unit] section of A.
# /etc/systemd/system/unit.d/example_unit.conf [Unit] Requires=B After=B
If the dependency is optional, add Wants=B and After=B.
/etc/systemd/system/unit.d/example_unit.conf [Unit] Requires=B After=B Wants=B
Note: Dependencies are placed on services and not on targets.
You can set different start-up service types in a custom service file with the Type= parameter in the [Service] section:
/etc/systemd/system/unit.d/example_unit.conf [Unit] Requires=B After=B Wants=B [Service] Type=simple
- Type=simple (default): systemd considers the service to start up immediately.
- Type=forking: systemd considers the service started up once the process forks and the parent has exited.
- Type=oneshot: You can use it for scripts that do a single job and then exit. You can set RemainAfterExit=yes so that systemd can still consider the service active after the process has exited.
- Type=idle: systemd will delay the execution of the service binary until all jobs are dispatched.
- Type=notify: You can use it like Type=simple, but the daemon will signal the systemd when it is ready.
- Type=dbus: the service is considered ready when the specified BusName appears on DBus’s system bus.
Replacement unit files
You can replace a unit file in [/usr/lib/systemd/system/] by creating a new unit file with a similar name and reenable the Unit to update the symlinks.
# systemctl reenable Unit
Alternatively, you can run the [# systemctl edit –full Unit] command that opens the unit file in your editor and automatically reloads it when you finish editing.
# systemctl edit --full Unit
You can create a drop-in unit file by creating the directory /etc/systemd/system/unit.d/ and place your new .conf file. The file will override or add new configuration options. systemd will parse and apply these files on top of the original Unit file.
Alternatively, run the following command [# systemctl edit unit] to open the file /etc/systemd/system/unit.d/new_override.conf in a text editor and automatically reloads the Unit file.
Revert changes of a unit file
Use the following command to revert any changes you made to a unit using the systemctl edit command.
# systemctl revert unit
Add additional dependency to a Unit
/etc/systemd/system/unit.d/newcustomdependency.conf [Unit] Requires=new customdependency After=new customdependency
Systemd provides different commands to reboot or power off your system.
systemctl reboot #shut down and reboot the system systemctl poweroff #shut down and power-off the system systemctl suspend #suspend the system systemctl hibernate #put the system into hibernation systemctl hybrid-sleep #put the system into hybrid-sleep state
Mount file systems and partitions
systemd is in charge of mounting the partitions and filesystems specified in /etc/fstab. You can manage or mount a file system by defining all the required parameters in a unit file. Include details of the filesystem and mount point. Systemd gives you more flexibility while working with mount units. It uses the /etc/fstab file for filesystem configuration and mounting. The process involves using the systemd-fstab-generator tool to create mount units from the data in the fstab file.
Create a systemd mount unit
The illustration is on Fedora 33 running a btrfs filesystem.
Verify that you have free space on the volume group.
List available mount unit files:
[root@foss]# systemctl list-unit-files -t mount OR [root@foss]# systemctl status *mount
Create a systemd .mount unit file:
Check the filesystem UUID using the blkid command.
[root@foss]# blkid /dev/sda2 /dev/sda2: LABEL="fedoraworkstation33" UUID="688a6af2-xxx-4da4-xxx-878c5b0f063b" UUID_SUB="690a86e7-yyyy-9a92-4bc6-49cca4yyyy8df" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="0byyyb88-02"
Create a new file [var-lib-docker.mount] in the etc/systemd/system directory. Add configuration data below. Note that the unit file name and mount point must be identical.
# vi /etc/systemd/system/var-lib-docker.mount [Unit] Description=docker mount [Mount] What=/dev/disk/by-uuid/688a6af2-77e1-4da4-bc63-878c5b0f063b Where=/var/lib/docker Type=btrfs Options=defaults [Install] WantedBy=multi-user.target
Note that the “what” argument can take UUID, LABEL, and path to the disk.
Description in the [Unit] section provides the mount name, displaying with the systemctl -t mount. The configuration data in the [Mount] section contains the same data in the fstab file.
Enable the Mount Unit to start after boot:
[root@foss]# systemctl enable var-lib-docker.mount Created symlink /etc/systemd/system/multi-user.target.wants/var-lib-docker.mount → /etc/systemd/system/var-lib-docker.mount.
The command creates a symlink in the /etc/systemd/system directory, enabling the mount Unit to be mounted on all subsequent boots.
Start and mount the filesystem:
# systemctl start var-lib-docker.mount
Verify that the filesystem has been mounted:
# systemctl status var-lib-docker.mount ● var-lib-docker.mount - Docker mount Loaded: loaded (/etc/systemd/system/var-lib-docker.mount; enabled; vendor preset: disabled) Active: inactive (dead) Where: /var/lib/docker What: /dev/disk/by-uuid/688a6af2-77e1-4da4-bc63-878c5b0f063b
You can also modify the docker service unit file to ensure the service only starts after the mount service is called.
# cat /usr/lib/systemd/system/docker.service [Unit] Description=docker service unit file After=network.target var-lib-docker.mount # Added mount unit for docker service to wait Requires=docker.socket
Reboot and check the status of the mount service unit.
# systemctl reboot # systemctl status var-lib-docker.mount
Check for the [var-lib-docker] mount point:
# mount | grep var-lib-docker
Tips and tricks
Run a service after the network is up
You can delay a service from starting until after the network is up and running by adding the following dependencies to your .service file.
/etc/systemd/system/test_foo.service [Unit] Wants=network-online.target After=network-online.target
Moreover, you can add the nss-lookup.target if a service needs to perform DNS queries.
/etc/systemd/system/test_foo.service [Unit] Wants=network-online.target After=network-online.target nss-lookup.target ...
Use the following command to check which service is pulling in nss-lookup.target.
# systemctl list-dependencies --reverse nss-lookup.target
Install systemd GUI configuration tools
You can also work with systemd using the following GUI tools.
- SystemdGenie – is a systemd management tool based on KDE.
- Systemadm – is a graphical browser for systemd units.
Systemd offers fast boot times of <2s for up-to-date desktop environments. However, we could optimize it further without having to write any code with the following steps:
- First, consider bypassing the initrd if you use one in your system.
- Consider disabling SELinux and auditing by adding selinux=0 on the kernel command-line. However, note that sysadmins recommend leaving SElinux on for security reasons.
- Consider uninstalling Syslog and use journal instead. Journal is the default logging tool in newer systemd systems.
- If your console output is slow, use the quiet flag on the command line and disable system debug logging.
- Consider removing cron and use systemd timers instead.
- Use a modern desktop environment like GNOME 40 that does not pull in ConsoleKit.
- Check and disable any unnecessary boot processes or services. A system boot gets faster if you start fewer processes at boot-up.
- Please get rid of shell-based services like SysV init scripts and replace them with unit files.
- Avoid using Type=forking and ordering dependencies. Instead, replace them with socket activation and Type=simple whenever possible. It will allow a better-parallelized start-up for services.
Investigate failed services
Use the following command to find systemd services that failed to start:
# systemctl --state=failed
Diagnosing a service
You can also get more troubleshooting information about a service by setting the SYSTEMD_LOG_LEVEL environment variable to debug.
For example, to run the systemd-networkd daemon in debug mode, add a drop-in file for the service and the following additional configurations.
Alternatively, you can set the environment variable manually with the following command:
# SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
After completing your setting, restart the service and monitor the service journal with the -f/–follow option to view all the logs.
System logs stores a history of system activities such as boot-up time, when services launched, system jobs, background services, failed activities, and much more. Systemd maintains a “catalog” of errors, messages, possible solutions and highlights critical contexts in the logs messages which could go unnoticed. You can review systemd logs with the journalctl command.
$ journalctl --pager-end
The flag –pager-end starts your log review at the end of the journalctl output.
- Systemd.unit manual page.
- Fedora project’s guide to systems.
- Description of systemd from Freedesktop.org.
- Systemd ArchWiki.
systemd provides a robust way of managing Linux start-up with systemd units. The article has highlighted different ways to use the systemctl command to edit unit files, manage systemd units. It has highlighted how to create a new systemd mount unit to mount a new filesystem and enable it to initiate during start-up. Finally, I have shared some systemd optimization tips and how to troubleshoot failed services.