I’m sorry Martin, I’m afraid I can’t do that [boot, because your systemd logs are too large]

The Problem

Today I solved a problem that was bugging me the past few days: my Linux machine would not boot anymore and instead just pause with no apparent reason.

I could switch to another virtual terminal and login there, do stuff, update my machine, anything restricted to the terminal.

But trying a sudo systemctl restart sddm.service to get to the display mananager did not work and no error was displayed.

The first thing to do in such cases is looking at the log files. They are stored in /var/log/ and you can see the content of the folder with, for example, ls -l /var/log/:

total 3192
-rw-------  1 root utmp                768 Jun 25  2016 btmp
-rw-------  1 root utmp                  0 Jan  1  2016 btmp.1
drwxr-xr-x  2 root root               4096 Jan  3  2016 cups
-rw-------  1 root root              32032 Apr  3 20:34 faillog
drwxr-sr-x+ 4 root systemd-journal    4096 Aug  9  2015 journal
-rw-r--r--  1 root root             292292 Apr  3 19:59 lastlog
drwxr-xr-x  2 root root               4096 Feb 15  2015 old
-rw-r--r--  1 root root            1527815 Apr  3 20:23 pacman.log
drwxr-xr-x  3 root root               4096 Jan 19 11:50 samba
drwxr-xr-x  2 root root               4096 Jul 23  2015 speech-dispatcher
-rw-rw-r--  1 root utmp            1473024 Apr  3 20:44 wtmp
-rw-r--r--  1 root root              58495 Apr  3 20:34 Xorg.0.log
-rw-r--r--  1 root root              51511 Apr  3 19:59 Xorg.0.log.old
-rw-r--r--  1 root root              47470 Sep 22  2016 Xorg.1.log

I did not find anything out of the ordinary in the log files. Then I thought that maybe this is one of those situations that are resolved via an update. But when the error persisted and no package was being updated that could be the culprit I knew that I had to fix it by myself.

The first hint

When the command systemctl status returned State: degraded and showed me that four units failed I had something I could search for.

● Host
State: degraded
Jobs: 0 queued
Failed: 4 units
Since: Mon 2017-04-03 19:56:51 CEST; 8min ago

Soon enough, searching for systemd state degraded yielded—nothing helpful. Then I looked at the systemd-specific files.

The folder /var/log/journal contains all log files which are managed by systemd. Instead of viewing those files with less or tail you use journalctlto look at the content. I noticed that I had a lot of data in the logs: They were going back to 2015! This was also the reason that the folder contained over 1G of data. The files were actually so large that there was no space left on the partition:

systemd-journald[182]: System journal (/var/log/journal/8c5[...]) is 1.2G, max 1.2G, 0B free.

I decided to do a little spring cleaning.

The solution

Fortunately, my next search immediately brought me to unix.stackexchange.com and this question about clearing journalctl. Old data can either be removed by size or time:

journalctl --vacuum-size=100M
journalctl --vacuum-time=10d

Another answer explained how to configure a limit for systemd. The option SystemMaxUse= can be set in /etc/systemd/journald.conf.

When I started SDDM again it started as usual. A final sudo systemctl reset-failed to start all the failed units and I was good to go.

Title image by Samuel Zeller.

Martin Rüßler Written by:

Martin likes to write about R