The Problem
Today I solved a problem that was bugging me the past few days: my Linux machine would not boot anymore and instead just pause with no apparent reason.
I could switch to another virtual terminal and login there, do stuff, update my machine, anything restricted to the terminal.
But trying a sudo systemctl restart sddm.service to get to the display mananager did not work and no error was displayed.
The first thing to do in such cases is looking at the log files.
They are stored in /var/log/ and you can see the content of the folder with, for example, ls -l /var/log/:
total 3192
-rw------- 1 root utmp 768 Jun 25 2016 btmp
-rw------- 1 root utmp 0 Jan 1 2016 btmp.1
drwxr-xr-x 2 root root 4096 Jan 3 2016 cups
-rw------- 1 root root 32032 Apr 3 20:34 faillog
drwxr-sr-x+ 4 root systemd-journal 4096 Aug 9 2015 journal
-rw-r--r-- 1 root root 292292 Apr 3 19:59 lastlog
drwxr-xr-x 2 root root 4096 Feb 15 2015 old
-rw-r--r-- 1 root root 1527815 Apr 3 20:23 pacman.log
drwxr-xr-x 3 root root 4096 Jan 19 11:50 samba
drwxr-xr-x 2 root root 4096 Jul 23 2015 speech-dispatcher
-rw-rw-r-- 1 root utmp 1473024 Apr 3 20:44 wtmp
-rw-r--r-- 1 root root 58495 Apr 3 20:34 Xorg.0.log
-rw-r--r-- 1 root root 51511 Apr 3 19:59 Xorg.0.log.old
-rw-r--r-- 1 root root 47470 Sep 22 2016 Xorg.1.logI did not find anything out of the ordinary in the log files. Then I thought that maybe this is one of those situations that are resolved via an update. But when the error persisted and no package was being updated that could be the culprit I knew that I had to fix it by myself.
The first hint
When the command systemctl status returned State: degraded and showed me that four units failed I had something I could search for.
● Host
State: degraded
Jobs: 0 queued
Failed: 4 units
Since: Mon 2017-04-03 19:56:51 CEST; 8min agoSoon enough, searching for systemd state degraded yielded—nothing helpful.
Then I looked at the systemd-specific files.
The folder /var/log/journal contains all log files which are managed by systemd.
Instead of viewing those files with less or tail you use journalctlto look at the content.
I noticed that I had a lot of data in the logs:
They were going back to 2015!
This was also the reason that the folder contained over 1G of data.
The files were actually so large that there was no space left on the partition:
systemd-journald[182]: System journal (/var/log/journal/8c5[...]) is 1.2G, max 1.2G, 0B free.I decided to do a little spring cleaning.
The solution
Fortunately, my next search immediately brought me to unix.stackexchange.com and this question about clearing journalctl.
Old data can either be removed by size or time:
journalctl --vacuum-size=100M
journalctl --vacuum-time=10dAnother answer explained how to configure a limit for systemd.
The option SystemMaxUse= can be set in /etc/systemd/journald.conf.
When I started SDDM again it started as usual.
A final sudo systemctl reset-failed to start all the failed units and I was good to go.
Title image by Samuel Zeller.
