Explorations into Angstrom syslog and systemd

July 10th, 2012

One basic thing I like to have running in any system is system level logging. This is very important when it comes to debugging problems, especially on embedded systems like the Beaglebone. The main objectives I had were:
1. Application level logging with multiple levels
2. Service/daemon based logging. This makes sure if the app crashes that we still get the last bits of log output
3. Non-volatile storage and log file rotation

In Linux systems, this is typically handled by the syslog function and syslogd. Looking into what was supported on the Beaglebone, I decided that syslog-ng or Busybox’s syslog functionality would work for my needs. The weird part that I noticed was that when I called syslog() from my application, there was no log file output. Very strange. Running ps -A showed that syslogd and klogd were not running. Going to the /etc/init.d and running ./syslog start would happily start the daemons and magically my log output would show up in /var/log/messages (Given that I was running the Busybox syslog daemon instead of syslog-ng). So, the question was, why was the system not starting syslog on initialization, and how do I start it?

After struggling with this for a couple of days, I check the Google groups for the Beagleboard and found only a few entries such as the following.  In summary, system initialization is now handled by systemd.  (you can find the man pages here)  Don’t bother with trying to find your rc.local, adding entries to /etc/init.d, or editing the rc0-6.d files found under /etc.  None of these options work.  If you want to start services/daemons, you have to use systemd, since this is the only method supported by Angstrom.

Digging further, I found this little tidbit of knowledge.   For Angstrom/Debian, you should note the following:
1. Syslog is now supported by the systemd journal
2. The journal is stored under /run/log/journal.  The actual file is found under a crazy directory called something like 3dcac98d32a34633b2e9d755150dad4c, and is called system.journal. This file is binary, and cannot be read directly though.
3. To read the journal, you should use the following command: > journalctl. (man page for journalctl) This dumps all the contents of the journal in a human readable form. You can also issue the option -f (or –follow) to show the tail output (NOTE: In previous versions of Angstrom, the command was systemd-journalctl.  This has been replaced in the latest v44-45 systemd)
4. You may have to modify the systemd configuration to set the log level. This can be found in /etc/systemd in the file system.conf. Uncomment the option LogLevel and set it to: LogLevel=debug

Using journalctl, I could confirm that my syslog() output was being logged correctly to the journal and I was slightly happier. It turns out that when you turn on syslog (Busybox), its actually hooking to the journal socket to get its output. So, there should be no reason to run syslog or syslog-ng given the systemd journal feature.

The journal is written to /run/log, which is volatile and will be lost on reboot. To persist the journal, you should create a directory called /var/log/journal (you will note it does not exist on stock Angstrom file systems). Once you create it, magically you will see a sub-directory created and the journal will be stored there.  Running journalctl will show persistent (and historical) version of the journal (instead of just the /run/logs version)  If you get an error like: ”Failed to iterate through journal: Cannot allocate memory“. , this is a bug in older versions of systemd.  Upgrading your img or systemd should solve this issue.

To control the log file size, you can use the journald.conf file (found under /etc/system/systemd-journald.conf) and modify the SystemMaxUse options.  From initial testing, it appears to be working well, but I will continue testing.

I’ve removed my last paragraph from the original post because it was hating on systemd.  As the comments point out, the systemd documentation is quite good.  I was mainly frustrated at lack of information on Angstrom and the incorrect information on system logging using syslog/syslog-ng.

Entry Filed under: Uncategorized