RAM disk for "/var/hdd.log/" too small. Can't sync. (but system is reporting: 4.0K /var/hdd.log) - Pop!_OS 21.04

[MostHated]

Hey there,
I came across this project today when looking for, well, something that did exactly what this does so I decided to give it a try as this new firewall software I was having a go at seems to write nonstop logs to disk on every network action for every application, it seems. (Portmaster) I figured that instead of subjecting my SSD's to all those writes constantly, why not log it to ram first, then just dump that in one go.

That said, I installed it, cleaned up my logs a bit before hand to get the size down to a reasonable amount and restarted and went to check if it was running, which was when I saw the message of:

● log2ram.service - Log2Ram
     Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2021-07-10 16:08:56 CDT; 19min ago
    Process: 1377 ExecStart=/usr/local/bin/log2ram start (code=exited, status=1/FAILURE)
   Main PID: 1377 (code=exited, status=1/FAILURE)

Jul 10 16:08:56 pop-os log2ram[1377]: ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync.
Jul 10 16:08:56 pop-os log2ram[1424]: /usr/local/bin/log2ram: 45: mail: not found

I was under the impression that it only dealt with /var/log/*, and I can't even say I remember seeing /var/hdd.log before either? I tried to take a look in the folder to see if there were files and their size using the line you recommended in the readme, it reported back the following:

zsh: sudo du -hs /var/h* | sort -h | tail -n 3                                                                   mosthated@pop-os
4.0K    /var/hdd.log

I entered the folder and did ls -lah but nothing came back.

Any recommendations on what I might need to check/do? My system is running Pop!_OS 21.04.

Thanks,
-MH

[MostHated]

Nevermind about it, I believe I have got it working.

● log2ram.service - Log2Ram
     Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sun 2021-07-11 17:48:10 CDT; 49s ago
    Process: 1602947 ExecStart=/usr/local/bin/log2ram start (code=exited, status=0/SUCCESS)
   Main PID: 1602947 (code=exited, status=0/SUCCESS)

...
Jul 11 17:48:10 pop-os log2ram[1602959]: sent 39,384,400 bytes  received 10,239 bytes  78,789,278.00 bytes/sec
Jul 11 17:48:10 pop-os log2ram[1602959]: total size is 39,332,259  speedup is 1.00

[azlux]

The code just check the size of destination folder:
https://github.com/azlux/log2ram/blob/master/log2ram#L40

I'm surprise you get an error on /var/hdd.log/

Is that Pop!_OS specific ?

[TupaNegreiros]

Same error here. Ubuntu for Raspberry Pi (64bits) and Raspbian (32bit). Empty /var/hdd.log folder. /var/log/log2ram.log isn't even created.

How did you solve it?

[MostHated]

I just made it bigger, something like 400mb and removed as much as I could and set limits.

[TupaNegreiros]

My Ubuntu /var/log has about 150mb, I set log2ram to 160mb and it started.
But Raspbian /var/log has almost 900mb and only 1Gb of RAM.

@azlux Should it store on RAM even old rotated logs? (daemon.log.1, syslog.1, *.gz, *.[0-9])

[xtvdata]

My Ubuntu /var/log has about 150mb, I set log2ram to 160mb and it started.
But Raspbian /var/log has almost 900mb and only 1Gb of RAM.

Should it store on RAM even old rotated logs? (daemon.log.1, syslog.1, *.gz, *.[0-9])

It is technically not possible to sync selectively, since the Ram disk is mounted over the root disk at the /var/log location and synced back to the root disk periodically. When the RAM disk is mounted you can only see the content of the RAM disk, thus a selective sync would break the standard Linux log rotation mechanism (and possibly even something else).

However you could eventually change logrotate’s configuration to rotate some or all of the old logs to a different archive directory which would not be synced to the RAM disk (not under /var/log). But that is out of the log2ram’s scope.

Anyway 900MB of logs is quite significant for a Raspberry Pi. Are you sure that you don’t have some applications installed by you on top of the vanilla install, that are not managed by log rotation properly and are generating an ever growing log? How big are the 3 largest files you have in /var/log (or in one of the subdirectories)?
If those are not rotated during the night you are missing some system the logrotate.

Really, 900MB of textual logs is really huge for a non large business/production system. Consider that in 1MB of plain text you can, more or less, fit in a 500 pages novel or 10,000 events described with a 100 characters messages. Moreover, if rotation is configured properly, most of the files in /var/log should be already compressed (with an average efficiency of 4-5:1)… Are you analyzing all of that info?
If logs are that large for debug purposes (e.g. an alpha-test/development server) you can rotate more frequently and probably keep just a few days of history to check the tests results.

I’d recommend to check what is generating so much logging, confirm that you really need that level of detail and modulate at the origin log level based on you needs, otherwise, if you really need that much logging, you should consider a different log strategy (e.g. a dedicated log server), a structured log server will indeed serve you much better than just plain text files (in this case log2ram could help in keeping the low the server load).

[bubuntux]

i ran into the same issue and fixed :
checking the size of the current log folder
du -h /var/log/ | tail -n1

change the config SIZE in /etc/log2ram.conf to something grater than the value of the size of the log folder

restart the service

[azlux]

@TupaNegreiros log2ram store everything. Storing only new log was tricky for this kiss project.

[Blindfreddy]

Followed instructions for installation with apt. I got the same problem on two different pis:

Feb 28 08:20:47 rpi1 log2ram[314]: ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync.
Feb 28 08:20:47 rpi1 log2ram[314]: /usr/local/bin/log2ram: 45: /usr/local/bin/log2ram: mail: not found

strangely, the file is actually very small, which is quite misleading:

$ ll /var/hdd.log/
total 0

increased size to 500MB in conf but same problem. Then noticed that there are two places to set sizes: at the top

# Size for the ram folder, it defines the size the log folder will reserve into the RAM.
# If it's not enough, log2ram will not be able to use ram. Check you /var/log size folder.
# The default is 40M and is basically enough for a lot of applications.
# You will need to increase it if you have a server and a lot of log for example.
SIZE=40M

and at the bottom

LOG_DISK_SIZE=500M

I increased that latter one but that's about Zram. Increasing the first one made it work for me :-)

May I suggest that the error message is changed to something like

RAM disk is too small, please reduce existing logs and/or increase memory in configuration file.

raspbian buster and bullseye 32bit

[azlux]

You're right, I will make a better error message.

About

Feb 28 08:20:47 rpi1 log2ram[314]: /usr/local/bin/log2ram: 45: /usr/local/bin/log2ram: mail: not found

You have set MAIL to true that why log2ram try to send one.

[Blindfreddy]

Ok great. And thanks for the hint about Mail, I will change the value in my config file.

It might also be useful to include some instructions and/or links on how to reduce the log size before activating log2ram in the documentation:

something like this:

before activating log2ram with sudo systemctl enable log2ram, you may want to reduce the logsize to 20MB, set the maximum size and disable mail:

• to disable mail: sudo perl -pe 's/^MAIL=.*$/MAIL=false/g' -i /etc/log2ram.conf
• to set the maximum log size: sudo perl -pe 's/^#?SystemMaxUse=.*$/SystemMaxUse=20M/g' -i followed by /etc/systemd/journald.conf sudo systemctl restart systemd-journald
• to force log size reduction: sudo journalctl --vacuum-size=16M

[thewriteway]

I couldn't get the service to start due to my Pi having excessively large log files.

firstly I tried the journal reduction: sudo journalctl --vacuum-size=16M
That wasn't sufficient so I removed the other legacy logs and truncated the active ones from the log directory:

sudo rm -f *.gz
sudo truncate -s50 *.log

[xtvdata]

I couldn't get the service to start due to my Pi having excessively large log files.

firstly I tried the journal reduction: sudo journalctl --vacuum-size=16M That wasn't sufficient so I removed the other legacy logs and truncated the active ones from the log directory:

sudo rm -f *.gz sudo truncate -s50 *.log

Well, just "truncating" the logs will not solve your issue.
It will improve the situation, but only for a few time, then eventually /var/log will grow too large again.
If you had to do that it means that you have a problem with log verbosity (e.g. debug) and/or with log rotation.

Thus, you should reduce verbosity of your apps and/or fine tune the configuration log logrotate (see other discussions on the log size topic).

[i-am-stu]

Using the official imaging tool, I just installed Raspberry Pi OS Lite 64bit on an RPi 3b and, after updating packages, installed log2ram. I was presented with the same error as noted on this issue:

ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync

I eventually determined it was because /var/log was 41M, slightly too large for the default SIZE=40M. I resolved it editing /etc/log2ram.conf and setting SIZE=80M, and then rebooting.

Strangely, this was not the case when I installed Ubuntu 22.04 LTS on the same device. I guess a fresh install of Raspberry Pi OS tends to log a bit more?

[DarkSkull91]

I faced this same problem on a fresh install of Raspberry Pi OS. Increasing the size to SIZE=80M solves the problem.

[heutger]

Same for me with a fresh lite Raspberry Pi OS 64 bit, seems like new installations take more log file size and default setting is too low for them. With 32 bit and Buster it worked, but 64 bit and Bullseye it didn't.

[gellnerm]

Also consider enabling log compression in /etc/systemd/journald.conf for systemd:

Compress=yes

[maximelafarie]

Even if it's strange, setting SIZE=80M solved the problem for me too (fresh Raspbian OS 64-bits Lite install)

[bmn001]

It seems like the install process should check current logfile sizes and offer to purge excess before installing to avoid this error, which seems now to be nearly-default behavior for installs across several OSs, no?