Avoid Linux locking up in low memory situations using earlyoom

Solene · September 28, 2022, 7:23pm

I really needed it, my laptop with 8 GB of memory can’t handle nix-env on nixpkgs-unstable anymore

peterhoeg · September 29, 2022, 3:27am

I use this in my build wrapper script which obviously only addresses the nix part, but is helpful nonetheless:

MEM=$(free --giga | grep Mem | gawk '{print $2}')
if [ "$MEM" -lt 12 ]; then
  echo "Detected memory: ${MEM}GB. Lowering GC heap size"
  export GC_INITIAL_HEAP_SIZE=1m
fi

Solene · September 29, 2022, 7:17am

Thank you! This seems useful, the default garbage collector size is 384M per Common Environment Variables - Nix Reference Manual

I understand your command saves up to 383 MB of memory for a nix process? Seems useful for my systems with 1 or 2 GB!

peterhoeg · September 29, 2022, 9:04am

I understand your command saves up to 383 MB of memory for a nix process?

I don’t know enough about the specific garbage collector or how the various nix commands do allocations to give you anything tangible in terms of quantifiable memory savings.

My only data point - on an 8GB laptop, it would go into swap hell when running other things and doing nixos-rebuild without this. With this, the laptop would remain usable.

woile · May 9, 2025, 7:47am

Hi everyone, is this still working well for you?
Do you also disable systemd.oomd or can they live together?

Thanks!

waffle8946 · May 10, 2025, 11:56pm

I use earlyoom, it works. It fires well before systemd would oom-kill.

phanirithvij · May 11, 2025, 2:22am

There is services.swapspace which is tagentially related, it allows using your storage as swap when it detects memory is below set threshold and releases the storage space afterwards.

Frontear · May 11, 2025, 4:54pm

Another thing I’d encourage looking into is increasing the memory thresholds for the nix daemon systemd service. I was struggling with OOM conditions for a while until I increased the memory thresholds available to the daemon, which made the daemon survive perfectly fine in almost all cases without degrading performance. 12GB laptop, and of course, ymmv. I also use zram for swapping purposes.

waffle8946 · May 11, 2025, 6:29pm

Zram reduces available memory and keeps the cpu tied up longer in swapping, if anything it’d make builds more likely to stall in a different way (not technically OOM but memory pressure would force quasi-thrashing).

malteneuss · May 11, 2025, 7:11pm

This tool is a blessing. Summary for NixOS from the blog post to prevent Linux freezes from out of memory situations:

{
  services.earlyoom = {
      enable = true;
      freeSwapThreshold = 2;
      freeMemThreshold = 2;
      extraArgs = [
          "-g" "--avoid '^(X|plasma.*|konsole|kwin)$'"
          "--prefer '^(electron|libreoffice|gimp)$'"
      ];
  };
}

woile · May 16, 2025, 8:28am

Thanks everyone, so in a nutshell, you enable oom and earlyoom + in some cases zram, therefore, all of them can be used together.

I did a very conservative change, and the situation has improved, if I see it deteriorates, I’ll enable earlyoom as well:

{
  zramSwap.enable = true;
  systemd.oomd.enableUserSlices = true;  # take action on user-space process hierarchies
}

lux · May 16, 2025, 2:15pm

getting the error

systemctl status earlyoom
× earlyoom.service - Early OOM Daemon
     Loaded: loaded (/etc/systemd/system/earlyoom.service; enabled; preset: ignored)
    Drop-In: /nix/store/wb3d1kqsqlfxmq6kyf4ihzv62lj24njk-system-units/earlyoom.service.d
             └─overrides.conf
     Active: failed (Result: exit-code) since Fri 2025-05-16 19:39:25 IST; 4min 17s ago
   Duration: 54ms
 Invocation: b0f68ef5c4c8400b964813281ca728c8
       Docs: man:earlyoom(1)
             https://github.com/rfjakob/earlyoom
    Process: 2006 ExecStart=/nix/store/cpl768r9vzf8zdyg8zh9lziab4ps3y42-earlyoom-1.8.2/bin/earlyoom $EARLYOOM_ARGS (code=exited, status=13)
   Main PID: 2006 (code=exited, status=13)

May 16 19:39:25 leptup systemd[1]: earlyoom.service: Scheduled restart job, restart counter is at 5.
May 16 19:39:25 leptup systemd[1]: earlyoom.service: Start request repeated too quickly.
May 16 19:39:25 leptup systemd[1]: earlyoom.service: Failed with result 'exit-code'.
May 16 19:39:25 leptup systemd[1]: Failed to start Early OOM Daemon.

with

  services.earlyoom = {
    enable = true;
    freeSwapThreshold = 2;
    freeMemThreshold = 2;
    extraArgs = [
      "-g"
      "--avoid '^(X|init|Xorg|ssh|gnome.*|ghostty|kwin)$'"
      "--prefer '^(electron|libreoffice|gimp|zen*)$'"
    ];
  };

i use Gnome btw

malteneuss · May 16, 2025, 2:19pm

Can you past the result of

$ journalctl -xu earlyoom

Should look like

May 11 21:06:53 pc systemd[1]: Started Early OOM Daemon for Linux.
░░ Subject: A start job for unit earlyoom.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit earlyoom.service has finished successfully.
░░
░░ The job identifier is 8940.
May 11 21:06:53 pc earlyoom[82412]: earlyoom 1.8.2
May 11 21:06:53 pc earlyoom[82412]: Preferring to kill process names that match regex '^(hls.*)$'
May 11 21:06:53 pc earlyoom[82412]: Will avoid killing process names that match regex '^(X|xmonad.*|firefox.*)$'
May 11 21:06:53 pc earlyoom[82412]: mem total: 23788 MiB, user mem total: 22214 MiB, swap total: 8191 MiB
May 11 21:06:53 pc earlyoom[82412]: sending SIGTERM when mem avail <=  6.00% and swap free <=  2.00%,
May 11 21:06:53 pc earlyoom[82412]:         SIGKILL when mem avail <=  3.00% and swap free <=  1.00%
May 11 21:06:53 pc earlyoom[82412]: killing whole process group 4428 (-g flag is active)
May 11 21:06:53 pc earlyoom[82412]: mem avail: 15173 of 22213 MiB (68.31%), swap free: 8191 of 8191 MiB (100.00%)

lux · May 16, 2025, 2:38pm


i presume the options passed are incorrect?
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit earlyoom.service has entered the 'failed' state with result 'exit-code'.
lines 1-25...skipping...
May 16 18:00:41 leptup systemd[1]: Started Early OOM Daemon.
░░ Subject: A start job for unit earlyoom.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit earlyoom.service has finished successfully.
░░
░░ The job identifier is 3784.
May 16 18:00:41 leptup earlyoom[24316]: earlyoom 1.8.2
May 16 18:00:41 leptup earlyoom[24316]: /nix/store/cpl768r9vzf8zdyg8zh9lziab4ps3y42-earlyoom-1.8.2/bin/earlyoom: unrecognized option '--avoid '^(X|init|Xorg|ssh|gnome.*|ghostty|kwin)$''
May 16 18:00:41 leptup earlyoom[24316]: Try 'earlyoom --help' for more information.
May 16 18:00:41 leptup systemd[1]: earlyoom.service: Main process exited, code=exited, status=13/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit earlyoom.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 13.
May 16 18:00:41 leptup systemd[1]: earlyoom.service: Failed with result 'exit-code'.

lux · May 16, 2025, 2:41pm

also i have ran into memory issues in the past. even though i have 16Gb RAM and use 17ishGb swap. i tried system-oom killer. but i’d always kill the gnome-shell and i’ll be thrown back to the login screen. and then i experience a few too many artifacts and glitches. so i’d have to restart anyways. and neither does my work get saved.

malteneuss · May 16, 2025, 6:36pm

I’m not sure what could be wrong here, but you can leave it out, and then add one item at a time.

NoRePercussions · May 17, 2025, 1:47am