systemd OOMD Maturing Nicely


Systemd-oomd as the out-of-memory daemon originally developed by Facebook has been maturing nicely since being merged last year and then its most notable deployment to date has been with Fedora 34's debut earlier this year. Anita Zhang of Facebook provided an update today on the systemd-oomd effort. 

During the Linux Plumbers Conference (LPC2021) today was this update by Anita on systemd-oomd. Systemd-oomd being used by default in Fedora 34 has been one of the big drivers this year for the project. Some early issues since resolved with systemd-oomd have revolved around initial limits for initiating the OOM killing being too low, swap killing being too aggressive, and high CPU resources. 

A new feature for systemd-oomd is bringing support around settings for user units. That pull request was just merged into systemd yesterday around supporting user unit ManagedOOM property updates. One of the patches explains:

  • Compared to PID1 where systemd-oomd has to be the client to PID1 because PID1 is a more privileged process than systemd-oomd, systemd-oomd is the more privileged process compared to a user manager so we have user managers be the client whereas systemd-oomd is now the server.
  • The same varlink protocol is used between user managers and systemd-oomd to deliver ManagedOOM property updates. systemd-oomd now sets up a varlink server that user managers connect to to send ManagedOOM property updates.
  • We also add extra validation to make sure that non-root senders don't send updates for cgroups they don't own.

Another area for improvement being discussed is offering more insight when killing a cgroup in a user slice. Right now when user programs are killed, there is little information conveyed to the user that it was killed in the name of out-of-memory management. The current thinking is for systemd-oomd to have a D-Bus interface to let clients know about OOMD notifications. 

More details on the state of systemd-oomd for autumn 2021 can be found via Anita's presentation embedded below (if YouTube is acting up, it begins about three quarters the way into the stream) and this slide deck from the presentation. 

Out of memory killing has historically happened inside kernel space. On a memory overcommitted linux system, malloc and friends usually never fail.