OOM issue for patchman server with ~115 clients

[pwolynec-swin]

One of the Nectar nodes in Melbourne (SUT) has started to use patchman (thanks to TomF) and every ~second month it has an OOM issue.

At 6:30 in the morning, there is a step of ~1 GB of memory usage.

We altered on the clients (puppet erb code below) -

was originally 3 and 15 minutes

Sleep between 3 and 120 minutes before running to avoid "herding":

SLEEP_MINUTES=$(/usr/bin/shuf -i 3-120 -n 1)
SLEEP_SECONDS=$((60 * $SLEEP_MINUTES))

Since we can sleep up to 90 minutes, we put the command in the background so it

does not block any other cron jobs.

sleep $SLEEP_SECONDS && patchman-client -c <%= @patchman_conf_path %> -s http://<%= @patchman_server %>/patchman &

and now the step up in memory usage at ~6:30 is more modest (0.2 GB)

The patchman server currently has 32 GB.

Just wondering what settings and where we can can keep patchman running without an OOM episode.

[pwolynec-swin]

[furlongm]

Is the OOM happening on the server?

If so, is DEBUG set to true on the server?

[pwolynec-swin]

the OOM issue is indeed happening on the patchman server
DEBUG is now set to TRUE in both /etc/patchman/local_settings.conf + /etc/patchman/local_settings.py + apache2.service has been reloaded (puppet is disabled)

[furlongm]

Is the OOM issue happening on the server when DEBUG is set to True? Or is it happening when DEBUG is set to False? DEBUG uses a lot more memory so I would expect the issue to occur if it set to True.

[pwolynec-swin]

DEBUG was initially set to False - it was just changed to True this morning local time - the OOM happened before the change to DEBUG = True

[pwolynec-swin]

apologies - I have misunderstood you - I will change DEBUG = True back to DEBUG = False (as it was)

[furlongm]

Is there a cronjob running at 6:30?

[pwolynec-swin]

on the server =

root@patchman:/etc/cron.d# cat patchman
*/10 0-2,4-23 * * * root /usr/bin/patchman -p -u >/dev/null 2>&1
35 3 * * * root /usr/bin/patchman -a >/dev/null 2>&1

and on a client =

root@gs-os-n06:/etc/cron.daily# cat patchman-client-10_8_30_246
#!/bin/sh

was originally 3 and 15 minutes

Sleep between 3 and 120 minutes before running to avoid "herding":

SLEEP_MINUTES=$(/usr/bin/shuf -i 3-120 -n 1)
SLEEP_SECONDS=$((60 * $SLEEP_MINUTES))

Since we can sleep up to 90 minutes, we put the command in the background so it

does not block any other cron jobs.

sleep $SLEEP_SECONDS && patchman-client -c /etc/patchman/10.8.30.246.conf -s http://10.8.30.246/patchman &

cat /etc/patchman/10.8.30.246.conf

Patchman server

server=http://10.8.30.246/patchman

Options to curl

curl_options="--netrc-file /etc/patchman/10.8.30.246.netrc --connect-timeout 60 --max-time 300"

Space delimited tags to send

tags="Server"

Does the client output a report of the upload (e.g. for cronjob output)

report=0

[furlongm]

Is celery running on the server?

What process uses more memory at 6:30am?

What process does the OOM kill?

[pwolynec-swin]

What process does the OOM kill? - mysql for one
What process uses more memory at 6:30am? - let me check and get back to you

[pwolynec-swin]

Thu 15 Jan 2026 06:15:01 AEDT
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1129 38.8 1.6 1845116 544740 ? Ssl Jan07 4243:19 /usr/sbin/mysqld
root 841 0.0 0.3 164380 106056 ? Ss Jan07 1:03 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
www-data 169535 0.0 0.2 1187900 93324 ? S Jan14 0:00 /usr/sbin/apache2 -k start
root 329 0.0 0.2 115980 79536 ? S<s Jan07 0:46 /usr/lib/systemd/systemd-journald
root 186627 61.5 0.2 147852 76764 ? R 06:10 3:04 /usr/bin/python3 /usr/bin/patchman -p -u
root 169532 0.0 0.0 38184 29708 ? Ss Jan14 0:02 /usr/sbin/apache2 -k start
root 377 0.0 0.0 288952 27008 ? SLsl Jan07 0:52 /sbin/multipathd -d -s
root 935 0.0 0.0 111700 22912 ? Ssl Jan07 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
node-ex+ 823 0.0 0.0 717636 22528 ? Ssl Jan07 6:05 /usr/local/bin/node_exporter --no-collector.rapl --no-collector.infiniband
19.6719 MB /usr/sbin/ntpd
22 MB /usr/local/bin/node_exporter
22.375 MB /usr/bin/python3
26.375 MB /sbin/multipathd
29.0117 MB /usr/sbin/apache2
74.9648 MB /usr/bin/python3
77.6719 MB /usr/lib/systemd/systemd-journald
91.1367 MB /usr/sbin/apache2
103.57 MB /opt/puppetlabs/puppet/bin/ruby
531.973 MB /usr/sbin/mysqld
Thu 15 Jan 2026 06:30:01 AEDT
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1129 38.8 1.6 1845116 545416 ? Ssl Jan07 4249:58 /usr/sbin/mysqld
root 841 0.0 0.3 164380 106056 ? Ss Jan07 1:03 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
root 186843 59.8 0.3 180780 102668 ? R 06:25 2:58 /usr/bin/python3 /usr/bin/patchman -a -q
www-data 169535 0.0 0.2 1187900 93324 ? S Jan14 0:00 /usr/sbin/apache2 -k start
www-data 180685 0.0 0.2 1694060 90836 ? Sl 00:00 0:00 /usr/sbin/apache2 -k start
root 329 0.0 0.2 115980 79664 ? S<s Jan07 0:46 /usr/lib/systemd/systemd-journald
root 169532 0.0 0.0 38184 29708 ? Ss Jan14 0:02 /usr/sbin/apache2 -k start
root 377 0.0 0.0 288952 27008 ? SLsl Jan07 0:52 /sbin/multipathd -d -s
root 935 0.0 0.0 111700 22912 ? Ssl Jan07 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
22 MB /usr/local/bin/node_exporter
22.375 MB /usr/bin/python3
26.375 MB /sbin/multipathd
29.0117 MB /usr/sbin/apache2
77.7969 MB /usr/lib/systemd/systemd-journald
88.707 MB /usr/sbin/apache2
91.1367 MB /usr/sbin/apache2
100.262 MB /usr/bin/python3
103.57 MB /opt/puppetlabs/puppet/bin/ruby
532.633 MB /usr/sbin/mysqld
Thu 15 Jan 2026 06:45:01 AEDT
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1129 38.8 1.6 1845116 544828 ? Ssl Jan07 4256:09 /usr/sbin/mysqld
root 841 0.0 0.3 164380 106056 ? Ss Jan07 1:03 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
www-data 169535 0.0 0.2 1187900 93324 ? S Jan14 0:00 /usr/sbin/apache2 -k start
www-data 180684 0.0 0.2 1185948 92704 ? S 00:00 0:00 /usr/sbin/apache2 -k start
www-data 187297 0.3 0.2 1694140 92460 ? Sl 06:41 0:00 /usr/sbin/apache2 -k start
www-data 180685 0.0 0.2 1186100 92436 ? S 00:00 0:00 /usr/sbin/apache2 -k start
root 329 0.0 0.2 115980 79664 ? S<s Jan07 0:46 /usr/lib/systemd/systemd-journald
root 187286 62.0 0.2 149012 77448 ? R 06:40 3:06 /usr/bin/python3 /usr/bin/patchman -p -u
root 169532 0.0 0.0 38184 29708 ? Ss Jan14 0:02 /usr/sbin/apache2 -k start
26.375 MB /sbin/multipathd
29.0117 MB /usr/sbin/apache2
75.6328 MB /usr/bin/python3
77.7969 MB /usr/lib/systemd/systemd-journald
90.2695 MB /usr/sbin/apache2
90.293 MB /usr/sbin/apache2
90.5312 MB /usr/sbin/apache2
91.1367 MB /usr/sbin/apache2
103.57 MB /opt/puppetlabs/puppet/bin/ruby
532.059 MB /usr/sbin/mysqld

[furlongm]

That's a little hard to read.

Could you try pasting the output of

ps -o user,pid,rss,vsz,%mem,command --sort=-rss | head -n 15 | sed 's/  */ | /g; s/^/| /; s/$/ |/' | sed '2i\|---|---|---|---|---|---|'

[furlongm]

And maybe pgrep -a -f patchman to see how many patchman instances are running at the same time?

[pwolynec-swin]

root@patchman:~# ps -o user,pid,rss,vsz,%mem,command --sort=-rss | head -n 15 | sed 's/ */ | /g; s/^/| /; s/$/ |/' | sed '2i|---|---|---|---|---|---|'

USER	PID	RSS	VSZ	%MEM	COMMAND
root	190912	5632	10564	0.0	-bash
root	191779	4352	12504	0.0	ps
root	191781	2304	8272	0.0	sed
root	191782	2304	8272	0.0	sed
root	191780	1920	7348	0.0	head
root@patchman:~# pgrep -a -f patchman
191394 /bin/sh -c /usr/bin/patchman -p -u >/dev/null 2>&1
191395 /usr/bin/python3 /usr/bin/patchman -p -u

[furlongm]

If not running celery (can you check this too?) then maybe patchman -p is all you need for processing reports, and let the daily cronjob find the updates with patchman -a. Otherwise you are finding all updates (with the older slower -u, rather than -A) every X minutes.

If you are running celery, it should process the reports and find updates for that particular host only, so patchman -p is not needed at all.