Apparently having the worker processes of four unrelated server programs running next to those server programs on the same node is not ideal. No matter how much memory the node has, the out-of-memory killer will eventually kill the web server. It seems that the worker processes look at how much memory is in the machine and use it, not at how much memory is left unused in the machine.
I've reinstated the swapfile and changed the cron schedule of the workers to not happen at the same time anymore. That was a bit optimistic...