You set up a nice ZFS pool, everything runs great, and then you wake up to an OOM killer event that took out your database. It happens because ZFS likes to claim almost all your RAM for its Adaptive Replacement Cache, assuming it’s the only thing that matters on the machine. If you’re running anything else alongside your data, you need to put the ARC on a leash.

Why ZFS defaults to gluttony

The kernel module is designed to grow the ARC until it hits your system’s total memory capacity minus a sliver of overhead. In a dedicated storage server, that’s fine. On a box running your web server or monitoring stack, it’s a liability. You’ll see the cache hit rate look amazing, right until your actual applications start hitting swap.

echo 16G > /sys/module/zfs/parameters/zfs_arc_max

• Check your current usage with arcstat to see the real numbers
• Set the cap in bytes to account for your application’s baseline footprint
• Make it permanent by adding the setting to /etc/modprobe.d/zfs.conf
• Keep at least 20 percent of your RAM outside the cache for kernel overhead and buffers

Don’t just set this once and walk away. If you add more apps or containers, you’ll need to tighten these bounds again. If you have the luxury of monitoring tools, alert on memory pressure stalls before the OOM killer decides to make your decisions for you.