You probably noticed that ZFS is a greedy neighbor. Left to its own devices, it will happily consume almost all of your system memory for the Adaptive Replacement Cache (ARC). Most of the time this is fine, but it becomes a disaster when your applications start hitting OOM kills because the kernel refused to reclaim memory from the ARC fast enough. I have seen this cause entire production database clusters to flap under load, and the standard docs make it sound like the kernel handles it automatically. It doesn’t.

Why the defaults lie to you

The kernel is supposed to reclaim memory from ZFS when things get tight, but the ARC is technically counted as cache. In practice, the pressure signal is often ignored until it is too late. If you are running services that aren’t just I/O bound, you need to hard-cap the ARC to ensure the OS and your applications have enough breathing room. Setting a floor and a ceiling is the only way to sleep through the night without worrying about midnight paging alerts.

echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max

• Calculate your cap based on physical RAM minus at least 4GB for the OS overhead
• Add the setting to /etc/modprobe.d/zfs.conf so it persists after reboot
• Monitor the hit ratio using arcstat to ensure you haven’t choked your performance too hard

If you don’t define these limits, ZFS will eventually push your system into swap, and you know as well as I do that once you start swapping under heavy I/O, the whole stack dies a slow, miserable death. Don’t wait for a prod outage to find your memory ceiling. Set a sane limit today and adjust it only when you actually see cache miss rates climbing.