You probably think the ZFS Adaptive Replacement Cache is smart enough to handle itself. Most of the time it is, but I have seen production servers crawl to a halt because the ARC kept holding onto cold file system metadata instead of letting the kernel use that memory for actual application needs. When your application starts swapping while ZFS is sitting on 64 gigabytes of cache it doesn’t really need, you are in for a bad afternoon.

Why the defaults are killing your headroom

By default, ZFS tries to use half of your total system RAM. That sounds reasonable until you realize it doesn’t account for what your database or container runtime actually requires. It is not uncommon for the ARC to become greedy and force the OS into the swap partition, which turns your disk I/O wait times into a total nightmare. If you see your swap usage spiking alongside high ARC hit rates, you have already lost the battle.

echo 32212254720 > /sys/module/zfs/parameters/zfs_arc_max

• Calculate your ceiling based on active memory usage, not total physical RAM
• Set zfs_arc_max in your modprobe.d file to make the change survive a reboot
• Monitor the difference between arc_size and system free memory to find your sweet spot

If you are running ZFS on a dedicated storage node, leave the defaults alone. But if you are mixing storage and application workloads on the same bare metal, hard-capping the ARC is the only way to keep the system responsive. Check your current stats with arcstat or cat /proc/spl/kstat/zfs/arcstats and verify how much memory you are actually wasting on cached files that nobody has touched in days.