You probably think the ZFS ARC is smart enough to handle heavy file operations without starving the rest of your system. Usually, you are right. But I have seen production servers turn into bricks during an rsync of a massive directory because the ARC decided to cache every single metadata entry and file block it encountered. Suddenly, your applications are swapping to disk, and the kernel starts firing off OOM killer events because the memory pressure just spiked through the roof.

Why the defaults are actually lying to you

By default, ZFS is configured to use up to 50 percent of your physical RAM for the ARC. In a server with 128GB of memory, that is 64GB. That sounds reasonable until you trigger an operation that reads millions of small files. The ARC prioritizes caching those reads, and before you know it, the kernel has pushed your active process memory into swap. It is not that ZFS is broken; it is that the default upper bound is often way too high for systems running more than just a storage daemon.

echo 32212254720 > /sys/module/zfs/parameters/zfs_arc_max

• Check your current hard limit with cat /sys/module/zfs/parameters/zfs_arc_max
• Adjust the value to something sane based on your actual workload, not the total RAM
• Make the change permanent by adding options zfs zfs_arc_max=32212254720 to /etc/modprobe.d/zfs.conf
• Remember to run update-initramfs -u or equivalent if you are on a system where this must be baked into the early boot environment

When to actually cap the ARC

If you are running a dedicated storage head, leave the defaults alone. But if this box is also running Postgres or a fleet of containers, cap that ARC immediately. A good rule of thumb is to calculate your absolute minimum required RAM for non-storage services and set your zfs_arc_max to whatever is left over plus a 20 percent buffer. You will notice a slight dip in cache hit rates, but your system will stay upright when the backup job kicks off.

Next time you see load averages climbing while IO wait remains suspiciously low, check if your memory has been claimed by the ARC. It is a common blind spot, but it is one you can fix in five minutes with a simple module parameter. Just make sure you track your cache hit stats afterwards; sometimes the fix is worse than the problem if you choke the ARC too aggressively.