ZFS ARC Cache Saturation on Linux – Quick Fixes

The Short Version

You're seeing arc_kmem_reap_now in dmesg. Your ZFS ARC cache is fighting the Linux kernel for RAM. This usually happens on systems with heavy file I/O (like a file server or backup target) and not enough headroom. The fix is to cap ARC memory usage. Don't bother tweaking arc_min – that rarely helps here.

30-Second Fix (Quick Relief)

Run this as root. It lowers ARC max to 50% of system RAM. The change is immediate – no reboot.

echo "zfs_arc_max=$(($(free -b | grep Mem | awk '{print $2}') / 2))" > /sys/module/zfs/parameters/zfs_arc_max

Check it worked:

cat /sys/module/zfs/parameters/zfs_arc_max

If the kernel log stops showing arc_kmem_reap_now in the next minute, you're good for now. This won't survive a reboot – but it buys you time.

5-Minute Fix (Permanent Tweak)

This makes the change stick across reboots. You'll edit a file and reload ZFS settings.

Step 1: Create a module config file

echo "options zfs zfs_arc_max=$(($(free -b | grep Mem | awk '{print $2}') / 2))" > /etc/modprobe.d/zfs-arc.conf

This sets ARC max to half your RAM. If you have 8GB, that's 4GB for ARC. Adjust if you need more – but don't go above 80%. ZFS needs room for other stuff.

Step 2: Check current tunables

cat /proc/spl/kstat/zfs/arcstats | grep -E "c_max|c_min|size"

You should see c_max close to what you set. If not, reload ZFS or reboot.

Step 3: Reload ZFS (optional – skip if you can reboot)

modprobe -r zfs && modprobe zfs

This disconnects all ZFS pools for a second. Only do this if you have no active I/O. Safer to just reboot.

15-Minute Fix (Deep Tuning)

If the problem still shows up, it's not just ARC size. You have a workload that's constantly hitting the cache. Time to look at your system's memory pressure and swap.

Check swap usage

High swap usage means the kernel is pushing out ARC data. Run this:

vmstat 1 10

Look at the si and so columns. If they're non-zero consistently, you're swapping. That kills performance.

Reduce ARC min and max together

Sometimes the kernel doesn't reclaim ARC fast enough. Set a lower min to let the kernel take memory back quicker.

echo "options zfs zfs_arc_min=$(($(free -b | grep Mem | awk '{print $2}') / 8))" >> /etc/modprobe.d/zfs-arc.conf

This sets min to 12.5% of RAM. Reboot. Now the kernel can shrink ARC down to that level before you hit swapping.

Monitor with arc_summary

Install arc_summary (part of zfs-zed or zfsutils-linux). Run it after a few hours:

arc_summary -s arc

Look at ARC size vs Target size. If they're close and still showing saturation, lower zfs_arc_max further.

One more thing

If you have a deduplication table (dedup=on), ARC saturation is almost guaranteed. Dedup tables are huge and live in ARC. Turn off dedup unless you absolutely need it. It's the biggest RAM hog in ZFS.

When to call it

After these changes, check dmesg | grep arc_kmem_reap_now. If you see no new messages after a day, you're fixed. If you still see them, you need more RAM or a smaller ARC limit. Sometimes the answer is just hardware.

Pro tip: On systems with 4GB RAM or less, cap ARC at 1GB. ZFS will still be fast, and your desktop won't freeze.