Backup & Recovery (Rsync/Tar/Dd)

Stop rsync from choking on millions of tiny files

🧩 The Challenge

Trying to rsync a directory with a few million tiny cache files is a nightmare that will keep you waiting for an hour just to build the file list. I’ve wasted so many afternoons watching a cursor blink while rsync sits there doing nothing but memory bloat.

💡 The Fix

Don’t let it crawl through the entire file system hierarchy before it starts moving data. Pipe the file list from find into rsync or use the –files-from option to keep the initial load lightweight.

find /source/directory -type f | rsync -avz --files-from=- / /destination/

⚙️ Why It Works

By explicitly feeding the file list to rsync, you bypass the massive pre-scan phase that kills performance on deep, bloated directory trees. It starts transferring almost immediately instead of buffering the whole structure into RAM first.

🚀 Pro-Tip: Use –remove-source-files if you are moving data rather than backing it up to keep the source from blowing up your disk space.

Linux Tips & Tricks | © ngelinux.com | 7/15/2026