Backup & Recovery (Rsync/Tar/Dd)

Stop rsync from choking when you have millions of tiny files

🧩 The Challenge

You ever try to sync a webroot with half a million small images and watch rsync spend twenty minutes just building the file list while your disk I/O dies? It’s miserable, and I’ve definitely walked away to get coffee only to come back and find the process hasn’t actually copied a single byte yet.

💡 The Fix

Don’t let rsync try to scan everything at once. Use a find-based approach to chunk the work or, even better, tell rsync to use the –files-from option so it isn’t wasting cycles traversing a massive directory tree on both ends.

find /source/path -type f > filelist.txt
rsync -avz --files-from=filelist.txt /source/path/ /destination/path/

⚙️ Why It Works

By passing a static list of files directly to the utility, you skip the recursive directory scan that burns through memory and CPU when you’re dealing with massive, flat file structures. It turns a recursive nightmare into a targeted transfer.

🚀 Pro-Tip: Add –delete-after to your rsync command if you need to clean up the destination, but be careful because it will prune files that aren’t in your explicit list.

Linux Tips & Tricks | © ngelinux.com | 8/18/2026