Backup & Recovery (Rsync/Tar/Dd)
Stop rsync from choking on millions of tiny files
🧩 The Challenge
Trying to rsync a directory with a few million tiny cache files is a nightmare that will keep you waiting for an hour just to build the file list. I’ve wasted so many afternoons watching a cursor blink while rsync sits there doing nothing but memory bloat.
đź’ˇ The Fix
Don’t let it crawl through the entire file system hierarchy before it starts moving data. Pipe the file list from find into rsync or use the –files-from option to keep the initial load lightweight.
find /source/directory -type f | rsync -avz --files-from=- / /destination/⚙️ Why It Works
By explicitly feeding the file list to rsync, you bypass the massive pre-scan phase that kills performance on deep, bloated directory trees. It starts transferring almost immediately instead of buffering the whole structure into RAM first.
🚀 Pro-Tip: Use –remove-source-files if you are moving data rather than backing it up to keep the source from blowing up your disk space.
Linux Tips & Tricks | © ngelinux.com | 7/15/2026