Backup & Recovery (Rsync/Tar/Dd)

Stop burning your entire evening waiting for rsync to build its file list

🧩 The Challenge

You know that feeling when you trigger a massive rsync job across a network share with millions of small files, and the prompt just hangs there forever? It’s soul-crushing watching a blinking cursor while the server chokes on memory just trying to build the initial file list.

💡 The Fix

Start using the –files-from option with a pre-generated list of files so rsync can jump straight into the data transfer without wasting minutes or hours walking the tree. It keeps your memory usage predictable and prevents those annoying timeouts on the source side.

find /path/to/source -type f > file_list.txt
rsync -avz --files-from=file_list.txt /path/to/source/ destination/

⚙️ Why It Works

By offloading the file discovery to find, you decouple the heavy directory traversal from the transfer logic. The kernel handles the listing phase much more efficiently as a separate pass, and rsync stops acting like it’s trying to hold the entire file metadata tree in RAM at once.

🚀 Pro-Tip: Pipe that find command through grep or awk first if you need to exclude specific patterns before the transfer starts.

Linux Tips & Tricks | © ngelinux.com | 11/14/2026