Backup & Recovery (Rsync/Tar/Dd)

Stop rsync from choking on massive file lists with memory errors

🧩 The Challenge

Dealing with a directory containing millions of small files makes rsync crawl and eventually dump a memory error. It feels like the tool just gives up before it even starts copying anything.

💡 The Fix

Use the file-list-from option to feed the source paths to the process in chunks, which keeps the memory usage low and stable. It saves you from having to break the copy into dozens of manually managed sub-directories.

find /source/path -type f > filelist.txt && rsync -av --files-from=filelist.txt / /destination/path

⚙️ Why It Works

By offloading the file enumeration to the find utility, you bypass the internal rsync logic that attempts to load the entire recursive directory tree into RAM at once. You get the same sync result without the shell hanging on a memory allocation failure.

🚀 Pro-Tip: Add the -P flag to that rsync command so you can resume if the connection drops halfway through the transfer.

Linux Tips & Tricks | © ngelinux.com | 9/4/2026