Backup & Recovery (Rsync/Tar/Dd)
Speed up your massive rsync jobs with a bit of parallelism
🧩 The Challenge
Dealing with a massive directory full of millions of tiny files is a total nightmare when you’re trying to back it up. I’ve wasted entire days watching rsync crawl through a single file stream while my network pipe sat completely bored and idle.
đź’ˇ The Fix
Throw away the basic rsync command and pipe your file list into xargs to fire off multiple concurrent rsync processes. It’ll saturate your bandwidth and finish the transfer in a fraction of the time.
find /source/dir -maxdepth 1 -type f | xargs -P 8 -n 100 -I {} rsync -avz {} /dest/dir/⚙️ Why It Works
Breaking the task into chunks and firing them off in parallel lets you overcome the single-threaded bottleneck that keeps your CPU and network waiting around for no reason. Using the -P flag in xargs is the secret sauce here because it spawns multiple workers to handle the heavy lifting.
🚀 Pro-Tip: Keep your process count lower than the number of available CPU cores so you don’t end up trashing your disk I/O performance.
Linux Tips & Tricks | © ngelinux.com | 10/2/2026