How to copy files and keep it in sync using rsync command in Linux ?
In this post, we will look at an interesting tip to keep two directories in sync with each other.
It helps us to take backup of a directory and perform incremental backup from time to time.
The differential incremental backup saves a lot of time and helps us to easily keep our data backup.
1. Introduction of rsync utility is Linux
rsync or remote sync is an utility to sync our data from one location to any other local/remote location and helps us to perform differential sync time to time to keep it updated.
It helps in efficient data transfer and file synchronization.
rsync copies the data by comparing modification time(s) and size(s) of all files.
It helps us to minimize the network usage and hence mostly preferred for remote data copy and hence its name “Remote SYNC”
rsync utility usually available on all Linux systems in default install and is licensed under GNU GPL.
2. Syntax
$ rsync [options] source destination
Most common Options as seen in man page:
-v : verbose.
-r : copies data recursively however it don’t preserve timestamps and permissions.
-a : archive mode, it copies files recursively and preserves symbolic links, permissions, timestamps and user & group ownerships.
-z : compress file data and then transfer, saves network bandwidth.
-l, --links, copy symlinks as symlinks
-L, --copy-links, transform symlink into referent file/dir
-h : human-readable format.
-d : transfer directories without recursing
-e : specify the ssh as remote shell.
3. Copy one location to another.
### Create Source directory [root@ngelinux ~]# mkdir dir1 [root@ngelinux ~]# mkdir dir1/first dir1/second [root@ngelinux ~]# echo "hello" > dir1/file1 [root@ngelinux ~]# echo "hello" > dir1/file2 [root@ngelinux ~]# echo "hello" > dir1/file3 [root@ngelinux ~]# ln -s /etc/passwd dir1/passwd ### Select destination path [root@ngelinux ~]# mkdir dir2 ### Copy the data [root@ngelinux ~]# rsync -vrhL dir1 dir2/ sending incremental file list dir1/ dir1/file1 dir1/file2 dir1/file3 dir1/passwd dir1/first/ dir1/second/ sent 2.96K bytes received 104 bytes 6.12K bytes/sec total size is 2.59K speedup is 0.85 ### We can see below -L copies the symlink file also. [root@ngelinux ~]# ls -l dir1/passwd dir2/dir1/passwd lrwxrwxrwx. 1 root root 11 Apr 7 01:11 dir1/passwd -> /etc/passwd -rw-r--r--. 1 root root 2556 Apr 7 01:11 dir2/dir1/passwd [root@ngelinux ~]#
4. Make changes and sync the new files.
### Change a file and view timestamps of old backup [root@ngelinux ~]# echo "new content" > dir1/file2 [root@ngelinux ~]# ls -l dir2/dir1/ total 16 -rw-r--r--. 1 root root 12 Apr 7 01:21 file1 -rw-r--r--. 1 root root 17 Apr 7 01:21 file2 -rw-r--r--. 1 root root 12 Apr 7 01:21 file3 drwxr-xr-x. 2 root root 6 Apr 7 01:11 first -rw-r--r--. 1 root root 2556 Apr 7 01:17 passwd drwxr-xr-x. 2 root root 6 Apr 7 01:11 second ### sync dir1 into dir2 with -u option to update only changed files [root@ngelinux ~]# rsync -vru dir1 dir2/ sending incremental file list skipping non-regular file "dir1/passwd" dir1/file2 sent 248 bytes received 82 bytes 660.00 bytes/sec total size is 52 speedup is 0.16 [root@ngelinux ~]# ls -l dir2/dir1/ total 16 -rw-r--r--. 1 root root 12 Apr 7 01:21 file1 -rw-r--r--. 1 root root 17 Apr 7 01:24 file2 -rw-r--r--. 1 root root 12 Apr 7 01:21 file3 drwxr-xr-x. 2 root root 6 Apr 7 01:11 first -rw-r--r--. 1 root root 2556 Apr 7 01:17 passwd drwxr-xr-x. 2 root root 6 Apr 7 01:11 second [root@ngelinux ~]#
5. Command to use while copying data remotely and to sync it.
### Copy the data, see we have used the option ### -z to compress and transfer the data here. [root@ngelinux ~]# rsync -vaLhz dir1 root@172.21.49.239:/tmp/ The authenticity of host '172.21.49.239 (172.21.49.239)' can't be established. ECDSA key fingerprint is SHA256:mamw1Ja6pL5iT2XCvkURvY7bBLv+THGYOon8c93kCg8. ECDSA key fingerprint is MD5:5b:3c:f9:0f:0e:b7:f5:51:72:76:b4:9f:4e:d0:d7:73. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.21.49.239' (ECDSA) to the list of known hosts. root@172.21.49.239's password: sending incremental file list dir1/ dir1/file1 dir1/file2 dir1/file3 dir1/passwd dir1/first/ dir1/second/ sent 2.96K bytes received 104 bytes 292.29 bytes/sec total size is 2.60K speedup is 0.85 ### Sync the copied data. [root@ngelinux ~]# rsync -vaLhzu dir1 root@172.21.49.239:/tmp/ root@172.21.49.239's password: sending incremental file list sent 187 bytes received 19 bytes 27.47 bytes/sec total size is 2.60K speedup is 12.61 [root@ngelinux ~]#
6. Skip files/directories during transfer.
### Exclude a file [root@ngelinux ~]# rsync -vaLhz --exclude 'file2' dir1 root@172.21.49.239:/tmp/ root@172.21.49.239's password: sending incremental file list dir1/ dir1/file1 dir1/file3 dir1/passwd dir1/first/ dir1/second/ sent 1.31K bytes received 85 bytes 186.67 bytes/sec total size is 2.58K speedup is 1.84 [root@ngelinux ~]# ### Exclude a directory [root@ngelinux ~]# rsync -vaLhz --exclude 'first' dir1 root@172.21.49.239:/tmp/ root@172.21.49.239's password: sending incremental file list dir1/ dir1/file1 dir1/file2 dir1/file3 dir1/passwd dir1/second/ sent 1.36K bytes received 100 bytes 172.35 bytes/sec total size is 2.60K speedup is 1.77 [root@ngelinux ~]# ### Exclude files based on pattern. [root@ngelinux ~]# rsync -vaLhz --exclude 'file*' dir1 root@172.21.49.239:/tmp/ root@172.21.49.239's password: sending incremental file list dir1/ dir1/passwd dir1/first/ dir1/second/ sent 1.17K bytes received 47 bytes 162.67 bytes/sec total size is 2.56K speedup is 2.10 [root@ngelinux ~]# ### Exclude multiple files of different types [root@ngelinux ~]# rsync -vaLhz --exclude 'file2' --exclude "pass*" dir1 root@172.21.49.239:/tmp/ root@172.21.49.239's password: sending incremental file list dir1/ dir1/file1 dir1/file3 dir1/first/ dir1/second/ sent 261 bytes received 66 bytes 43.60 bytes/sec total size is 24 speedup is 0.07 [root@ngelinux ~]#
Hence in the above post, we have seen all common options of rsync command, how to use it to copy data, sync data, exclude files, include symlinks’ files.
However there are many other options available which can be seen in its man page.