How to copy files and keep it in sync using rsync command in Linux ?

In this post, we will look at an interesting tip to keep two directories in sync with each other.

It helps us to take backup of a directory and perform incremental backup from time to time.

The differential incremental backup saves a lot of time and helps us to easily keep our data backup.

1. Introduction of rsync utility is Linux
rsync or remote sync is an utility to sync our data from one location to any other local/remote location and helps us to perform differential sync time to time to keep it updated.

It helps in efficient data transfer and file synchronization.

rsync copies the data by comparing modification time(s) and size(s) of all files.
It helps us to minimize the network usage and hence mostly preferred for remote data copy and hence its name “Remote SYNC”

rsync utility usually available on all Linux systems in default install and is licensed under GNU GPL.

2. Syntax

$ rsync [options] source destination

Most common Options as seen in man page:
    -v : verbose.
    -r : copies data recursively however it don’t preserve timestamps and permissions.
    -a : archive mode, it copies files recursively and preserves symbolic links, permissions, timestamps and user & group ownerships.
    -z : compress file data and then transfer, saves network bandwidth.
    -l, --links, copy symlinks as symlinks
    -L, --copy-links, transform symlink into referent file/dir
    -h : human-readable format.
    -d : transfer directories without recursing 
    -e : specify the ssh as remote shell.

 

3. Copy one location to another.

### Create Source directory
[root@ngelinux ~]# mkdir dir1
[root@ngelinux ~]# mkdir dir1/first dir1/second
[root@ngelinux ~]# echo "hello" > dir1/file1
[root@ngelinux ~]# echo "hello" > dir1/file2
[root@ngelinux ~]# echo "hello" > dir1/file3
[root@ngelinux ~]# ln -s /etc/passwd dir1/passwd

### Select destination path
[root@ngelinux ~]# mkdir dir2

### Copy the data
[root@ngelinux ~]# rsync -vrhL dir1 dir2/
sending incremental file list
dir1/
dir1/file1
dir1/file2
dir1/file3
dir1/passwd
dir1/first/
dir1/second/
sent 2.96K bytes  received 104 bytes  6.12K bytes/sec
total size is 2.59K  speedup is 0.85

### We can see below -L copies the symlink file also.
[root@ngelinux ~]# ls -l dir1/passwd dir2/dir1/passwd 
lrwxrwxrwx. 1 root root   11 Apr  7 01:11 dir1/passwd -> /etc/passwd
-rw-r--r--. 1 root root 2556 Apr  7 01:11 dir2/dir1/passwd
[root@ngelinux ~]# 

 

4. Make changes and sync the new files.

### Change a file and view timestamps of old backup
[root@ngelinux ~]# echo "new content" > dir1/file2 
[root@ngelinux ~]# ls -l dir2/dir1/
total 16
-rw-r--r--. 1 root root   12 Apr  7 01:21 file1
-rw-r--r--. 1 root root   17 Apr  7 01:21 file2
-rw-r--r--. 1 root root   12 Apr  7 01:21 file3
drwxr-xr-x. 2 root root    6 Apr  7 01:11 first
-rw-r--r--. 1 root root 2556 Apr  7 01:17 passwd
drwxr-xr-x. 2 root root    6 Apr  7 01:11 second

### sync dir1 into dir2 with -u option to update only changed files
[root@ngelinux ~]# rsync -vru dir1 dir2/
sending incremental file list
skipping non-regular file "dir1/passwd"
dir1/file2
sent 248 bytes  received 82 bytes  660.00 bytes/sec
total size is 52  speedup is 0.16

[root@ngelinux ~]# ls -l dir2/dir1/
total 16
-rw-r--r--. 1 root root   12 Apr  7 01:21 file1
-rw-r--r--. 1 root root   17 Apr  7 01:24 file2
-rw-r--r--. 1 root root   12 Apr  7 01:21 file3
drwxr-xr-x. 2 root root    6 Apr  7 01:11 first
-rw-r--r--. 1 root root 2556 Apr  7 01:17 passwd
drwxr-xr-x. 2 root root    6 Apr  7 01:11 second
[root@ngelinux ~]# 

 

5. Command to use while copying data remotely and to sync it.

### Copy the data, see we have used the option
### -z to compress and transfer the data here.

[root@ngelinux ~]# rsync -vaLhz dir1 root@172.21.49.239:/tmp/
The authenticity of host '172.21.49.239 (172.21.49.239)' can't be established.
ECDSA key fingerprint is SHA256:mamw1Ja6pL5iT2XCvkURvY7bBLv+THGYOon8c93kCg8.
ECDSA key fingerprint is MD5:5b:3c:f9:0f:0e:b7:f5:51:72:76:b4:9f:4e:d0:d7:73.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.21.49.239' (ECDSA) to the list of known hosts.
root@172.21.49.239's password: 
sending incremental file list
dir1/
dir1/file1
dir1/file2
dir1/file3
dir1/passwd
dir1/first/
dir1/second/
sent 2.96K bytes  received 104 bytes  292.29 bytes/sec
total size is 2.60K  speedup is 0.85

### Sync the copied data.
[root@ngelinux ~]# rsync -vaLhzu dir1 root@172.21.49.239:/tmp/
root@172.21.49.239's password: 
sending incremental file list

sent 187 bytes  received 19 bytes  27.47 bytes/sec
total size is 2.60K  speedup is 12.61
[root@ngelinux ~]#

 

6. Skip files/directories during transfer.

### Exclude a file
[root@ngelinux ~]# rsync -vaLhz --exclude 'file2' dir1 root@172.21.49.239:/tmp/
root@172.21.49.239's password: 
sending incremental file list
dir1/
dir1/file1
dir1/file3
dir1/passwd
dir1/first/
dir1/second/
sent 1.31K bytes  received 85 bytes  186.67 bytes/sec
total size is 2.58K  speedup is 1.84
[root@ngelinux ~]# 

### Exclude a directory
[root@ngelinux ~]# rsync -vaLhz --exclude 'first' dir1 root@172.21.49.239:/tmp/
root@172.21.49.239's password: 
sending incremental file list
dir1/
dir1/file1
dir1/file2
dir1/file3
dir1/passwd
dir1/second/
sent 1.36K bytes  received 100 bytes  172.35 bytes/sec
total size is 2.60K  speedup is 1.77
[root@ngelinux ~]# 

### Exclude files based on pattern.
[root@ngelinux ~]# rsync -vaLhz --exclude 'file*' dir1 root@172.21.49.239:/tmp/
root@172.21.49.239's password: 
sending incremental file list
dir1/
dir1/passwd
dir1/first/
dir1/second/
sent 1.17K bytes  received 47 bytes  162.67 bytes/sec
total size is 2.56K  speedup is 2.10
[root@ngelinux ~]# 


### Exclude multiple files of different types
[root@ngelinux ~]# rsync -vaLhz --exclude 'file2' --exclude "pass*"  dir1 root@172.21.49.239:/tmp/
root@172.21.49.239's password: 
sending incremental file list
dir1/
dir1/file1
dir1/file3
dir1/first/
dir1/second/
sent 261 bytes  received 66 bytes  43.60 bytes/sec
total size is 24  speedup is 0.07
[root@ngelinux ~]# 

 

Hence in the above post, we have seen all common options of rsync command, how to use it to copy data, sync data, exclude files, include symlinks’ files.

However there are many other options available which can be seen in its man page.

Leave a Reply

avatar
  Subscribe  
Notify of