rsync is a powerful tool for efficiently synchronizing files and directories. Its key features include:
- Transfers only differences, minimizing network load.
- Can increase transfer speed further with compression options.
- Allows both local and remote file transfers.
- Highly flexible for synchronization and backups.
Basic rsync Command
Let’s first take a look at the basic format of the rsync command. The following command is a basic example of transferring files from a local PC to a remote server:
rsync -av /path/to/local/dir/ user@remote:/path/to/remote/dir/
Explanation of Options
-a: Archive mode. Suitable for complete backups as it recursively copies directories and preserves file ownership, permissions, and timestamps.-v: Verbose mode. Displays progress and file names.
Example
If you want to back up the local directory /home/user/documents/ to a remote server, you would use the following command:
rsync -av /home/user/documents/ user@remote:/backup/documents/
This will fully synchronize the local documents directory with the /backup/documents/ directory on the remote server.
Try It Out
rsync -av ubuntu@10.0.0.4:/var/www /tmp
[root@instance-20220518-2006 tmp]# rsync -av ubuntu@10.0.0.4:/var/www /tmp
The authenticity of host ‘10.0.0.4 (10.0.0.4)’ can’t be established.
ECDSA key fingerprint is SHA256:okV6TespGViKlme5EioFCSZIzoDSHdaaT1KXDK3YF/M.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘10.0.0.4’ (ECDSA) to the list of known hosts.
ubuntu@10.0.0.4: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]
However, the error mentioned above occurred.
It seems that you must use the private key from the source of the migration. You can transfer the key using TeraTerm’s SSH SCP feature.
Advanced Usage: Receiving Data from a Remote Server
Next, let’s move on to an advanced use case. If you want to receive files from a remote PC to your local PC, you would use the following command:
rsync -av ubuntu@10.0.0.4:/var/www /tmp
This command copies the /var/www directory from the remote server at 10.0.0.4 to the /tmp directory on your local PC.
Handling Errors
If a common error, like the one mentioned earlier, occurs, it may be due to an issue with the private key.
ubuntu@10.0.0.4: Permission denied (publickey)
In such a case, you can transfer the private key using tools like TeraTerm.
Modifying the rsync Command as Follows
rsync -e 'ssh -i /path/to/private-key' -av ubuntu@10.0.0.4:/var/www /tmp
Here is the actual command. This command uses the specified key to transfer files:
rsync -e 'ssh -i /tmp/ssh-key-2020-12-08.key' -av ubuntu@10.0.0.4:/var/www /tmp
The file transfer was successful. However, an error occurred where some files could not be transferred.
Error: Some Files Could Not Be Transferred (rsync error: code 23)
When using the rsync command to transfer files, the following error may occur:
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1663) [generator=3.1.3]
This error indicates that some files were not transferred correctly for several reasons. The main causes include:
- Permission Issues
If the permissions of the files or directories are not properly set, rsync may not be able to transfer them. For example, this error may occur if the owner is root or if there is no read permission.
Solution: Check the permissions on both the source and destination, and use the chmod command to set the correct permissions.
sudo chmod -R 755 /path/to/directory
- Symbolic Link Issues
If you attempt to transfer files or directories that contain symbolic links, an error may occur if the target of the link cannot be found.
Solution: Either ignore symbolic links or use the correct option to transfer the target of the link. For instance, you can use the -L option to transfer the actual file behind the link.
rsync -avL /path/to/source/ user@remote:/path/to/destination/
- Insufficient Disk Space
If the destination has insufficient disk space, the file transfer may stop midway.
Solution: Check the disk usage on the destination and delete unnecessary files if needed to free up space.
df -h
- Special Attributes of Certain Files
rsync may encounter issues when trying to transfer files with special attributes (e.g., xattr or ACL). These attributes can cause errors during transfer.
Solution: Add the --no-xattrs option to disable the transfer of file attributes and retry.
rsync -av --no-xattrs /path/to/source/ user@remote:/path/to/destination/
Even after trying these solutions, the issue persisted, especially when there were too many directories and files to handle individually. Therefore, I took the following steps.
Here, I propose an alternative method to address situations where there are too many directories or files to resolve each issue manually—compressing the files before transferring them. First, let me explain why this method is effective.
Why Compression is Effective
When there are many files, and issues like symbolic links or permissions make it difficult to resolve each problem individually, you can temporarily bypass these complexities by compressing all the files into a single archive. By transferring everything as a single file, you can avoid multiple issues and transfer files more efficiently.
How to Compress
You can use the tar command to compress an entire directory into a single file as shown below. Compression can also speed up network transfers, especially when dealing with large directory structures.
tar -cvf bk.tar /path/to/directory
This command compresses the specified directory (/path/to/directory) into an archive file named bk.tar.
Compress Before Transferring
Next, transfer the compressed file to the destination. This method treats everything as a single file, preventing errors caused by handling individual files and directories.
rsync -e 'ssh -i /tmp/ssh-key-2020-12-08.key' -av /path/to/bk.tar user@remote:/path/to/destination
Post-Compression Process
On the destination machine, extract the archive to restore the files to their original state. This ensures the integrity of the files and completes the migration.
tar -xvf /path/to/bk.tar -C /path/to/extract/directory
The Command I Actually Entered
tar -cvf bk.tar minokamo.tokyo
The meaning of the above command is that I compressed the minokamo.tokyo directory into a single file before transferring it. By compressing, multiple files and subdirectories are combined into a single file, making the transfer simpler and more efficient.
Compression Method: Using the tar Command
First, use the tar command to archive (compress) the entire minokamo.tokyo directory. Use the following command:
tar -cvf bk.tar minokamo.tokyo
Details of the Command
- tar: A tool to archive files.
- -c: Create a new archive.
- -v: Display the progress (verbose).
- -f: Specify the name of the archive file (in this case,
bk.tar). - minokamo.tokyo: The name of the directory to compress.
By running this command, the entire minokamo.tokyo directory is archived into a single file named bk.tar. This file contains all files and subdirectories within the minokamo.tokyo directory.
Why Compress?
- Efficient Transfer: Instead of transferring many individual files with rsync, compressing everything into a single file makes the transfer smoother and reduces the chance of errors.
- Preserving File Structure: The compressed archive retains the directory structure, file permissions, timestamps, etc., so that everything is restored to its original state after extraction on the destination.
Retrieve the Compressed File (bk.tar) on the Destination PC
rsync -e 'ssh -i /tmp/ssh-key-2020-12-08.key' -av ubuntu@10.0.0.4:/var/www/bk.tar /tmp
If a permission error occurs, ensure the correct permissions are applied to the key. This error did not occur when logged in as root.
sudo chmod 600 ssh-key-2020-12-08.key
Below is the error that occurred because I did not take the precautions mentioned above.
[opc@instance-20220518-2006 tmp]$ rsync -e ‘ssh -i /tmp/ssh-key-2020-12-08.key’ -av ubuntu@10.0.0.4:/var/www/bk.tar /tmp
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
Permissions 0644 for ‘/tmp/ssh-key-2020-12-08.key’ are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key “/tmp/ssh-key-2020-12-08.key”: bad permissions
ubuntu@10.0.0.4: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]
Transferring Compressed Files and Handling Permission Errors
When transferring compressed files to the destination PC, you can use the following rsync command. This method allows for efficient transfer by consolidating individual files.
bashコードをコピーするrsync -e 'ssh -i /tmp/ssh-key-2020-12-08.key' -av ubuntu@10.0.0.4:/var/www/bk.tar /tmp
This command transfers the compressed file bk.tar from the /var/www/ directory on the remote server 10.0.0.4 to the /tmp directory on your local PC.
Handling Permission Errors
However, you may encounter permission errors during the transfer, such as the following:
WARNING: UNPROTECTED PRIVATE KEY FILE! @
Permissions 0644 for ‘/tmp/ssh-key-2020-12-08.key’ are too open.
This private key will be ignored.
Load key “/tmp/ssh-key-2020-12-08.key”: bad permissions
This error indicates that the permissions on the private key are not set correctly.
In SSH, private keys must have restricted permissions to prevent access by other users. If the permissions are too broad, such as 0644, the private key will be ignored, and the connection will fail.
Solution
In this case, set the private key’s permissions to 600, which prevents access by other users. Use the following command to correct the permissions:
sudo chmod 600 /tmp/ssh-key-2020-12-08.key
The chmod 600 command grants read and write permissions only to the file’s owner and denies all access to other users. This ensures that the SSH connection can proceed correctly.
If an Error Occurs
Even after adjusting the permissions, rsync may still not work properly, and it might disconnect partway through the process. Below is an example of an error message you might encounter:
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]
In this case, check the following points:
- Is the private key the correct one?
- Are there any issues with the SSH settings on the server?
- Are the permissions set correctly?
Bonus Article: WordPress Database Backup and Restoration
When migrating WordPress, backing up the database is one of the most important tasks. Since all WordPress posts, user information, and settings are stored in the database, it’s essential to restore this information on the destination server.
Step 1: Back Up the Database on the Source Server
First, create a backup of the database from the source server. Use the following command:
mysql -u root -p
This command will prompt you to enter the password for the MySQL root user. Next, display the list of current databases to confirm which one you need to back up:
show databases;
exit
Once you have confirmed the name of the database to back up, use the mysqldump command to export the database to a file. In this example, we will back up the database named my_db.
mysqldump -u root my_db -p > /tmp/bk.sql
This command backs up the my_db database to a file named /tmp/bk.sql.
Step 2: Transfer the Backup File to the Destination
Next, transfer the backed-up database file to the destination server. You can use the rsync command for efficient file transfer.
rsync -e 'ssh -i /tmp/ssh-key-2020-12-08.key' -av ubuntu@10.0.0.4:/tmp/bk.sql /tmp
This command transfers the /tmp/bk.sql file from the remote server (10.0.0.4) to the /tmp directory on the local machine.
Step 3: Create a Database on the Destination Server
After transferring the backup file to the destination server, create an empty database where the backup will be imported.
mysql -u root -p
create database my_db;
exit
This creates a new database named my_db. You will restore the backup into this database.
Step 4: Import the Database
Next, import the database file transferred from the source server into the newly created database.
mysql -u root -p my_db < /tmp/bk.sql
This command restores the backup from /tmp/bk.sql into the my_db database.
Step 5: Extract and Place Compressed Files
After completing the database restoration, the next step is to transfer the content files. Refer to the examples above for transferring content files. Once you’ve set up the web server and PHP configuration using Docker or a LAMP stack, depending on your destination environment, the WordPress migration will be complete.
In fact, there’s another useful tool I’d like to introduce: rclone.
Let me explain it while comparing it to rsync.
Pros and Cons of rsync
Pros:
- Fast and Efficient Data Transfer: rsync performs differential transfers, ignoring files that already exist and transferring only modified or new files.
- Available for Both Local and Remote: It can be used for file synchronization between local systems or between local and remote servers.
- Detailed Options: It can accurately sync file metadata, such as permissions, timestamps, and symbolic links.
- Secure Communication via SSH: rsync allows for secure data transfer using SSH.
Cons:
- Limited Integration with Cloud Storage: rsync is mainly designed for server-to-server data transfer and lacks direct integration with cloud services like Google Drive or Dropbox.
- Complexity for Beginners: rsync’s many options and settings can be overwhelming for beginners.
Pros and Cons of rclone
Pros:
- Compatibility with Cloud Storage: rclone supports many cloud storage services (e.g., Google Drive, Dropbox, Amazon S3) and makes syncing between clouds or with local storage very easy.
- Encryption Support: rclone makes it simple to encrypt data, ensuring security when uploading to cloud storage.
- Rich Features: Beyond file transfer and synchronization, rclone offers additional features such as restore, backup, checksum verification, and bandwidth control.
- Cross-Platform Support: rclone works seamlessly on Windows, Linux, and macOS, making it easy to use in a wide range of environments.
Cons:
- Not Optimized for Local Transfers: Compared to rsync, rclone is less suited for large-scale local data transfers.
- Limited Secure Transfer Options like SSH: While rclone is optimized for cloud use, it may not be ideal for detailed configurations using SSH.
Other Methods
- scp (Secure Copy): A tool for copying files via ssh, but lacks differential transfer and synchronization features, making it less efficient than rsync or rclone.
- Cloud Sync Tools: Official tools from services like AWS S3 and Google Cloud Storage exist, but rclone is superior when working across multiple cloud services.
Why rsync is Suitable for This Article
In the context of this article, rsync is more suitable.
Reasons:
- If the main purpose is basic file synchronization and transfer, rsync excels in simplicity and efficiency. The article explains migrating WordPress databases and transferring files between servers, where rsync’s differential transfers and SSH connections are highly effective.
- For syncing files between local and remote environments, rsync is highly reliable and can be executed with simple commands. rsync’s strengths in server management and backups make it convenient for beginners performing WordPress migrations.
- Secure data transfer via SSH is essential when handling important data like WordPress content. rsync supports SSH-based transfers, ensuring a high level of security.
When rclone is Suitable
If the article were to cover cloud storage migration or backup (e.g., Google Drive or Amazon S3), rclone would be more useful. However, since this content focuses on server-to-server migration and file synchronization, rsync is more understandable and suitable for beginners.
Conclusion
For the scenarios described in the current article, rsync is the most appropriate tool for file transfers and backups.