MENU
| Whimsical Blog
  1. Home
  2. education
  3. What is rclone for cloud storage?

What is rclone for cloud storage?

education
mamushi

rclone is an open source tool for synchronizing and managing files across various cloud storage services using the command line. rclone supports many cloud storage providers, including Google Drive, Amazon S3, Dropbox, OneDrive, etc. It is available for multiple platforms, including Linux, Windows, and macOS.

  1. 1Fichier
  2. Akamai Netstorage
  3. Alibaba Cloud (Aliyun) Object Storage System (OSS)
  4. Amazon Drive (See note)
  5. Amazon S3
  6. Backblaze B2
  7. Box
  8. Ceph
  9. China Mobile Ecloud Elastic Object Storage (EOS)
  10. Arvan Cloud Object Storage (AOS)
  11. Citrix ShareFile
  12. Cloudflare R2
  13. DigitalOcean Spaces
  14. Digi Storage
  15. Dreamhost
  16. Dropbox
  17. Enterprise File Fabric
  18. Fastmail Files
  19. FTP
  20. Google Cloud Storage
  21. Google Drive
  22. Google Photos
  23. HDFS
  24. Hetzner Storage Box
  25. HiDrive
  26. HTTP
  27. ImageKit
  28. Internet Archive
  29. Jottacloud
  30. IBM COS S3
  31. IDrive e2
  32. IONOS Cloud
  33. Koofr
  34. Levvia Object Storage
  35. Liara Object Storage
  36. Linkbox
  37. Linode Object Storage
  38. Mail.ru Cloud
  39. Memset Memstore
  40. Mega
  41. Memory
  42. Microsoft Azure Blob Storage
  43. Microsoft Azure Files Storage
  44. Microsoft OneDrive
  45. Nextcloud
  46. OVH
  47. Blomp Cloud Storage
  48. OpenDrive
  49. OpenStack Swift
  50. Oracle Cloud Storage Swift
  51. Oracle Object Storage
  52. ownCloud
  53. pCloud
  54. Petabox
  55. PikPak
  56. premiumize.me
  57. put.io
  58. Proton Drive
  59. QingStor
  60. Qiniu Cloud Object Storage (Kodo)
  61. Quatrix by Maytech
  62. Rackspace Cloud Files
  63. rsync.net
  64. Scaleway
  65. Seafile
  66. Seagate Lyve Cloud
  67. SeaweedFS
  68. SFTP
  69. Sia
  70. SMB / CIFS
  71. StackPath
  72. Storj
  73. Synology
  74. SugarSync
  75. Tencent Cloud Object Storage (COS)
  76. Uptobox
  77. Wasabi
  78. WebDAV
  79. Yandex Disk
  80. Zoho WorkDrive
  81. The local filesystem

The main features of Zoho WorkDrive include

  • Data Synchronization: Easily synchronize data between local files and cloud storage, or between different cloud storage services.
  • Backup: Can be used to back up files on a regular basis and can be fine-tuned through command line options.
  • File Transfer Optimization: Includes features designed to increase transfer speed and reduce the amount of data transferred.
  • Security: Data can be transferred and stored in encrypted form.
  • Multifunctionality: supports many operations such as copy, move, delete, size check, list view, etc.

The basic steps to use rclone are as follows

  1. Installation: Download rclone from the official website and install it for your operating system.
  2. Configuration: Use the rclone config command to configure the cloud storage provider you wish to use. This may include entering API keys and setting permissions.
  3. Perform operations: Perform any necessary operations from the command line, such as synchronizing, copying, or moving files. For example, rclone copy /path/to/local/folder remote:folder copies files from a local folder to a folder in the cloud.

While it is important to become familiar with command line operations when using rclone, it is a very powerful and flexible tool that is useful in many situations once learned. For more information about each command and its options, you may want to refer to the official rclone documentation or the help command ( rclone help ).

Let’s check it out in practice. At first, download the Windows version for clarity.

https://rclone.org/

After downloading the file, unzip it and place it in an appropriate folder. Since we will be using commands, create a folder directly under the C drive and make sure that some of the unzipped files are in this folder. Set up the cloud storage with the following command. The cloud storage to be checked is yandex disk.

rclone config

There should be nothing, so select “n” to connect.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/ s/ q> n

Next, give this configuration a name (alias): test.

Enter name for new remote.
name> test

You will see a lot of options, enter 53 or yandex as you want to use a Yandex Disk.

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ 1 / 1Fichier
2 / Akamai NetStorage
\ 2 / Akamai NetStorage
3 / Alias for an existing remote
\ alias
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\cH3}6 / Backblaze B2
6 / Backblaze B2
\ \b2
7 / Better checksums for other remotes
\ (hasher)
8 / Box
\ (box)
9 / Cache a remote
\ (cache)
10 / Citrix Sharefile
\ (sharefile)
11 / Combine several remotes into one
\ (combine)
12 / Compress a remote
\ (compress)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
18 / Google Drive
\ (drive)
19 / Google Photos
\ (google photos)
20 / HTTP
\ (http)
21 / Hadoop distributed file system
\ (hdfs)
22 / HiDrive
\ \blog}hidrive
23 / ImageKit.io
\ (imagekit)
24 / In memory object storage system.
\ \ (memory)
25 / Internet Archive
\ \{internetarchive)
26 / Jottacloud
\ jottacloud
27 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ \ (koofr)
28 / Linkbox
\ (linkbox)
29 / Local Disk
\ \ (local)
30 / Mail.ru Cloud
\ (mailru)
31 / Mega
\ mega
32 / Microsoft Azure Blob Storage
\ (azureblob)
33 / Microsoft Azure Files
\files
34 / Microsoft OneDrive
\(onedrive)
35 / OpenDrive
\ (opendrive)
36 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
\ (swift)
37 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
38 / Pcloud
\ (pcloud)
39 / PikPak
\ (pikpak)
40 / Proton Drive
\ \blur} (protondrive)
41 / Put.io
\ (putio)
42 / QingCloud Object Storage
\ (qingstor)
43 / Quatrix by Maytech
\{{quatrix)
44 / SMB / CIFS
\ (smb)
45 / SSH/SFTP
\ (sftp)
46 / Sia Decentralized Cloud
\ (sia)
47 / Storj Decentralized Cloud Storage
\ (storj)
48 / Sugarsync
\ (sugarsync)
49 / Transparently chunk/split large files
\ 49 / Transparently chunk/split large files
50 / Union merges the contents of several upstream fs
\{union)
51 / Uptobox
\ (uptobox)
52 / WebDAV
\ (webdav)
53 / Yandex Disk
\{yandex)
54 / Zoho
\ \ \foo.me
55 / premiumize.me
\ \ \f009} (premiumizeme)
56 / seafile
\ \foo.me} (seafile)
Storage> 53

Enter nothing for Client ID or Client Secret and press Enter. When asked if you want to use a browser, select y.

Option client_id.
OAuth Client Id.
Leave blank normally.
Press Enter to leave empty.
Option client_id>

Option client_secret.
OAuth Client Secret.
Leave blank normally.
Press Enter to leave empty.
client_secret>

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Use web browser to automatically authenticate rclone with remote?

  • Say Y if the machine running rclone has a web browser you can use
  • Say N if running rclone on a (remote) machine without web browser access
    If not sure try Y. If Y failed, try N.

Y) Yes (default)
n) No
y/n> y

Then your browser will start and the yandex login screen will appear. If all goes well, you should see something like the following

All done.
Please go back to rclone.

From here, you can use commands to manipulate the storage.

rclone lsd test:
rclone mkdir test:dic1
rclone ls test:dic1
rclone sync --interactive c:\temp1 test:dic1

1. rclone lsd test:

This command lists the directories (folders) in the root directory of test:remote (the alias of the storage service configured with rclone). lsd stands for “list directories”, which means that no files are displayed, only directories (folders) is listed.

2. rclone mkdir test:dic1

The mkdir command creates a new directory (named dic1 in this example) on the specified remote storage. In this example, the dic1 directory is created on test:remote.

3. rclone ls test:dic1

The ls command lists the files in a given directory. This command lists the files in the test:dic1 directory. However, if dic1 is newly created and nothing has been uploaded, nothing may be displayed.

4. rclone sync --interactive c:\temp1 test:dic1

The sync command synchronizes files between the source (in this example, the local c:\temp1 directory) and the destination (the test:dic1 directory).-The -interactive option causes the command to ask for user confirmation before deleting a file. This command will copy all files and directories in c:\temp1 to test:dic1 so that the contents of both locations match. Note that sync deletes files that exist only on the destination, so be careful when using it.

These commands have been used to illustrate the basic file manipulation and synchronization process using rclone. Because the sync command is particularly useful, it is important to familiarize yourself with the options and implications before executing it.

The Windows version of rclone allows storage authorization via the GUI, making it relatively easy to complete the configuration. However, command line-based configuration, especially when accessing remote cloud storage, may require authentication information such as API tokens or access keys.

The rclone configuration process involves the following steps

  1. Running rclone config: Running this command starts the rclone configuration wizard.
  2. Select a storage provider: From the list of supported storage providers, select the provider you wish to configure.
  3. Enter authentication information: Depending on the provider, you may need to enter information such as API key, access token, client ID, and client secret. This information can be obtained from the provider’s developer console or API management page.
  4. Authentication Process: After entering the credentials, rclone may open a browser to complete the authentication process. This step requires the user to log in to the cloud storage account and grant rclone access.

While the command line setup may seem a bit complicated, most users can succeed if they enter their credentials correctly and follow the instructions. Also, once you have completed the setup, you can then easily sync and back up your files from the command line.

Check with OneDrive as well. Now install rclone on ubuntu22.04 with SSH connection. We want to install the latest version so we will get the link to the file to download.

If you install rclone using sudo apt install rclone on Ubuntu, the version of the repository may be out of date. For this reason, the recommended way to install the latest version is to download and install the .deb package directly from the official rclone website.

The sudo dpkg -i command is used to install locally downloaded .deb package files. However, it is not possible to execute the dpkg -i command by specifying the URL directly. Instead, you must first download the .deb file using the wget orcurl command and then install it with dpkg -i. Here are the steps

  1. Download the .deb file: wget
    wget https://downloads.rclone.org/v1.65.2/rclone-v1.65.2-linux-amd64.deb

    The URL above is an example, please go to rclone’s download page to get the exact link for the latest or required version.
  2. Install the .deb package: sudo dpkg -i
    sudo dpkg -i rclone-v1.65.2-linux-amd64.deb

    Replace this command with the name of the package file you downloaded.

If you get an error about dependencies, you can run sudo apt -f install to resolve the missing dependencies.

The process is the same as before. When asked if you want to use a browser, select n this time.

rclone config

Option region.
Choose national cloud region for OneDrive.
Choose a number from below, or type in your own string value.
Press Enter for the default (global).
1 / Microsoft Cloud Global
\ Press Enter for the default (global). 2 / Microsoft Cloud for US Government
2 / Microsoft Cloud for US Government
\ 2 / Microsoft Cloud for US Government
3 / Microsoft Cloud Germany
\ (de)
4 / Azure and Office 365 operated by Vnet Group in China
\ (cn)
region> 1

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Use web browser to automatically authenticate rclone with remote?

  • Say Y if the machine running rclone has a web browser you can use
  • Say N if running rclone on a (remote) machine without web browser access
    If not sure try Y. If Y failed, try N.

Y) Yes (default)
n) No
y/n> n

Option config_token.
For this to work, you will need rclone available on a machine that has
For this to work, you will need rclone available on a machine that has a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended): rclone
rclone authorize “onedrive”.
Then paste the result.
Enter a value.
config_token>

If you are configuring rclone on a remote machine using an SSH connection and the remote machine does not have a web browser installed (or access to one), then selecting n is the correct choice.

By making this choice, rclone will suggest a way to proceed with authentication without the use of a web browser. Specifically, you must run a command like rclone authorize "onedrive" on the local machine (the machine where the web browser is available) and enter the resulting authentication token at the config_token> prompt on the remote machine The “onedrive” command will be used to authenticate the remote machine. Since we have just installed the Windows version, we will use this. Type the displayed rclone authorize "onedrive" at the command prompt and you will see a long string of characters.

C:\rclone>rclone authorize "onedrive"
2024/02/15 20:08:09 NOTICE: If your browser doesn’t open automatically go to the following link: http://127.0.0.1:53682/auth?state=IRGcdby8T8u_ pPJa-dX19Q
2024/02/15 20:08:09 NOTICE: Log in and authorize rclone for access
2024/02/15 20:08:09 NOTICE: Waiting for code…
2024/02/15 20:08:59 NOTICE: Got code
Paste the following into your remote machine —>
{“access_token”: “EwBwA8l6BAAUs5 HQn0N h2FxWzLS31ZgQVuHsYAAfGHM/cK5wvWkfFzU9A0X5/1DqjxLSL0mbfQUVqLEvvawlE3b8J83A30TS/ Y
Omitted.
Cj! zbiplyDpWjrZNwdrIhmgUknYApZ6BnTBKbD4sxj1w6t3EoZ8iQz2SVJwyuTN9MpAzqO6jcIX2G6mGIYB0BddwwAoVqpn8VES26Dinu3aGWk3EgnWx5brFu3 jumAiFBaAVF8ADBIjmno4Hjvwfwfd5hpgJFyV2dEWYfUyt1gMQqdrTy4k6″, “expiry”: “2024-02-15T21:09:01.627858 09:00”}
<—End paste

This process is required to authenticate to the cloud storage provider (OneDrive in this example) when setting up rclone on the remote machine. By obtaining an authentication token on the local machine and manually entering it into the rclone configuration on the remote machine, access to the cloud storage can be established from the remote machine.

This displayed token information is the authentication token used to authenticate access to the One Drive. This must be applied to the rclone configuration on the remote machine (SSH connected CUI environment). Proceed as follows

  1. Copy all the token information in JSON format starting with ( { "access_token": "EwBwA8l6.... ) in JSON format as displayed above. All information displayed, including omitted parts, is required.
  2. Return to the SSH-connected remote machine and paste this token information into the location where you are continuing the rclone configuration process. Specifically, paste the copied token information where the config_token> prompt appears and press Enter.

config_token> {“access_token”: “EwBwA8l6BAAUs5 HQn0N h2FxWzLS31ZgQVuHsYAAfGHM/cK5wvWkfFzU9Cj!zbiplyDpWjrZNwdrIhmgUknYApZ6BnTBKbD4 sxj1w6t3EoZ8iQz2SVJwyuTN9MpAzqO6jcIX2G
Omitted
zHxUPv4ZbPxnVZS!U8xGZKE4LRajj5brFu3jumAiFBaAVF8ADBIjmno4Hjvwfwfd5hpgJFyV2dEWYfUyt1gMQqdrTy4k6 “, “expiry”: “2024-02-15T21:09:01. 627858 09:00”}

Option config_type.
Type of connection
Choose a number from below, or type in an existing string value.
Press Enter for the default (onedrive).
1 / OneDrive Personal or Business
\ 1 / OneDrive Personal or Business
2 / Root Sharepoint site
\ 2 / Root Sharepoint site
/ Sharepoint site name or URL
3 / E.g. mysite or https://contoso.sharepoint.com/sites/mysite
\ (url)
4 / Search for a Sharepoint site
\ (search)
5 / Type in driveID (advanced)
\ (driveid)
6 / Type in SiteID (advanced)
\ (siteid)
/ Sharepoint server-relative path (advanced)
7 | E.g. /teams/hr
\ (path)
config_type> 1

Option config_driveid.
Select drive you want to use
Choose a number from below, or type in your own string value.
Press Enter for the default (10aeefd6b862db5f).
1 / (personal)
\ Press Enter for the default (10aeefd6b862db5f).
config_driveid> 1

Drive OK?

Found drive “root” of type “personal”
URL: https://onedrive.live.com/?cid=10aeefd6b862db5f

y) Yes (default)
n) No
y/n> y

Configuration complete.

This will give rclone on the remote machine the credentials to access the One Drive and complete the configuration. This credentials can then be used to execute commands to access files and folders on the One Drive.

Memo
Example: rclone copy oned:youtube /home/mamushi/test
When using the rclone copy command, even if the destination directory ( /home/mamushi/test in this example) does not exist, rclone will automatically create it before performing the copy operation. In other words, when performing a copy operation from oned:youtube to /home/mamushi/test, it does not matter if the test directory does not exist beforehand.

This command copies files from the youtube directory on the OneDrive (remote labeled oned ) to the /home/mamushi/test directory on the local system. If the /home/mamushi/test directory does not exist, rclone will create this directory and then copy the files.

rclone is a very flexible and useful tool, and this automatic directory creation feature makes it easier for users to synchronize and back up their files. If the destination directory structure is complex, rclone will automatically create all necessary intermediate directories.

If rclone automatically creates a directory when copying directly to a new directory, rclone will set the appropriate permissions on that directory to ensure a smooth copy operation. On the other hand, if a permission error occurs when trying to copy to an existing directory, it is possible that the permission settings that directory has conflict with rclone’s operation.

Permission errors are particularly common on UNIX-like operating systems (such as Linux and macOS). This is because part of the file system security model controls whether files and directories are allowed to be read, written, or executed by certain users or groups. when attempting to access an existing directory using rclone, it is important to check the following points The following points are important to check when trying to access an existing directory using rclone

  • Directory permissions: Use the ls -l command to check the permissions of a directory. If necessary, the permissions can be changed using the chmod command.
  • User performing the operation: Verify that the user to whom rclone is run has the appropriate access rights to the destination directory. If missing, the owner of the directory must be changed or the permissions adjusted.
education
Please share if you like it!
TOC