Rclone

This guide serves for rclone tool configuration. Rclone is the swiss knife for connection to multiple types of storage. However, the following guides are limited only to Object Storage namely S3 service. The following guides are suited for Windows users as well as Linux users. Using rclone tool for S3 enables the creation of the buckets, syncing folders, uploading and downloading files, and much more.

Downloading and installation of rclone tool

Firstly you need to download and unzip the desired rclone version according to the system you operate.

We strongly recommend to use up-to-date versions of rclone tool available at rclone websites. Using rclone version from system repositories can cause some errors due to the outdated rclone version. In case of a manual installation into the user profile, see below, you can use for an update following command:

rclone selfupdate

Linux - manual installation into the user profile

We need to unzip the rclone archive after download.

unzip rclone-v1.59.1-linux-amd64.zip

Then we need to copy rclone binary file into the pre-prepared bin folder in the user profile/home.

cp ./rclone-v1.59.1-linux-amd64/rclone /home/user/bin/

In the last step, we need to put the path to the rclone binary file into PATH environment variable.

PATH=/home/user/bin:$PATH

Persistent setup

For persistent presence of the rclone binary file path in the PATH variable you can add the following line into .bashrc file:

echo 'PATH=/home/user/bin:$PATH' >> .bashrc

Alternatively you can place the rclone binary file into the system path:

/usr/local/bin

In the end, we can make a quick check of PATH variable, whether the desired path is present.

echo $PATH

rclone selfupdate

Windows - manual installation in to user profile

Firstly you need to prepare bin directory in your user profile, where we will place rclone.exe file. Please prepare the directory in your user profile. You can just put the following command into the file browser:

%USERPROFILE%

In the displayed directory you can click right mouse button -> New –> Directory/Folder. The directory should be named bin. Then move the rclone.exe file into it. The file rclone.exe is present in the unzipped rclone archive, which you downloaded from the rclone websites.

In the next step click to Start (1) and search for Edit the system environment variables (2).

Then click at Environment variables (1) in the displayed window.

In the section User variables for UserXY (1) you will select the line with variable Path (2) and then you will click on the Edit (3) button.

You can add the new path by clicking on the New (1) button and then you have to insert the path to the pre-prepared “bin” folder (2), see below. The setup is then confirmed by clicking on the OK (3) button.

%USERPROFILE%\bin

In the end, you will click OK and Apply.

rclone selfupdate

Basic configuration of rclone

Below you can find the guide for the elementary configuration of rclone tool. Below are two guides. First describes configuration using the command line and second guide describes configuration using the graphical user interface.

Rclone configuration using the command line

Rclone has a configuration wizard, which will guide you step-by-step through the configuration of your S3 data storage.

Command line in Windows and Linux: Windows users need to run Command Prompt and then run the command below. Linux users can open the terminal window and then run the command below.

    rclone config

From the list of options, we will choose New remote via inserting n letter. Then we need to type the name of our connection/storage, e.g. cesnet_s3. Then we will choose the Option Storage, here Amazon S3 Compliant Storage Providers…

In the next step, we need to choose Option provider, here Ceph Object Storage.

Then it is necessary to choose how to enter S3 credentials. Here we choose Enter AWS credentials in the next step.

In the next steps we enter our credentials access_key_id and secret_access_key which we obtained from the administrator or which we generated.

Then we need to select Option region. We will leave this option empty and then continue with Enter.

Then we need to insert Option endpoint according to the data center, where we have generated the credentials, here for example s3.cl2.du.cesnet.cz.

Next is Option location_constraint. We will leave this option empty and then continue with Enter.

Next point si Option acl, here we can either choose Owner gets FULL_CONTROLL, or we can leave this option empty and continue with Enter.

In the next step, we need to choose Option server_side_encryption, here we pick the option None.

Then we can choose Option see_kms_key_id, here we pick the option None.

In the next step we can choose option Edit advanced config, here we choose No, option n.

In the last step, we check the configuration and we will confirm it by typing y letter.

Rclone configuration using graphical user interface

Firstly you need to run the GUI. Windows users need to open Command Prompt and run the command below. Linux users need just open the terminal window and run the command below.

rclone rcd --rc-web-gui

The next steps are identical for Windows and Linux.

After starting up the graphical interface we click in the left menu on the Configs (1) button and then on the Create a New Config (2) button.

In the displayed window, we insert the connection name in our example cesnet_s3cl2 and then we choose the option Amazon S3 Compliant Storage Providers.

In the next step, we need to insert credentials (1) which we obtained from the administrator or which we generated. You have to insert Ceph into the line denoted Choose your S3 provider. Then we need to insert the S3 endpoint address (2). Then we can click Next.

Uploading the data from your local machine

After the configuration, we can start to transfer the data.

In the left menu click on the Explorer (1) button. Then select the name of configuration (2), for example cesnet_s3cl2. Then you click on the Open (3) button. Then there should be a window with the buckets and files from the configured data storage.

If you wish to upload your data then in the displayed window click on upload icon (1). Then you can select the data from your disk or drag and drop them into the window.

Configuration file

Windows config file

[cesnet_s3]
type = s3
provider = Ceph
access_key_id = my-access-key
secret_access_key = my-secret-key
endpoint = s3.cl2.du.cesnet.cz
acl = private

**Linux config file

[cesnet_s3]
type = s3
provider = Ceph
access_key_id = my-access-key
secret_access_key = my-secret-key
endpoint = s3.cl2.du.cesnet.cz
acl = private

Rclone basic controls

All available commands for rclone can be listed using the command

rclone help

Alternatively you can find rclone guide on the rclone websites. Below are described the selected commands to control buckets, directories and files.

Listing buckets and directories

Listing of the available profiles/connections.

rclone listremotes
cesnet_s3_encrypted:
cesnet_s3cl2:
sftp_du4:
sftp_du5:

Listing of buckets of the selected profile/connection.

rclone lsd cesnet_s3cl2:
  -1 2020-11-11 08:53:48        -1 111
  -1 2022-07-28 10:03:20        -1 test

Creation of the bucket, copying, deletion…

Creation of the new bucket.

rclone mkdir cesnet_s3cl2:test-bucket

Deletion of the bucket.

rclone rmdir cesnet_s3cl2:test-bucket

Copying the files.

rclone mkdir cesnet_s3cl2:test-bucket/empty-directory
 
2022/08/24 12:18:36 NOTICE: S3 bucket test-bucket path empty-directory: Warning: running mkdir on a remote which can't have empty directories does nothing

The solution is to use a full path inside the bucket including a non-existing directory during copying. In case you type non-existing directory rlone will create it, see the example below.

Let’s copy the files:

rclone copy /home/user/test_file1.pdf cesnet_s3cl2:test-bucket/new-dir1/new-dir2/

Then we can check the files using ls command, where we can see that the folders have been created. Namely new-dir1 a new-dir2:

rclone ls cesnet_s3cl2:test-bucket
3955866 new-dir1/new-dir2/test_file1.pdf

File deletion.

To delete a particular file, we can use either command deletefile or the command delete to remove all files in the given path.

rclone deletefile cesnet_s3cl2:test-bucket/new-dir1/new-dir2/test_file1.pdf

Directory syncing

To sync the directories you can use the option sync. Synchronization is affecting the content only on the target side, no changes are performed on the source side.

Below is the exemplary command of rclone sync. The command contains recommended options which are described below.

rclone sync --dry-run --progress --fast-list /home/user/source-dir cesnet_s3cl2:test-bucket/

The command above contains several recommended options.

Option dry-run allows performing the dry-run sync with listing the potential changes.

--dry-run

Option progress allows seeing the continuous progress of the sync.

--progress

Option fast-list allows limiting the number of API requests. This option can enhance the transfer of larger datasets. It uses one request to read the information about 1000 objects and store it in the memory.

--fast-list

Option interactive allows interactively deciding which change (on the target data storage) we want to accept or reject.

--interactive

Data integrity checks

Rclone allows testing the integrity of transferred data.

rclone check --fast-list C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync

The command checks the checksums on the source side as well as on the target side. For fast checks you can use the option --size-only, where are checked only file sizes.

rclone check --fast-list --size-only C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync

Configuration and controls of encryted bucket

This section describes the configuration and controls of encrypted buckets using rclone tool. It goes about client-side encryption. Below are the guides for setup using the command line and for setup using the graphical user interface.

Configuration using the command line

Rclone has a wizard that eases the setup of an encrypted bucket.

Windows user needs the Command Prompt tool, where he/she can directly start the rclone configuration using the command below.

Linux user needs just to open the Terminal window and continue with following rclone.

rclone config

On the displayed list of the options, we will select New remote via typing n. Then we will insert the name of our data storage, for instance, cesnet_s3_encrypted. Then we will select Option Storage, and here Encrypt/Decrypt a remote.

In the next step, we have to define Option remote. Here we need to select existing S3 profile/connection and define the name of the bucket where will rclone create the encrypted space. We have to use the format s3-profile:bucket-name.

Then we need to select Option filename_encryption. There we can select Encrypt the filenames alternatively, we can keep it empty if we wish to not encrypt the filenames.

Then we can select Option directory_name_encryption. There we can select Encrypt directory names alternatively, we can keep it empty if we wish to not encrypt the directory names.

In the next step Option password we have to choose an encryption password.

Furthermore, we recommend choosing Option password2. This password will be used as salt for consequencing encryption.

Option Edit advanced config can be skipped, option n.

The configuration is completed now. In the next step, we can confirm the option Keep this encrypted config remote using option y.

The last step is to check the encryption. Firstly we need to list available configurations/connections.

rclone listremotes
cesnet_s3_encrypted:
cesnet_s3cl2:

Then we can using sync command upload three pictures into decrypted bucket.

rclone sync --progress --fast-list /home/user/source-dir cesnet_s3_encrypted:

Now we can list decrypted bucket, where we have uploaded three pictures.

rclone ls cesnet_s3_encrypted:
  256805 DSC_0004.jpg
  337491 DSC_0006.jpg
  251493 DSC_0005.jpg

In the end, we can list the encrypted bucket, where we can see three encrypted files.

  rclone ls cesnet_s3cl2:test-encryption
    256901 1er0np7kppc9jvkt7kr8f9sn90
    337619 cuqqkkhsklbnf1eegkujfkrcl4
    251589 pelqqer8osssa4k8uon95a4o6c

Configuration of the encrypted bucket using the graphical user interface

Firstly you need to deploy the graphical user interface. Windows users need the Command Prompt tool and then run the command below. The command below should open your web browser with rclone GUI. The same process is valid for Linux users, who need to open the Terminal window and run the command listed below.

rclone rcd --rc-web-gui

The following steps are identical for Windows as well as for Linux users.

After GUI startup we will click in the left menu on the Configs (1) button and then on the Create a New Config (2) button.

Firstly, we need to type Name of this drive (1) and then we will select from the menu option Encrypt/Decrypt a remote (1). Then we will click on the Next button.

In the next step, we need to specify Remote to encrypt/decrypt (1). Here is important to define the already existing S3 profile/connection and the bucket name where we wish to create encrypted space. The input must be here in the following format s3-profile:bucket-name. If you choose non-existing bucket rclone will create it. Then we will choose the Password for encryption (2) and also recommended Password for salt (2).

Then we need to click on the Explorer (1) button. Now we are in browser mode and then via clicking at + (2) we can open a new tab with an encrypted bucket.

Then we need to click in the field Type the name of remote you want to open (1) and select the corresponding name of the encrypted bucket (1). Then we can continue by clicking on the Open (2) button.

At this moment we can start to upload the data which we wish to be encrypted. Just click on the Upload (1) icon and then you can select the data from the local disk or you can drag-and-drop your data using interactive window (2).

In the example below we have uploaded three pictures (1) into decrypted volume. We can check the upload in explorer by opening the remote S3 storage in the tab (2).

Now we can have a look into encrypted bucket (1).

Indeed we can see that our three pictures (1) have been encrypted.

Check of encrypted data integrity

To check encrypted data integrity it is necessary to use the command cryptcheck, see below. Using the common workflow for data integrity checks will cause significant difficulties in the encrypted bucket. It can result in forced downloading of all data from the remote site so it can stall your client.

  rclone cryptcheck --fast-list C:\Users\Albert\Desktop\test_sync shared_encrypted:dir01/
 
    2022/08/29 16:57:45 NOTICE: Encrypted drive 'shared_encrypted:dir01/': 0 differences found
    2022/08/29 16:57:45 NOTICE: Encrypted drive 'shared_encrypted:dir01/': 14 matching files

Sharing of encrypted buckets

The buckets can be shared within the mutual space called the tenant or between users using the bucket policy. If you wish to share the buckets equipped with the encrypted volume you need to share the credentials (for encrypted volume in your bucket) with your colleagues. A shared bucket has to have a properly set up bucket policy.

Once you configure the encryption in your bucket you just need to share the encryption passwords, you used during the encrypted bucket creation and the bucket name with your colleague. Your colleague can use the guide above to configure corresponding encrypted buckets on his/her machine using the passwords, you shared.

Compromitting of encrypting passwords vs. loss of encrypting passwords

In case of compromitting or leakage of your encrypting passwords or in the situation that you need to change the passwords is only possible to create a new encrypted volume with new encrypting passwords. All data has to be transferred to the new encrypted volume and the old one should be deleted.

Here you have two general options. The first option is to upload your data from the local machine to the encrypted volume if you have them locally. Then you can delete the old encrypted volume.

The second option is to transfer the data using rclone. You can use rclone to copy the data from the old encrypted volume to the new encrypted volume. The advantage of this method is that you don’t have to download all data locally to your machine and then upload it again, see the example below.

rclone copy old_encrypted_drive:dir01 new_encrypted_drive:dir01

In case of loss of encryption passwords you lost your data as well!

Loss of encrypting passwords always means data loss!!!