Table of Contents

Basic commands and controls for rclone tool

To be able to configure the rclone tool using this guide first, you have to download, unzip and install rclone, the guide can be found at the rclone main page and then do the Configuration of S3 connection using rclone.

Basic commands and controls for rclone tool

All available commands for rclone can be listed using the command: 
rclone help

Alternatively you can find rclone guide on the rclone websites. Below are described the selected commands to control buckets, directories and files.

Listing buckets and directories

Listing of the available profiles/connections. 

rclone listremotes
   cesnet_s3_encrypted:
   cesnet_s3cl2:
   sftp_du4:
   sftp_du5:

Listing of buckets of the selected profile/connection. 

rclone lsd cesnet_s3cl2:
   -1 2020-11-11 08:53:48        -1 111
   -1 2022-07-28 10:03:20        -1 test

Listing the content of the selected bucket. Below we can see the content of the bucket. There is one folder named “test_data” and three files in it. 

rclone ls cesnet_s3cl2:test
   106996 test_data/DSC_0001.jpg
   367329 test_data/DSC_0003.jpg
   256805 test_data/DSC_0004.jpg

Creation of the bucket, copying, deletion...

Creation of the new bucket. 

rclone mkdir cesnet_s3cl2:test-bucket

Deletion of the bucket. 

rclone rmdir cesnet_s3cl2:test-bucket

Copying the files.

Rclone cannot create empty folders, see the error below. 
rclone mkdir cesnet_s3cl2:test-bucket/empty-directory
2022/08/24 12:18:36 NOTICE: S3 bucket test-bucket path empty-directory: Warning: running mkdir on a remote which can't have empty directories does nothing

The solution is to use a full path inside the bucket including a non-existing directory during copying. In case you type non-existing directory rlone will create it, see the example below. 

rclone copy /home/user/test_file1.pdf cesnet_s3cl2:test-bucket/new-dir1/new-dir2/

Then we can check the files using ls command, where we can see that the folders have been created. Namely new-dir1 a new-dir2: 

rclone ls cesnet_s3cl2:test-bucket
  3955866 new-dir1/new-dir2/test_file1.pdf

File deletion.
To delete a particular file, we can use either command deletefile or the command delete to remove all files in the given path. 

rclone deletefile cesnet_s3cl2:test-bucket/new-dir1/new-dir2/test_file1.pdf
In case you delete the only file (object) in the directory resulting in empty directories structure the empty directories will be deleted! Directories are in object technology always represented by the name of a particular object (file), deletion of empty directories is thus expected behavior.

Directory syncing

To sync the directories you can use the option sync. Synchronization is affecting the content only on the target side, no changes are performed on the source side.

Below is the exemplary command of rclone sync. The command contains recommended options which are described below. 

rclone sync --dry-run --progress --fast-list /home/user/source-dir cesnet_s3cl2:test-bucket/
The command above always syncs the content of the source directory. It does not matter if you DO NOT use the slash at the end of the source directory. Behavior of rclone is in this perspective different than rsync behavior.

The command above contains several recommended options.
Option dry-run allows performing the dry-run sync with listing the potential changes. 

--dry-run

Option progress allows seeing the continuous progress of the sync. 

--progress

Option fast-list allows limiting the number of API requests. This option can enhance the transfer of larger datasets. It uses one request to read the information about 1000 objects and store it in the memory. 

--fast-list

Option interactive allows interactively deciding which change (on the target data storage) we want to accept or reject. 

--interactive

Data integrity checks

All commands related to data integrity check should contain fast-list option, see above. Using the fast-list option will enhance the speed of the integrity checks.

Rclone allows testing the integrity of transferred data. 

rclone check --fast-list C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync

The command checks the checksums on the source side as well as on the target side. For fast checks you can use the option size-only, where are checked only file sizes. 

rclone check --fast-list --size-only C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync
To check data integrity on the encrypted buckets please use the option cryptcheck which is described in the guides related to encrypted buckets. In the case of using the option check on the encrypted volume, there will occur the forced download of all data in the checked path. Forced downloads are unnecessary and can stall your client.

Additional information (for instance usage of external checksum file) to check the data integrity can be found on rclone websites.