Table of Contents

Linux client s3cmd

S3cmd is a free command-line tool. It allows you to upload and download your data to the S3 object storage. S3cmd is written in Python. S3cmd is an open-source project available under GNU Public License v2 (GPLv2) and it is free for personal as well as commercial usage.

We recommend you use preferably AWS CLI.
We encountered some issues while using s3cmd. For instance, bucket names cannot begin with numbers or capital letters.

Installation of s3cmd tool

S3cmd is available in the system repositories for CentOS, RHEL and Ubuntu. You can install it via following guide.

On CentOS/RHEL: 

$ sudo yum install s3cmd

On Ubuntu/Debian: 

$ sudo apt install s3cmd

s3cmd configuration

Please insert the following lines into the config file located at /home/user/.s3cfg: 

[default]
host_base = https://s3.clX.du.cesnet.cz
use_https = True
access_key = xxxxxxxxxxxxxxxxxxxxxx
secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
host_bucket = s3.clX.du.cesnet.cz

Host base and Host bucket is S3 endpoint URL, which you received via email together with Access Key and Secret Key. You should receive it via email during the S3 account creation.

Config file with GPG encryption

Config file with GPG encryption 

[default]
host_base = https://s3.clX.du.cesnet.cz
use_https = True
access_key = xxxxxxxxxxxxxxxxxxxxxx
secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
host_bucket = s3.clX.du.cesnet.cz
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


Usage of s3cmd commands

S3cmd commands support elementary operations with buckets - creation, listing, and deletion.

Buckets operation

Buckets operation

Listing all s3 buckets 

$ s3cmd ls
The bucket name should be unique and should contain only small letters, capital letters, numerals, dashes, and dots. The bucket name must begin only with a letter or numeral and it cannot contain dots next to dashes or multiple dots.

Creation of new s3 bucket 

$ s3cmd mb s3://newbucket

Removing s3 bucket 

$ s3cmd rb s3://newbucket

Only emptied bucket can be removed!

Listing s3 bucket size 

$ s3cmd du s3://newbucket/


Files and directories operations

Files and directories operations

Listing of s3 bucket 

$ s3cmd ls s3://newbucket/

Data upload into s3 bucket

File upload 

$ s3cmd put file.txt s3://newbucket/

Upload of encrypted files 

$ s3cmd put -e file.txt s3://​newbucket/​

Directory upload 

$ s3cmd put -r directory s3://newbucket/

Please make sure, that you didn't forget to remove the trailing slash (e.g. .: directory/), trailing slash denotes uploading only the content of the desired directory.

File downloading from s3 bucket 

$ s3cmd get s3://newbucket/file.txt

Data deletion from s3 bucket 

$ s3cmd del s3://newbucket/file.txt 
$ s3cmd del s3://newbucket/directory

Data sync into s3 bucket 

$ s3cmd sync /local/path/ s3://newbucket/backup/

Data sync from s3 bucket 

$ s3cmd sync s3://newbucket/backup/ ~/restore/