In case you have a fast connection of about 1-2Gbps and you want to utilize it for data transfers, you can use the s5cmd tool. It allows you to fully optimize the data transfer. The tool is available in form of compiled binaries for Windows, Linux and macOS. It is also available as a source code or docker image. Detailed information can be found on the project Github page.
Please insert into .aws/credentials the folowing options.
[default] aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxx aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx max_concurrent_requests = 200 max_queue_size = 20000 multipart_threshold = 128MB multipart_chunksize = 32MB
Access Key a Secret Key has been provided by admins while creating the S3 account.
To list all buckets we can use:
s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz ls
Simple file upload:
s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz cp myfile s3://bucket
To achieve higher speed for data transfers it is necessary to modify the following parameters, particularly utilize or CPU cores and workers, eg.:
s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz cp -c=8 -p=5000 /adresar/velky_soubor s3://bucket
CESNET, z. s. p. o.
Generála Píky 26
16000 Prague 6
Tel: +420 234 680 222
Fax: +420 224 320 269
info@cesnet.cz
Tel: +420 234 680 222
GSM: +420 602 252 531
Fax: +420 224 313 211
support@cesnet.cz