S5cmd for Ultra-Fast Transfers

If you have a fast connection of around 1-2 Gbps and want to maximize its potential for data transfers, you can use the s5cmd tool. This tool optimizes data transfers for maximum efficiency. It is available as compiled binaries for Windows, Linux, and macOS, as well as in source code and Docker image formats. For more detailed information, visit the project Github page.

Please add the following options to your .aws/credentials.

[default]
    aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxx
    aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    max_concurrent_requests = 200
    max_queue_size = 20000
    multipart_threshold = 128MB
    multipart_chunksize = 32MB

aws_access_key_id and aws_secret_access_key has been provided by admins while creating the S3 account.

Listing all buckets

s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz ls

Simple file upload

s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz cp myfile s3://bucket

How to achieve high transfer speed?
To achieve faster data transfers, it’s essential to adjust the following parameters, particularly by utilizing CPU cores and workers. See the details below.

s5cmd  --endpoint-url=https://s3.clX.du.cesnet.cz cp -c=8 -p=5000 /directory/big-file s3://bucket

S5cmd for Ultra-Fast Transfers

On this page