s3cmd command line tool

S3cmd is a free command line tool. It allows you to upload and download your data to the S3 object storage. S3cmd is written in Python. S3cmd is an open-source project available under GNU Public License v2 (GPLv2) and it is free for personal as well as commercial usage.

⚠️

We recommend you use preferably AWS CLI. We encountered some issues while using s3cmd. For instance, bucket names cannot begin with numbers or capital letters.

Installation of s3cmd tool

S3cmd is available in the system repositories for CentOS, RHEL and Ubuntu. You can install it via following guide.

On CentOS/RHEL

sudo yum install s3cmd

On Ubuntu/Debian

sudo apt install s3cmd

Configuration of s3cmd tool

Please insert the following lines into the config file located at /home/user/.s3cfg.

[default]
    host_base = https://s3.clX.du.cesnet.cz
    use_https = True
    access_key = xxxxxxxxxxxxxxxxxxxxxx
    secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    host_bucket = s3.clX.du.cesnet.cz

host_base and host_bucket is S3 endpoint URL, which you received via email together with access_key and secret_key. You should receive it via email during the S3 account creation.

Config file with GPG encryption

[default]
host_base = https://s3.clX.du.cesnet.cz
    use_https = True
    access_key = xxxxxxxxxxxxxxxxxxxxxx
    secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    host_bucket = s3.clX.du.cesnet.cz
    gpg_command = /usr/bin/gpg
    gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
    gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
    gpg_passphrase = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Basic s3cmd commands

S3cmd commands support elementary operations with buckets - creation, listing, and deletion.

Bucket operations

Bucket name
The bucket name should be unique within tenant and should contain only small letters, capital letters, numerals, dashes, and dots. The bucket name must begin only with a letter or numeral and it cannot contain dots next to dashes or multiple dots.

Listing all s3 buckets

s3cmd ls

Creation of new s3 bucket

s3cmd mb s3://newbucket

Removing s3 bucket

s3cmd rb s3://newbucket

Only emptied bucket can be removed!

Listing s3 bucket size

s3cmd s3://newbucket/ du

Files and directories operation

Listing of s3 bucket

s3cmd ls s3://newbucket/

File upload

s3cmd put file.txt s3://newbucket/

Upload of encrypted files

s3cmd put -e file.txt s3://newbucket/

Directory upload

s3cmd put -r directory s3://newbucket/

Please make sure, that you didn’t forget to remove the trailing slash (e.g. .: directory/), trailing slash denotes uploading only the content of the desired directory.

Download file from s3 bucket

s3cmd get s3://newbucket/file.txt

Data deletion from s3 bucket

s3cmd del s3://newbucket/file.txt
 
s3cmd del s3://newbucket/directory

Data sync into s3 bucket from local machine

s3cmd sync /local/path/ s3://newbucket/backup/

Data sync from s3 bucket to local machine

3cmd sync s3://newbucket/backup/ ~/restore/