DocumentationAWS CLI as a command-line tool
AWS CLI is a widely used tool for managing the S3 service. It is written in Python.
AWS CLI installation
We recommend using the official AWS docummentation for installing AWS CLI. It provides detailed guides for installation on both Linux and Windows.
AWS-CLI in virtual environment
If you need to install AWS CLI in the virtual environment you can use this guide.
Configuration of AWS CLI
User profile
We recommend using the --profile option to configure AWS CLI, as it allows you to define multiple user profiles with different credentials. Alternatively, you can configure it without the --profile option; in this case, AWS will use the default settings, and all commands will remain the same, simply omitting the --profile option.
In the configuration wizard, make sure to insert the Default region name option. You can use us-east-1 option. If you don’t add the “Default region name”, the config file will lack the region parameter, leading to an InvalidLocationConstraint error when using aws s3.
Below, we will demonstrate the AWS CLI configuration. The example commands provided use the --profile option.
AWS Access Key ID [None]: xxxxxxxxxxxxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Default region name [None]: us-east-1
Default output format [None]: text
AWS Access Key ID - access key, obtained from data storage administrator
Secret Access Key - secret key, obtained from data storage administrator
Default region name - us-east-1 >> Hint: some software tools, such as Veeam, may have specific requirements. In such cases, enter the storage information.
Default output format - choose the output format (json, text, table)
Endpoint URL
For smooth operation, it is essential to use the --endpoint-url option with the specific S3 endpoint address provided by CESNET.
Multipart S3 upload - the maximum file size is limited to 5 GB. It is recommended to use aws s3 commands (such as aws s3 cp) for multipart uploads and downloads, as these commands automatically handle multipart uploading and downloading based on the file size. In contrast, aws s3api commands, such as aws s3api create-multipart-upload, should only be used when aws s3 commands do not meet specific upload requirements, such as when the multipart upload involves multiple servers, needs to be manually stopped and resumed later, or requires parameters not supported by the aws s3 command. More information can be found on the AWS website.
AWS CLI Controls - Overview (s3)
To view the available commands, use the help option. The aws s3 tool offers several advanced functions, as described below.
Operation with buckets
Unique name of the bucket
The bucket name must be unique within tenant. It should only contain lowercase letters, numbers, dashes, and dots. The bucket name must begin with a letter or number and cannot contain dots followed by a dash, dots preceded by a dash, or multiple consecutive dots. We also recommend avoiding the use of “slash” in the bucket name, as slashes will prevent the bucket from being accessed via the API.
Bucket creation
Bucket listing
Bucket deletion
Operation with files
File upload
File download
File deletion
Directory/Folder operation
The contents of the source folder are always copied when using the following command, regardless of whether there is a slash at the end of the source path. The behavior of aws differs from that of rsync. If you want the source directory to appear in the destination, you can add the source directory’s name to the destination path. The AWS tool will create the directory in the destination while copying the data, as shown in the exemple commands below. This also applies to directory downloads or synchronization using aws s3 sync.
Upload the directory
Download the directory
Directory deletion
Directory sync -> upload to cloud
Directory sync -> download from cloud
AWS CLI Controls - api-level (s3api)
The aws tool supports the use of the aws s3api module, which provides advanced functions for managing the S3 service, as detailed below. The configuration of credentials and connections is the same as for the aws tool, as outlined at the beginning of this guide.
the list of available commands can be viewed by using the help option with the following command. Alternatively, the complete list is available on the AWS website.
Exemple Configuration File for AWS-CLI
After a successful configuration, the configuration file will be created. An example is shown below. The credentials file can be found in the same path.
Config file
Windows: C:/Users/User/.aws/config
Linux: /home/user/.aws/config
[profile test-user]
region = us-east-1
output = text
Special Functions of AWS-CLI
AWS-CLI offers several advanced functions for data sharing and versioning.
Presign URLs
For objects in the S3 service, you can generate a presign URL to allow your colleagues to download the data. More information can be found in the section dedicated to advanced S3 features
Bucket policies
To share your data, you can setup so called bucket policies. You can share a specific bucket with a particular group (tenant) or make your bucket publicly readable. More information is available in the section dedicated to advanced S3 features
Bucket versioning
You can enable object versioning within your buckets, allowing you to restore any previous version of an object (file). More information can be found in the section dedicated to advanced S3 features
Last updated on
