DocumentationAdvanced S3 Features
The following sections provide a basic overview of advanced S3 features that can enhance the efficiency of your data workflow.
Sharing an S3 Object Using (Presigned) URL
To generate URL links for objects stored in S3 storage, you must first set up the aws tool.
By default, all objects and buckets are private. A pre-signed URL is a reference to a Ceph S3 object that allows anyone with the URL to retrieve the object via an HTTP GET request.
The following presigning command generates a pre-signed URL for a specified bucket and key, which remains valid for one hour:
To create a pre-signed URL with a custom expiration time that links to an object in an S3 bucket, use the following command:
This will create a URL that remains accessible for one month. The --expires-in parameter is specified in seconds.
Once the pre-signed URL has expired, you will see a message like the following:
Changing the URL lifetime
Once a pre-signed URL is generated, its lifetime cannot be changed. You will need to generate a new pre-signed URL, whether the original URL has expired or not.
S3 Object Versioning
Object Versioning is used to store multiple copies of an object within the same bucket, with each copy representing the object’s content at a specific point in time. This feature can help protect objects in a bucket from being overwritten or accidentally deleted.
To enable this functionality, it must be activated at the bucket level, resulting in three possible bucket states: ‘unversioned’, ‘versioning enabled’ or ‘versioning suspended’.
When a bucket is created, it starts in the ‘unversioned state’.
Once versioning is enabled, the bucket can switch between ‘versioning enabled’ and ‘versioning suspended’ states, but it can not revert to the ‘unversioned state’. In other words, versioning cannot be disabled; it can only be suspended.
Each version of an object is identified by a VersionID. If versioning is not enabled, the VersionID will be null. In a versioned bucket, updating an object with a PUT request will store a new version of the object with a unique VersionID.
You can access a specific version of an object in a bucket using either its name or the combination of the name and VersionID. If you access the object by name only, the most recent version will be retrieved.
If an object is deleted from a versioned bucket, any subsequent GET requests will return an error unless a VersionID is specified. To restore a deleted object, there is no need to download and re-upload it; you can simply issue a COPY operation using the specific VersionID. This process will be demonstrated in this guide.
To test object versioning, we will use the AWS CLI, an open-source tool that provides commands for interacting with AWS services via the terminal. Specifically, we will use the s3api command set, which contains API-level commands for working with S3.
Bucket Versioning
In a non-versioned bucket, uploading an object with the same key will overwrite the existing object. In a versioned bucket, uploading an object with the same key will make the newly uploaded object the current version, while the previous object will become a non-current version.
For proper functionality, it is essential to use the —endpoint-url option with all commands when specifying the relevant S3 service addresses operated by the CESNET association.
Bucket name restrictions
The bucket name must be unique within the tenant and can only contain uppercase and lowercase letters, numbers, dashes, and periods. The name must start with a letter or number and cannot have periods next to dashes or multiple consecutive periods. Additionally, we recommend avoiding the use of ”/” and ”_” in the name, as this may prevent it from being used via the API.
First, we need to create the bucket where we will set up versioning.
Next, we can check if versioning is enabled.
Now we enable versioning.
If we check the versioning status again, we will see that it is enabled.
Adding an Object
Now, we will put a new object to the created bucket.
We can change the file by updating the body.
Now, we can list the versions of the object.
Retrieve an Object
For a non-versioned bucket, object lookup always returns the most recent available object. In a versioned bucket, the search returns the current version of the object:
Now, we can retrieve desired object.
In a versioned bucket, inactive objects can be retrieved by specifying the Version ID:
Now we can list the particular version.
Object Removal
In a non-versioned bucket, the object is permanently deleted and cannot be recovered. In a versioned bucket, all versions remain in the bucket, and RGW adds a delete flag, which becomes the current version:
Now we can check the object versions again.
In the case of a versioned bucket, if an object with a specific VersionID is deleted, it is permanently removed:
Now we can check the object versions again.
Object Restoration
To restore an object, the recommended approach is to copy the previous version of the object to the same bucket. The copied object will become the current version, while all other versions of the object are preserved:
Now we can restore the particular version of the object.
And check the object versions.
Set Up Bucket Policies for Sharing (AWS-CLI S3 plugin)
Coming soon…
Last updated on
