In the recent times, I am exploring and playing with Top Tools for Object Storage and Data Management. In this journey, one of the tools that I was introduced to was s3cmd.

What is s3cmd?

S3cmd is a powerful open-source command-line tool designed for managing Amazon S3 and S3-compatible object storage services like IBM Cloud Object Storage. It provides a user-friendly interface for performing various operations on S3 buckets and objects, including creating and removing buckets, uploading, downloading, and deleting files.

Key features of s3cmd include:

1. Bucket management: Create, list, and delete buckets.
2. Object operations: Upload, download, and delete files within buckets.
3. Synchronization: Sync local directories with S3 buckets.
4. Access control: Modify bucket policies and access control lists.
5. Website hosting: Create and manage static websites hosted on S3.
6. Multipart uploads: Handle large file uploads efficiently.
7. Encryption: Support for server-side encryption.

S3cmd is compatible with Python 2.6 or newer, including Python 3.x, and runs on Linux and macOS systems. Its versatility and ease of use make it an excellent choice for both beginners and advanced users who need to interact with S3-compatible storage services.

This tool simplifies the process of managing cloud storage, allowing users to automate tasks, integrate S3 operations into scripts, and efficiently handle large amounts of data. Whether you’re a developer, system administrator, or data analyst, s3cmd can streamline your workflow when working with S3 and similar object storage platforms.

s3cmd installation and configuration

1. Install S3cmd following theinstructions here

s3cmd command on warp

1. To configure, you need to have an S3 cloud instance or any S3 compatible instance with HMAC credentials — Accesskey and Secrets key and endpoint that starts with or similar to s3://…

To understand what are HMAC credentials, refer to my article here

1. Once you have the HMAC credentials and endpoint, you can run the following command for an interactive configuration.

s3cmd --configure

Key commands

Here are some commonly used s3cmd commands:

• Create a bucket:

  s3cmd mb s3://bucket-name

• List all buckets:

  s3cmd ls

• Upload a file to a bucket:

  s3cmd put file.txt s3://bucket-name/

• Download a file from a bucket:

  s3cmd get s3://bucket-name/file.txt

• Delete a file from a bucket:

   s3cmd rm s3://bucket-name/file.txt

• List objects in a specific bucket:

   s3cmd ls s3://bucket-name/

• Synchronize a local directory with a bucket:

   s3cmd sync local_directory/ s3://bucket-name/

• Set public access for a file:

   s3cmd setacl s3://bucket-name/file.txt --acl-public

• Delete a bucket:

   s3cmd rb s3://bucket-name

These commands allow you to perform basic operations like creating and deleting buckets, uploading and downloading files, and managing access permissions. S3cmd offers many more advanced features for managing your S3-compatible storage, making it a powerful tool for both simple and complex storage management tasks.

403 error resolution

The most common error is 403 — Access denied. Which you may think is because of the AccessKeyID or the Secrets key you provided. But actually this is a generic error.

ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.

One of the many reasons for this error is this configuration parameter that you either forgot to set or wrongly configured

Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars can be used

if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

The right way to configure this parameter is

%(bucket)s.<END_POINT>

For example:
%(bucket)s.s3.eu-de.cloud-object-storage.appdomain.cloud

Once the right of credentials, endpoint & the DNS-style is setup, you won’t see the 403 (InvalidAccesskeyId) error.

If this article is helpful. please clap 👏 and ♻️ share with your network.