Skip to content

S3 Storage Configuration

James Fantin-Hardesty edited this page Oct 13, 2023 · 1 revision

To manage configuration settings for Cloudfuse for S3 storage you have four options (in order of precedence):

  1. Configuration file
  2. Environment variables
  3. AWS credential files (~./aws/credentials)

Using a configuration file is the preferred method, but the other methods might be useful in some circumstances. Only authentication can be setup with configuration file, environment variables, or AWS credential files.

Configuration file

Creating a configuration file is the preferred method to establish settings for Cloudfuse. When you've specified the settings you want in the configuration file, reference the configuration file when you use blobfuse2 mount or other commands.

Here's an example:

cloudfuse mount ./mount --config-file=./config.yaml

The Cloudfuse base configuration file contains a list of all settings and a brief explanation of each setting.

Use the sample file cache configuration file or the sample streaming configuration file to get started quickly by using some basic settings for each of those scenarios.

Here is the given s3storage section that you would need to fill in for your authentication method. If using keys, then you can add your access and secret keys. If using the AWS credential file, you can specify the profile in your configuration file to use.

s3storage:
  # Required
  bucket-name: <name of the bucket to be mounted>
  # Optional
  key-id: <S3 access key ID. Default - use credentials from environment variables or shared profile>
  secret-key: <S3 secret access key. Default - use credentials from environment variables or shared profile>
  region: <S3 region. Default - us-east-1>
  profile: <AWS shared configuration profile name. Credentials above take precedence over shared profile. Default - "default">
  endpoint: <S3 endpoint URL. Default - https://s3.[region].lyvecloud.seagate.com>

Environment variables

Setting environment variables is another way to configure some Cloudfuse settings. The supported environment variables are useful for specifying the Azure Blob Storage container to access and the authorization method to use.

The following environment variables for S3 are supported:

  • General options
    • AWS_ACCESS_KEY_ID: key ID, used as a pair with AWS_SECRET_ACCESS_KEY
    • AWS_SECRET_ACCESS_KEY: secret key, used as a pair with AWS_ACCESS_KEY_ID
    • AWS_SESSION_TOKEN: validates a temporary key pair (key ID & secret key)
    • AWS_WEB_IDENTITY_TOKEN_FILE: temporary credential from an external identity provider
    • AWS_REGION: the service region (e.g. us-east-1)
    • AWS_PROFILE: the profile name to use from shared configuration file(s)

AWS credential file

You can also use an existing AWS credential file with Cloudfuse. See guide from AWS for information for creating and setting up a credential file. Note that the credential and config files must be located at ~/.aws/credentials and ~/.aws/config.

If you a profile other than the default profile, you can specify the profile to use in the configuration file by setting profile: in the s3storage section or by setting the AWS_PROFILE environment variable.