Use conventional AWS environment variables for S3 credentials

[virtual-machinist]

Hi!

Is it possible to make use of the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) for S3 config instead of the credentials file option?

When running in Docker I can work around this with a custom entrypoint script that creates the credentials file before starting the server, but wouldn't it be better to simply delegate to something like

new AWSCredentialsProviderChain(
              new EnvironmentVariableCredentialsProvider(),
              new AWSStaticCredentialsProvider(new AnonymousAWSCredentials())
)

instead of static anonymous credentials.

If we add a new ProfileCredentialsProvider() before the anonymous credentials provider then there's even no need to provide a special S3 credentials file command line parameter as it may be fetched by the AWS library using conventional options AFAICT.

Or am I missing something and there's another way to use the environment variables already?

[afs]

Hi @virtual-machinist - have you tried no providing the keys file? From reading the code, (I don't have a setup to hand just at the moment) it defaults to standard AWS S3 setup, but maybe some earlier checking happens to require the keys file.

See S3.buildS3 . The AWS setup starts with AmazonS3ClientBuilder.standard() which has new S3CredentialsProviderChain()

[virtual-machinist]

Locally MinIO, but the project I'm trying to use RDF Delta in uses some proprietary solution, I've no way to tell what exactly.
The thing I haven't tried yet is setting up environment variables for everything related to S3 besides the required bucket name and region. That may help since then the endpoint can be null.

[virtual-machinist]

Nope, neither AWS_ENDPOINT_URL nor AWS_ENDPOINT_URL_S3 work, the client tries accessing AWS. Apparently setting service-specific endpoints is only supported for AWS CLI that I'm using elsewhere, not the Java client used here.

[virtual-machinist]

@afs what do you think, can this be implemented in any way? Can do a PR with my suggested change if that's an acceptable solution.

[afs]

@virtual-machinist yes - a PR to change that code would be great to see.

Would defaulting to "normal" (whatever AWS standard setup is) and let a file override that work? (I'm not a MinIO user.)

[virtual-machinist]

@afs I added the environment variable credentials provider with the fallback to anonymous credentials to minimize possible breakage. See #272 .