Honor explicit profile in client when credentials from AWS_ environment variables are present

[alextwoods]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

1. To make sure we include your contribution in the release notes, please make sure to add description entry for your changes in the "unreleased changes" section of the CHANGELOG.md file (at corresponding gem). For the description entry, please make sure it lives in one line and starts with Feature or Issue in the correct format.

2. For generated code changes, please checkout below instructions first:
   https://github.com/aws/aws-sdk-ruby/blob/master/CONTRIBUTING.md

Thank you for your contribution!

Fixes #2165

[mullermp]

Hooray

[ferricoxide]

Sorry to resurrect this, but…

Trying to get a GitLab installation to not be single-threaded when working with S3 (opened an issue with GitLab, but have gotten no response on it, yet). Haven't found anything in their documentation to suggest that they're explicitly using this feature. Looking at this code (which I'm hoping their S3 storage-driver uses), I'm only seeing references to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables and that it will also scan ~/.aws/credentials) in the README. Not seeing indications that it would make use of permissions from the EC2's attached instance role, nor am I seeing a method for telling it to use a custom config-file (i.e.., AWS_CONFIG_FILE)

Overall, my goal here is to see if I can force GitLab to reference my performance-optimizations from AWS_CONFIG_FILE so that, instead of being a single-threaded process, when GitLab reads from or writes to S3 it does so in both a mult-threaded manner and that it takes advantage of the the common runtime client (absent using those, transfers are happening at about 130MiB/s vs the 1000MiB/s that one achieves when using the AWS CLI).

[alextwoods]

What performance optimizations do you have in your config file? Unfortunately the Ruby SDK does not support the common runtime S3 client. We do support multi-threaded uploads and downloads, but only through the Resource interface such as Object#download_file and calling client.get_object or client.put_object won't give you multithreaded upload/downloads.

[mullermp]

AWS_CONFIG_FILE is supported, and so are credentials from your ec2 assumed role. Those should be mentioned in the readme and if not we can touch that up.

[ferricoxide]

@alextwoods sed:

What performance optimizations do you have in your config file? Unfortunately the Ruby SDK does not support the common runtime S3 client. We do support multi-threaded uploads and downloads, but only through the Resource interface such as Object#download_file and calling client.get_object or client.put_object won't give you multithreaded upload/downloads.

Bummer. What I've found, when attempting to transfer (with the AWS CLI) a multi-hundreds gigabyte TARfile created as an export (or as a full backup) is that, unless one uses the CRT, the transfer is very slow, even if one uses the non-CRT tuning-options. For a reference, when testing with a 500GiB TAR-file, with CRT, the transfer runs at 1GiB/s; without CRT – but with suitable values for number of threads and chunk-sizes – I can get it to run at 200-300MiB/s. Ultimately, it was AWS Support's recommendation to switch to CRT. Dunno if that's a limitation of the AWS CLI that I assumed carried over to the Ruby-based implementation (used by GitLab's S3 interfaces). Granted, my assumption that GitLab's FOG-based implementation is using this project's capabilities could be in error.

Suppose I should go hit up the FOG project to validate my assumptions. I think I ended up in this thread due to a Google search that was thin on (non "sponsored" hits).

[ferricoxide]

@mullermp sed:

AWS_CONFIG_FILE is supported, and so are credentials from your ec2 assumed role. Those should be mentioned in the readme and if not we can touch that up.

I assumed that EC2 instance-roles were supported by the Ruby SDKs (since the GitLab documentation had provided requisite guidance for setting up it's Ruby-based S3 driver to use it). However, when I'd looked for AWS_CONFIG_FILE in either the SDK's documentation or a quick search (using GitHub's project-level search capability against this project), I didn't see it in the code. Thus, asked here.

Thank you for confirming. 😄