Install a later version of AWS CLI v1

[paulca]

Is your feature request related to a problem? Please describe.
This is no longer a problem for me because I've since modified my pipeline to run the AWS CLI inside a Docker container, which is probably a much better way to do this anyway.

However, in case it helps anyone, there have been a number of updates to the AWS CLI, one in particular that broke our deploys because it was incompatible with an update to EKS. A later version of 1 would have worked. pip on Elastic Stack seems to install awscli 1.19, when the latest version on pypy is 1.24. I don’t know enough to know why it’s not installing the latest version.

Describe the solution you'd like
1.24 would have been totally fine.

Describe alternatives you've considered
Moving our deploy script inside a Docker container where we control the awscli version was a perfect solution for us, and in fact a much better solution overall because it meant that the awscli was project specific, rather than stack specific.

Additional context
It might be worth adding a note to the documentation recommending that folks actually run deploy scripts inside Docker containers if that's the preferred way of doing this.

[moskyb]

hey there @paulca! agreed, we totally need to be installing a later version of the AWS cli. Somewhat related, i think that when we do the next major version bump of the elastic stack, we'll probably only install the AWS CLI v2.

In re your specific issue, i thiiink the issue might be that we're using pip2 (😬) to install the AWS cli. I suspect that if we just switch to using pip3 in this case (and ideally scrub python 2 from the elastic stack entirely), we should get the latest v1 (1.25 at time of writing).

[pda]

Somewhat relevant to staying on awc-cli v1 until elastic stack v6: #670 (comment)

[admackin]

We (Culture Amp) would love to this upgrade too.

[admackin]

OK my workaround is to install a new version inside a virtualenv (avoiding extra work of using Docker)

If I source the following script I get the latest version when I run aws command

#!/bin/sh

# source this script if you need a newer version of `aws` than Buildkite provides (1.19)

python3 -m venv .aws-cli-venv
. .aws-cli-venv/bin/activate
pip3 install awscli

function aws {
  # for some reason the new installed aws-cli version ignores #! line on Buildkite so we
  # work around like so
  python3 $(which aws) $*
}

I'm still still confused about
a) why I need the aws function, but otherwise it tries to execute aws python script as shell script
b) why the function def doesn't work when I put it in pre-command hook (just gets ignored)

But that doesn't really matter, and it's probably better to opt-in only for steps which need it

[CaseyLabs]

Following up on this issue:

• Currently the Builkdite Elastic CI Amazon Linux ec2 image is running AWS CLI v1

• Upgrading the AWS CLI to v2 was held back until Elastic CI v5 was released [original comment]

Any chance we can get the AWS CLI upgraded to v2?

cc @pda @yob

[SoManyHs]

Should this be closed? I saw it was fixed in #1131.

[triarius]

Thanks for the catch @SoManyHs