pkg/rhcos: Default to the most-recent AMI

[wking]

We're pushing public AMIs since openshift/os#304. There's still no public analog to this, so I'm just scraping this from metadata on images available via the AWS API. The analogous AWS command line invocation is:

$ AWS_DEFAULT_REGION=us-east-1 aws ec2 describe-images --filter 'Name=name,Values=rhcos*' --query 'sort_by(Images, &CreationDate)[-1].ImageId' --output text

with a few extra filters thrown in. The full set of metadata on the most recent current image is:

$ AWS_DEFAULT_REGION=us-east-1 aws ec2 describe-images --filter 'Name=name,Values=rhcos*' --query 'sort_by(Images, &CreationDate)[-1]' --output json
{
    "VirtualizationType": "hvm",
    "Description": "Red Hat CoreOS 4.0.5846 (c9a6bb48b837b5bcfeb9bd427be9a18b5bd75b6c57cb289245f211ff98b2a740)",
    "Hypervisor": "xen",
    "EnaSupport": true,
    "SriovNetSupport": "simple",
    "ImageId": "ami-08a5792a684330602",
    "State": "available",
    "BlockDeviceMappings": [
        {
            "DeviceName": "/dev/xvda",
            "Ebs": {
                "Encrypted": false,
                "DeleteOnTermination": true,
                "VolumeType": "gp2",
                "VolumeSize": 8,
                "SnapshotId": "snap-00a45db4ad6173805"
            }
        },
        {
            "DeviceName": "/dev/xvdb",
            "VirtualName": "ephemeral0"
        }
    ],
    "Architecture": "x86_64",
    "ImageLocation": "531415883065/rhcos_dev_c9a6bb4-hvm",
    "RootDeviceType": "ebs",
    "OwnerId": "531415883065",
    "RootDeviceName": "/dev/xvda",
    "CreationDate": "2018-09-19T23:40:54.000Z",
    "Public": true,
    "ImageType": "machine",
    "Name": "rhcos_dev_c9a6bb4-hvm"
}

That doesn't include the "tested" information, so there's still no support for changing channels. We'll need to wait for a public analog of this, which is blocked on getting stable, production hosting for the release metadata.

I'd prefer to use JMESPath and server-side filtering in Go as well, to only return the latest matching AMI. But the AWS Go library doesn't seem to support server-side filtering at the moment (aws/aws-sdk-go#2156). Docs for the AWS Go APIs I'm using are here, here, here, here, here, and here.

The filters I'm adding here are similar to those we used for Container Linux before they were dropped in #233. I added a few more just to be conservative (e.g. we don't want to match a pending or failed image, so I require state to be available).

I haven't pushed the Context variables all the way up the stack yet, so there are some context.TODO() entries. The 30-second timeout keeps us from hanging excessively when the caller lacks AWS credentials; the error messages look like:

failed to init cluster: failed to parse test config: failed to determine default AMI: NoCredentialProviders: no valid providers in chain. Deprecated.
  For verbose messaging see aws.Config.CredentialsChainVerboseErrors

You can test this error condition by removing the explicit AMI values I've added to our fixtures in this commit and running:

$ AWS_PROFILE=does-not-exist go test ./installer/pkg/...

CC @ashcrow, @jupierce.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wking]

I've spun off some of the vendoring changes we need here into #291 (getting master back into a Glide-able state). There are some additional vendoring commits in this PR now as well, as take baby steps towards something that included a working glide get --strip-vendor github.com/aws/aws-sdk-go. Details in the commit messages.

[wking]

Rebased around #287 with 801c21f -> 2343639.

[ashcrow]

@yuqi-zhang is looking at making the json public.

[wking]

I'd prefer to use JMESPath and server-side filtering in Go as well...

Also, while I'd thought JMESPath was server-side on the CLI, it turns out that everybody is using JMESPath client-side.

@yuqi-zhang is looking at making the json public.

Will we be able to drop the AWS query then? Or do you think we'll still need it for regions not supported (yet?) by JSON files? Or to support more-current private releases?

[ashcrow]

@wking You should be able to read the file that @yuqi-zhang will be writing to an s3 bucket. It will denote the latest AMI. Unreleased AMI's won't be making it out to all the regions, but real releases will.