Fix AWS ECR private repository usage

[ErikEvenson]

This is my attempt to allow nomad to access AWS ECR private docker repositories. The problem is per the commit discussion between me and @diptanu here: c4fd236

This solution does require the user to add https:// to the image name in the docker config. I am not sure if that is in line with nomad's design vision. Other declarative approaches are possible, but this does work for my AWS ECR usage.

[ErikEvenson]

Here is a job that uses this solution successfully:

job "vbez-web" {
    region = "global"
    datacenters = ["dc0"]
    type = "service"
    priority = 50

    constraint {
        attribute = "${attr.kernel.name}"
        value = "linux"
    }

    update {
        stagger = "10s"
        max_parallel = 1
    }

    group "vbez-web" {
        count = 1

        restart {
            interval = "5m"
            attempts = 10
            delay = "25s"
            mode = "delay"
        }

        task "api" {
            driver = "docker"

            config {
                dns_servers = ["172.17.0.1", "75.75.76.76", "75.75.75.75"]
                image = "https://212181198242.dkr.ecr.us-east-1.amazonaws.com/vbenergy/vbez-web:2016.02.19.0"
                port_map {
                    web = 8080
                }
            }

            logs {
              max_files = 10
              max_file_size = 10
            }

            service {
                name = "vbez-web"
                tags = ["global"]
                port = "web"

                check {
                    name = "alive"
                    type = "tcp"
                    interval = "10s"
                    timeout = "2s"
                }
            }

            resources {
                cpu = 500
                disk = 101
                memory = 256
                network {
                    mbits = 10
                    port "web" {
                    }
                }
            }
        }
    }
}

[ErikEvenson]

I am not sure I understand the CI failure here. The client driver tests pass on my local machine.

[dadgar]

@ErikEvenson That test has become flakey recently, don't worry about that one.

[ErikEvenson]

Great -- I'll take that as good news.

[diptanu]

I think we don't need to set this as false as default value for bools are false

[ErikEvenson]

Agreed -- will change.

[diptanu]

@ErikEvenson Thanks for taking a stab at this! I left a few comments, I hope after a few iterations we can get this merged and get it out for 0.3.1 which is going to come out in a few days.

[ErikEvenson]

I incorporated/answered your comments @diptanu and brought the PR up to date with master. Cheers.

[diptanu]

@ErikEvenson I just had one last comment about introducing an Init method to do the transformation of the image name before Validate is being called. Once this is done, I think we can merge this PR.

[ErikEvenson]

@diptanu great -- I'll try to do this later today.

[ErikEvenson]

I have brought this PR up to date with master and made the changes @diptanu requested regarding the Init() function. Note: master is not passing the make test test suite on my local box.

[steve-jansen]

@ErikEvenson out of curiosity, how does your Nomad cluster authenticate with ECR?

My understanding is ECR requires a call to IAM for GetAuthorizationToken, then invoke a docker login. However, this token is only valid for 12 hours.

Are you handling this token out of band on your cluster?

[ErikEvenson]

@steve-jansen typically, before I run a nomad job, I have the cluster authenticate itself via a consul command. So something like this:

consul exec --http-addr=<IP_ADDR>:8500 "su docker -c '$(aws ecr get-login)'"

[diptanu]

@ErikEvenson Oh I see. Can we improve this? I would like this to be a no-op and Nomad handling this for the user.

[ErikEvenson]

@diptanu I'd love to see this being handled by nomad end-to-end, but I'm not sure how to have nomad do this for the general user.

Everyone is going to have a different opinion on how to handle AWS authentication. I use vault to populate the AWS credentials that the aws ecr get-login command uses -- but that assumes you are using vault. Executing the consul exec command also assumes that consul is in play as well.

This is a general problem for secured, private repositories as well. Non-AWS repos will still have to do docker login which is what $(aws ecr get-login) effectively does. Perhaps this is an integration point to vault per #597?

Thoughts?

[diptanu]

@pshima Any thoughts here?

[ErikEvenson]

If you are running nomad on an EC2 host, you could give the EC2 instance a role that gives the instance full access to your ECR repos. Then, you wouldn't have to store or even log in to AWS ECR.

[steve-jansen]

@ErikEvenson

Then, you wouldn't have to store or even log in to AWS ECR.

Very interesting. Have you been able to bypass the need for aws ecr get-login (or equivalent API call) with an IAM EC2 instance profile?

[ErikEvenson]

@steve-jansen I haven't actually used this technique with nomad as it is a sort of lock-in to the AWS platform. I would like my nomad scripts to be just as useful on other cloud provider platforms. My expectation is that you would not need to do aws ecr get-login prior to pulling from the repo though.

[diptanu]

@ErikEvenson This is something we might be able to pull off by having the concept of image providers for different type of environments - so we could have a image provider which works well with AWS ECR and another with GCP for example.

[ErikEvenson]

@diptanu Right, but you are always going to want to have the ability to, say, pull a docker image from AWS for use on a, say, Azure instance.

[jaygorrell]

I didn't test this but I don't believe it's accurate to say you don't need to use aws ecr get-login if you have an instance profile. That would only mean that you don't need a credentials file for that command to work.

The point of the command is still to run a docker login command.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.