Instructions for Setting up ECS Agent Container (i.e. Other Linux AMIs when awsvpc networking mode is enabled) Do Not Work

[Andrew5194]

Summary

I am unable to set up the ECS Agent container using the instruction set forth in https://github.com/aws/amazon-ecs-agent#on-other-linux-amis-when-awsvpc-networking-mode-is-enabled.

Description

A bit of context, I want to create an Ubuntu AMI with the AWS ECS Agent installed so that I can use https://github.com/nestybox/sysbox as the default container runtime. Related open issue is here: aws/containers-roadmap#673. For this reason, I am looking to use the following instructions to manually set up the ECS agent using a docker run command so that I can also use the --runtime flag to specify the ECS Agent to use runc while I set the default container runtime for the other containers to be sysbox-runc:

mkdir -p /var/log/ecs /etc/ecs /var/lib/ecs/data
touch /etc/ecs/ecs.config
echo ECS_CLUSTER=<cluster-id> >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
/usr/bin/docker run --name ecs-agent \
--init \
--restart=on-failure:10 \
--volume=/var/run:/var/run \
--volume=/var/log/ecs/:/log:Z \
--volume=/var/lib/ecs/data:/data:Z \
--volume=/etc/ecs:/etc/ecs \
--volume=/sbin:/host/sbin \
--volume=/lib:/lib \
--volume=/lib64:/lib64 \
--volume=/usr/lib:/usr/lib \
--volume=/usr/lib64:/usr/lib64 \
--volume=/proc:/host/proc \
--volume=/sys/fs/cgroup:/sys/fs/cgroup \
--net=host \
--env-file=/etc/ecs/ecs.config \
--cap-add=sys_admin \
--cap-add=net_admin \
--env ECS_ENABLE_TASK_ENI=true \
--env ECS_UPDATES_ENABLED=true \
--env ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1h \
--env ECS_DATADIR=/data \
--env ECS_ENABLE_TASK_IAM_ROLE=true \
--env ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true \
--env ECS_LOGFILE=/log/ecs-agent.log \
--env ECS_AVAILABLE_LOGGING_DRIVERS='["json-file","awslogs","syslog","none"]' \
--env ECS_LOGLEVEL=info \
--detach \
amazon/amazon-ecs-agent:latest

Using the above setup instructions, I can get the agent to connect to the ECS cluster; however, I cannot get my tasks to run and get the following error:

The closest matching container-instance <container-arn> is missing an attribute required by your task. For more information, see the [Troubleshooting section](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/troubleshooting.html).

I've already cross-referenced the required task definition attributes vs. the container instance attributes and they are matching:

Required Task Definition Attributes

"requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],

Container Instance Attributes

"attributes": [
                {
                    "name": "ecs.capability.secrets.asm.environment-variables"
                },
                {
                    "name": "ecs.capability.branch-cni-plugin-version",
                    "value": "199bfc6-1.2-5-g199bfc6"
                },
                {
                    "name": "ecs.ami-id",
                    "value": "ami-08d4ac5b634553e16"
                },
                {
                    "name": "ecs.capability.secrets.asm.bootstrap.log-driver"
                },
                {
                    "name": "ecs.capability.task-eia.optimized-cpu"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.none"
                },
                {
                    "name": "ecs.capability.ecr-endpoint"
                },
                {
                    "name": "ecs.capability.docker-plugin.local"
                },
                {
                    "name": "ecs.capability.task-cpu-mem-limit"
                },
                {
                    "name": "ecs.capability.secrets.ssm.bootstrap.log-driver"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.30"
                },
                {
                    "name": "ecs.capability.full-sync"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.31"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.32"
                },
                {
                    "name": "ecs.capability.firelens.options.config.file"
                },
                {
                    "name": "ecs.capability.logging-driver.awsfirelens.log-driver-buffer-limit"
                },
                {
                    "name": "ecs.availability-zone",
                    "value": "us-east-1d"
                },
                {
                    "name": "ecs.capability.aws-appmesh"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
                },
                {
                    "name": "ecs.capability.task-eni-trunking"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.26"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.27"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
                },
                {
                    "name": "com.amazonaws.ecs.capability.privileged-container"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
                },
                {
                    "name": "ecs.cpu-architecture",
                    "value": "x86_64"
                },
                {
                    "name": "com.amazonaws.ecs.capability.ecr-auth"
                },
                {
                    "name": "ecs.capability.firelens.fluentbit"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
                },
                {
                    "name": "ecs.os-type",
                    "value": "linux"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
                },
                {
                    "name": "ecs.capability.task-eia"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.23"
                },
                {
                    "name": "ecs.capability.private-registry-authentication.secretsmanager"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.syslog"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.awsfirelens"
                },
                {
                    "name": "ecs.capability.firelens.options.config.s3"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.json-file"
                },
                {
                    "name": "ecs.capability.execution-role-awslogs"
                },
                {
                    "name": "ecs.vpc-id",
                    "value": "vpc-093e498597bb32b94"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
                },
                {
                    "name": "ecs.capability.task-eni"
                },
                {
                    "name": "ecs.capability.firelens.fluentd"
                },
                {
                    "name": "ecs.capability.efs"
                },
                {
                    "name": "ecs.capability.execution-role-ecr-pull"
                },
                {
                    "name": "ecs.capability.task-eni.ipv6"
                },
                {
                    "name": "ecs.capability.container-health-check"
                },
                {
                    "name": "ecs.os-family",
                    "value": "LINUX"
                },
                {
                    "name": "ecs.subnet-id",
                    "value": "subnet-07e96d0adeeb4ce8d"
                },
                {
                    "name": "ecs.instance-type",
                    "value": "t2.micro"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role-network-host"
                },
                {
                    "name": "ecs.capability.increased-task-cpu-limit"
                },
                {
                    "name": "ecs.capability.container-ordering"
                },
                {
                    "name": "ecs.capability.cni-plugin-version",
                    "value": "db586472-2020.09.0"
                },
                {
                    "name": "ecs.capability.env-files.s3"
                },
                {
                    "name": "ecs.capability.secrets.ssm.environment-variables"
                },
                {
                    "name": "ecs.capability.pid-ipc-namespace-sharing"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role"
                }
            ],

Even running the following commands shows that there are no missing attributes:

(base) andrew@Andrews-MacBook-Pro-2 Desktop % ecs-cli check-attributes --cluster <cluster-id> --task-def <task-id> --container-instances <container-id>
Container Instance            Missing Attributes
<cluster-name>                 None

Looking into the docs further (https://aws.amazon.com/premiumsupport/knowledge-center/ecs-container-instance-requirement-error/), it seems that it could be that my ecs cluster and container instance may not be on the same subnet (as I'm using the awsvpc networking mode for my ECS cluster), but after confirming that they are on the same subnet, I still get the same issue.

What's strange is when I use the ECS-optimized image https://us-east-1.console.aws.amazon.com/systems-manager/parameters/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id/description?region=us-east-1# and use systemctl start ecs, my ECS agent works fine and tasks are picked up successfully, but when I try to manually invoke docker run to start the ECS agent using the instructions above (on the same ECS-optimized AMI), I get the same required attributes error.

Any thoughts? I've been stuck on this for a while :(

Expected Behavior

The ECS Agent should be able to connect to the ECS cluster and pick up tasks.

Observed Behavior

The ECS Agent connects to my ECS cluster but fails to pick up tasks due to an error that my container instance is missing an attribute required by your task

Environment Details

I'm attempting to set up an Ubuntu ECS AMI using the AMI https://aws.amazon.com/marketplace/pp/prodview-iftkyuwv2sjxi?sr=0-2&ref_=beagle&applicationId=AWSMPContessa
Ran a cross-test on the ECS-optimized AMI for https://us-east-1.console.aws.amazon.com/systems-manager/parameters/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id/description?region=us-east-1#

Supporting Log Snippets

[prateekchaudhry]

Hi, thank you for reaching with this issue. I am trying to repro this behavior, so far I am able to launch a simple awsvpc task on a ubuntu VM with agent installed this way. Some questions for further investigation -

• Could you share a simple, sample task definition which gets blocked?
• When a task gets started/running/blocked, could you collect ecs agent logs (https://github.com/aws/amazon-ecs-logs-collector) and email them to [email protected] ?
• I wonder if a different method of installation - using amazon-ecs-init deb package would help for your case (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html#ecs-agent-install-nonamazonlinux)?
• Could you cross verify if the instance started has proper instance role assigned (normally ecsInstaceRole https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html)?