Task containers dependencies resolution stuck forever #2579

taraspos · 2020-08-18T14:43:57Z

Summary

Task stuck in Pending state because of some problems during containers dependencies resolution.
Setting Start timeout doesn't have any effect as well.
This is reopening of my previous issue #2350, with the additional details and a Task Definition to reproduce the error.

Description

I have 3 containers:

Container name	Essential	Depends On Container	Depends On State	Start Timeout
exit1	false			10
mysql	true	exit1	SUCCESS	30
nginx	true	mysql	START	60

(nginx depends on mysql which depends on exit1)

When container exit1 fails with the exit code 1, containers mysql and nginx remains in the PENDING state forever.
When container exit1 succeeds with exit code 0 everything works fine.

I also tried to set container nginx dependencies like:

      [
        {
          "containerName": "mysql",
          "condition": "START"
        },
        {
          "containerName": "exit1",
          "condition": "SUCCESS"
        }
      ]

the result is the same.

However, behavior seems to be correct if the order of dependencies for nginx is changed:

      [
        {
          "containerName": "exit1",
          "condition": "SUCCESS"
        },
        {
          "containerName": "mysql",
          "condition": "START"
        }
      ]

Expected Behavior

The task is failed to start.

Observed Behavior

Task stuck in the PENDING state forever

Environment Details

ECS Agent version: 1.43.0

Task Definitions to reproduce:

With transitive dependencies - stuck as PENDING ❌

{
    "containerDefinitions": [
        {
            "command": [
                "cat",
                "123"
            ],
            "image": "alpine",
            "startTimeout": 10,
            "name": "exit1",
            "essential": false
        },
        {
            "image": "nginx",
            "startTimeout": 60,
            "dependsOn": [
                {
                    "containerName": "mysql",
                    "condition": "START"
                }
            ],
            "name": "nginx"
        },
        {
            "image": "mysql:5.7",
            "startTimeout": 30,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "mysql"
        }
    ],
    "memory": "100",
    "family": "reproduce-dependency-problem",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "128"
}

With unordered multiple dependencies - stuck as PENDING ❌

{
    "containerDefinitions": [
        {
            "command": [
                "cat",
                "123"
            ],
            "image": "alpine",
            "startTimeout": 10,
            "name": "exit1",
            "essential": false
        },
        {
            "image": "nginx",
            "startTimeout": 60,
            "dependsOn": [
                {
                    "containerName": "mysql",
                    "condition": "START"
                },
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "nginx"
        },
        {
            "image": "mysql:5.7",
            "startTimeout": 30,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "mysql"
        }
    ],
    "memory": "100",
    "family": "reproduce-dependency-problem",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "128"
}

With logically ordered multiple dependencies - task STOPPED as expected ✅

{
    "containerDefinitions": [
        {
            "command": [
                "cat",
                "123"
            ],
            "image": "alpine",
            "startTimeout": 10,
            "name": "exit1",
            "essential": false
        },
        {
            "image": "nginx",
            "startTimeout": 60,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                },
                {
                    "containerName": "mysql",
                    "condition": "START"
                }
            ],
            "name": "nginx"
        },
        {
            "image": "mysql:5.7",
            "startTimeout": 30,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "mysql"
        }
    ],
    "memory": "100",
    "family": "reproduce-dependency-problem",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "128"
}

The text was updated successfully, but these errors were encountered:

shubham2892 · 2020-08-18T19:02:01Z

@trane9991 Sorry that you are facing this issue, I will try to reproduce this on my end.

Meanwhile, Is it possible for you to send the task definition(with which you are seeing the issue) and Agent logs to [email protected].

taraspos · 2020-08-18T19:09:03Z

Hey @shubham2892
Thanks for the quick reply, I published the Task Definitions to reproduce the issues under the collapsable spoilers in the Task Definitions to reproduce section of the issue description :)

taraspos · 2020-08-18T19:17:38Z

Let me know if you are able to reproduce that because it reproduces in 100% cases for me with Task Definitions shared in the issue description.

shubham2892 · 2020-08-19T21:36:14Z

I was able to reproduce the pending state behavior with With transitive dependencies task def and With unordered multiple dependencies, will mark this as a bug and work on getting this fixed.

ubhattacharjya · 2020-09-08T17:06:24Z

Hi,

The PR for working on the fix for the ordered container dependency problem is #2615.

Regards,
Utsa

ellenthsu · 2020-09-18T22:38:26Z

This fix has been released as part of ECS Agent 1.44.4: https://github.com/aws/amazon-ecs-agent/releases/tag/1.44.4

Please perform an update of the Agent: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html or you can find the latest ECS Optimized AMIs containing ECS Agent 1.44.4 here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html

lucabelmonte · 2022-05-13T14:54:38Z

Hi,

As you can see in the attached screenshot, i have the same error as mentioned above.
I'm currently using the agent version 1.60.1.

Here my task definition:

{
  "ipcMode": null,
  "executionRoleArn": null,
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/test-sequential-container",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [],
      "command": [
        "/bin/bash",
        "-c",
        "echo ciao && sleep 10 && exit 1"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": 128,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "ubuntu",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": false,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "container-1"
    },
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/test-sequential-container",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [],
      "command": [
        "/bin/bash",
        "-c",
        "echo container2 && sleep 10"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": 128,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "ubuntu",
      "startTimeout": 10,
      "firelensConfiguration": null,
      "dependsOn": [
        {
          "containerName": "container-1",
          "condition": "SUCCESS"
        }
      ],
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": false,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "container-2"
    },
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/test-sequential-container",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [],
      "command": [
        "echo",
        "container3"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": 128,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "ubuntu",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": [
        {
          "containerName": "container-2",
          "condition": "SUCCESS"
        }
      ],
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "container-3"
    }
  ],
  "placementConstraints": [],
  "memory": null,
  "taskRoleArn": null,
  "compatibilities": [
    "EXTERNAL",
    "EC2"
  ],
  "taskDefinitionArn": "arn:aws:ecs:eu-west-1:XXXX:task-definition/test-sequential-container:15",
  "family": "test-sequential-container",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-ordering"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "EC2"
  ],
  "networkMode": null,
  "runtimePlatform": null,
  "cpu": null,
  "revision": 15,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

What am i missing?

Mentions: @ellenthsu

Thanks,

angelcar · 2022-05-13T23:03:05Z

Hi! @lucabelmonte
Thanks for reaching out.

I was able to repro the issue using your task definition. I will re-open this issue and label it as a bug. We'll work to resolve it.

Correct me if I'm wrong, but I think in this particular case the expectation would be that container-2 would just stop since SUCCESS is never going to happen.

lucabelmonte · 2022-05-16T08:46:52Z

Hi @angelocar,
Thank you for your answer.

Yes exactly. I was expecting the second container container-2 to go into a STOPPED state and also the runned task definition (for a broken dependence, in this case).
I also noticed that when there are only 2 containers in a chain, it works.

angelcar · 2022-05-25T19:00:22Z

@lucabelmonte, a fix has been merged and will soon be released.

gregmoy · 2022-07-06T22:58:01Z

I wasn't having this problem before but now I am after upgrading from 1.61.1 to 1.61.3...

mssrivas · 2022-07-12T18:54:23Z

The fix was released in 1.61.2

jonioni · 2022-10-14T01:50:22Z

Wondering if this has been applied to Fargate? Encountered similar issue on fargate tasks.

shubham2892 added the kind/bug label Aug 19, 2020

ubhattacharjya mentioned this issue Sep 3, 2020

Solve unordered container dependency problem #2615

Merged

ubhattacharjya added the pending release label Sep 9, 2020

ellenthsu closed this as completed Sep 18, 2020

angelcar reopened this May 13, 2022

angelcar removed the pending release label May 13, 2022

angelcar mentioned this issue May 19, 2022

Stop container when dependencies are non-resolvable #3218

Merged

angelcar added the kind/tracking This issue is being tracked internally label May 25, 2022

mssrivas closed this as completed Jul 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task containers dependencies resolution stuck forever #2579

Task containers dependencies resolution stuck forever #2579

taraspos commented Aug 18, 2020 •

edited

Loading

shubham2892 commented Aug 18, 2020 •

edited

Loading

taraspos commented Aug 18, 2020

taraspos commented Aug 18, 2020

shubham2892 commented Aug 19, 2020

ubhattacharjya commented Sep 8, 2020

ellenthsu commented Sep 18, 2020

lucabelmonte commented May 13, 2022 •

edited

Loading

angelcar commented May 13, 2022

lucabelmonte commented May 16, 2022 •

edited

Loading

angelcar commented May 25, 2022

gregmoy commented Jul 6, 2022

mssrivas commented Jul 12, 2022

jonioni commented Oct 14, 2022

Task containers dependencies resolution stuck forever #2579

Task containers dependencies resolution stuck forever #2579

Comments

taraspos commented Aug 18, 2020 • edited Loading

Summary

Description

Expected Behavior

Observed Behavior

Environment Details

Task Definitions to reproduce:

shubham2892 commented Aug 18, 2020 • edited Loading

taraspos commented Aug 18, 2020

taraspos commented Aug 18, 2020

shubham2892 commented Aug 19, 2020

ubhattacharjya commented Sep 8, 2020

ellenthsu commented Sep 18, 2020

lucabelmonte commented May 13, 2022 • edited Loading

angelcar commented May 13, 2022

lucabelmonte commented May 16, 2022 • edited Loading

angelcar commented May 25, 2022

gregmoy commented Jul 6, 2022

mssrivas commented Jul 12, 2022

jonioni commented Oct 14, 2022

taraspos commented Aug 18, 2020 •

edited

Loading

shubham2892 commented Aug 18, 2020 •

edited

Loading

lucabelmonte commented May 13, 2022 •

edited

Loading

lucabelmonte commented May 16, 2022 •

edited

Loading