ecs-patterns: When health check failed, ECS deployment stucks

[yoshizawa56]

Describe the feature

When deploying an ECS service using AWS CDK, the deployment process gets stuck until it times out if the health checks fail. This is causing issues in our deployment process and we would like to find a solution.

Use Case

Same as above

Proposed Solution

I propose creating a custom resource that monitors the deployment status of the ECS service and detects if the health checks fail. If the health checks fail, the custom resource will trigger a deployment error to prevent the deployment from stucking. By implementing this custom resource, we can ensure that failed health checks do not cause our deployment process to stall indefinitely.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.68.0

Environment details (OS name and version, etc.)

m1 mac

[pahud]

This is an interesting idea. I believe this is the design from cloudformation that only successful health check makes a successful ecs service deployment, and the health check usually comes from the load balancer.

I think this will need a lot of consideration about the design:

1. how would you implement the health check logic from a custom service for a common ecs service with ALB?
2. what happens if your health checker returns service "unhealthy" while ALB is still checking?
3. what happens if your health checker returns service "healthy" while ALB is still checking?
4. do we really this this feature in aws-ecs library or should we design this as a separate construct outsides aws-cdk-lib?

I am converting this to discussion and any more feedback with details are appreciated.

[yoshizawa56]

Thank you for your response.

Here are my answers to your points:

1. The simplest way to implement this feature is to monitor the deployments property, which is visible through describe-services for ECS. The deployments property has an item called failedTasks, so we can set a threshold for this value. If the threshold is exceeded, we can determine that the ECS deployment has failed and make the create/update of the custom resource fail accordingly. This method is independent of the ALB, so it can also be applied to ECS Services using Service Discovery. Another implementation method could be to actually reference the health check status of the ALB, but this would require accessing the ECS Task information and the implementation would be more complex, which could compromise stability.
2. In this case, the create/update of the health checker fails and CFn rolled back, causing ECS to be rolled back as well.
3. In this case, the create/update of the health checker succeeds, so a rollback will only occur if the deployment of the ECS Service itself fails (by times out).
4. Since this feature deviates from the original deployment philosophy of CFn, it may not be appropriate to include it in the aws-ecs library. However, it may be possible to implement it as part of the ApplicationLoadBalanced*Service functionality in aws-ecs-patterns or as a construct in aws-ecs-patterns.

The following is a partial excerpt of the describe-services result:

{
  "services": [
    {
      "deployments": [
        {
          "id": "ecs-svc/00000000000000000",
          "status": "PRIMARY",
          "taskDefinition": "arn:aws:ecs:ap-northeast-1:123456789012:task-definition/sample-task:2",
          "desiredCount": 1,
          "pendingCount": 1,
          "runningCount": 0,
          "failedTasks": 2,
          "createdAt": 1678842291.695,
          "updatedAt": 1678842291.695,
          "rolloutState": "IN_PROGRESS",
          "rolloutStateReason": "ECS deployment ecs-svc/00000000000000000 in progress."
        }
      ],
    }
  ]
}

[yoshizawa56]

The simplest way to implement this feature is to monitor the deployments property, which is visible through describe-services for ECS. The deployments property has an item called failedTasks, so we can set a threshold for this value. If the threshold is exceeded, we can determine that the ECS deployment has failed and make the create/update of the custom resource fail accordingly. This method is independent of the ALB, so it can also be applied to ECS Services using Service Discovery.

I found this solution to the problem.
The order of deployment is service -> health checker because health checker depends on the ecs service. The health checker needs to be deployed at the same time as the service to monitor the deployment status, so this mechanism is unlikely to work.

Assuming that the task definition is not shared by multiple services, it may be possible to eliminate the dependency on the service by finding the service that references the task definition in a custom resource.

[alper]

I'm working with ECS and it's awful. If there's the slightest problem with any of the containers, the entire deployment hangs. This can't be the way this is supposed to work?

I also can't redeploy because the entire stack is in a stuck state.