Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(*): wait_for_dependency will wait forever for unhealthy container #551

Closed
wants to merge 2 commits into from

Conversation

StarlightIbuki
Copy link

As the title says.

pongo.sh Outdated Show resolved Hide resolved
@@ -625,20 +625,26 @@ function healthy {
fi

local state
state=$(docker inspect "$iid")
state=$(docker inspect --format='{{.State.Health.Status}}' "$iid")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hesitant with these types of changes, on how we validate status and how we collect that input. We've had plenty of issues in the past due to different Docker versions, or even non-Docket tools being used.

So are these changes necessary?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runner blocks until it times out when a container fails to start, and it gives no reason. This makes it hard to diagnose issues with container setup. I'm trying to improve the situation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can find a use of this in a 7 years ago script: https://github.com/jordyv/wait-for-healthy-container/blob/master/wait-for-healthy-container.sh, so maybe we can consider it a stable interface.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StarlightIbuki don't get me wrong, I like the improvements, but what I'm hesitant about is changing the checks themselves as you do in the above lines.

@@ -609,7 +609,7 @@ function compose {
}


function healthy {
function status {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add comments here specifically what the returned values are?

Then I think there are issues here, it can return 0 for success (for example if health checks are disabled). Yet in the calling code, that is not checked, since it only checks "healthy" as return value.

So I think this currently breaks the "SERVICE_DISABLE_HEALTHCHECK" option

pongo.sh Outdated
msg "Waiting for $dep"

while ! healthy "$iid" "$dep"; do
while [ "$(status "$iid" "$dep")" == "starting" ]; do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we bail out of this loop after a timeout period? eg. 60 seconds?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. I was blindly following the original behavior.

@Tieske
Copy link
Member

Tieske commented Apr 17, 2024

@StarlightIbuki see #554 for an alternative implementation

@Tieske Tieske deleted the fix/infinite-wait-dep branch July 9, 2024 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants