RTD PR builds get stuck

[pllim]

Details

• Read the Docs project URL: https://readthedocs.org/projects/astropy/
• Build URL (if applicable): https://readthedocs.org/projects/astropy/builds/15566029/
• Read the Docs username (if applicable): pllim

Expected Result

Build to run after "concurrency limit" is gone.

Actual Result

The job got stuck at "triggered (Concurrency limit reached (2), retrying in 5 minutes.)" state for many hours. cc @astrojuanlu

I tried to close and re-open astropy/astropy#12587 in the hope to get past that roadblock but the new job won't start because it is marked as "duplicate".

Hope you can help. Thanks!

[astrojuanlu]

I'm away from keyboard now but indeed this looks like some sort of weird race condition to me. I recall that it has happened to other projects in the past.

[astrojuanlu]

After some time, the builds ended up passing:

So, one could say that "everything worked as expected", but it is still not clear to me what was the second build that was supposedly running.

In any case, I believe the solution here would have been to cancel the running and queued builds and start again.

[astrojuanlu]

Found the issue: #7660 (also opened by @pllim)

[humitos]

I think this problem is related to #4386.

We have a task that checked for blocked/invalid/ghost builds and cancel them. Those blocked/invalid/ghost builds are builds that for some reason failed but we weren't able to detect, so we think they are still "running". The task checks for "running" builds and kill those that have been running for a long long time --in the meanwhile, new builds may be blocked (because of concurrency limit) and that's why you get the ⚠️ icon in the build list. Once these ghost builds are killed, your quota is freed up and you are able to run a build again.

I tried to solve this problem by detecting those ghost builds earlier and kill them in #8269 but it requires more testing to be trustable.

[pllim]

If you think this problem is being tracked in other issues already, feel free to close. Thanks!