Prebuild runs only when opening new workspace

[shaal]

Bug description

ddev is set to have automatic prebuilds.
https://gitpod.io/t/ddev/ddev shows that master prebuild ran successfuly.

Whenever I open a new workspace for ddev https://gitpod.io/#https://github.com/drud/ddev - I see the prebuild steps running in the terminal, instead of pre-loading a prebuild and then in terminal display the message "x minutes saved by prebuild".

Steps to reproduce

1. Open https://gitpod.io/#https://github.com/drud/ddev
2. See that prebuild is running in terminal

Workspace affected

No response

Expected behavior

If a prebuild cannot be loaded, prebuild should run in the background so future workspaces can load the prebuild.

Example repository

No response

Anything else?

I am sure that if I push any code changes, or if I manually trigger a prebuild using the URL - a new prebuild process will run, and it will "fix" the current issue.

This issue is trying to catch a faulty prebuild and automatically run it again, or at least allow the UI to re-run prebuild, without committing new code.
cc: @jankeromnes, @rfay

[geropl]

This sounds as if the first problem is instransparency as to why the prebuild is not triggered earlier (bug or config issue?). The second one is to find out what the actual issues, and unblock @shaal

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaal]

Please mark this issue with the label meta: never-stale.

Also, did anyone find the automatic activities of stale bot useful? (Beside the obvious "the number of open issues gets magically reduced over time")

[rfay]

Again, this is an important issue, Please mark this issue with the label meta: never-stale and reopen it.

[AlexTugarev]

Hi @rfay and @shaal!

I just checked if a new workspace on drud/ddev repository is opening with a new prebuild 👍🏻 It does 😌
Then I checked with the database records for the past 500 prebuilds to see if there were any recent issues. The last error was 2 months ago, and seems to be somewhat exceptional ("workspace is not ready" error message.)

Considering this, I hope the bug fixes were made over the past month did improve your experience with prebuilds in general. I guess the urgent issue from the original report does no longer exist, but it's important to verify with you. Please feel free to open new tickets for issues (even if also related to prebuilds.)

Also, consider checking the Events page, which is currently a preview feature: https://gitpod.io/t/drud/ddev/events
In case you're missing a prebuild when opening a workspace, it could give you an initial sense of what happened with a prebuild trigger.

[rfay]

No @AlexTugarev it appears to me that the behavior is the same as it has been for many months. I just did as described in this issue and created https://drud-ddev-wvzajgo5czi.ws-us65.gitpod.io/ and nothing was any different. The prebuild had not been done, and it went through the whole prebuild again instead of using an existing prebuild.

BTW, https://www.gitpod.io/t/drud/ddev/events is a 404 for me. Sounds like a nice future feature though, would love to know more.

This issue remains consistent and a problem. Perhaps the prebuilds get done but they're never used?

[AlexTugarev]

@rfay, thanks for the workspace id, I'm looking into this shortly.

[AlexTugarev]

@rfay, once again, thanks for revealing this issue and showing great patience 🙏🏻

We looked into the project details and learned about a bug which leads to prebuilds being cancelled, because the project was considered inactive incorrectly. Database entries clearly show recent workspace starts days before a push event was registered, thus prebuild should have been executed as expected.

The fact that I started a workspace from a prebuild yesterday, but you couldn't afterwards, can be explained by the prebuild being cancelled due to "inactivity" in between.

Internally, we discussed how to unblock you now, and how to fix the UX in a way, that inactive project get reactivated automatically. For now, please try to rerun a prebuild on the default branch on https://www.gitpod.io/t/drud/ddev

BTW, https://www.gitpod.io/t/drud/ddev/events is a 404 for me. Sounds like a nice future feature though, would love to know more.

Ah, that's the wrong URL (www for website), try. https://gitpod.io/t/drud/ddev/events

[rfay]

The problem, you see, is that I'm not the only one using this, and it's used in many different ways. So manually running a prebuild (when???) is not really an option.

The same thing happens of course with ddev gitpod launcher as well. But it's run with different arguments by random users, requiring a different prebuild.

https://gitpod.io/#DDEV_REPO=https%3A%2F%2Fgithub.com%2Fdrud%2Fd9simple,DDEV_ARTIFACTS=https%3A%2F%2Fgithub.com%2Fdrud%2Fd9simple-artifacts/https://github.com/drud/ddev-gitpod-launcher/ for example.

There is a set of different interacting problems here,

• Prebuild runs only when opening new workspace #9564 (this one, prebuild isn't used so prebuild always happens)
• docker bridge gateway is not available during prebuilds #9565 (networking fails during prebuild, so people can't really use the results)

It's not me that you need to unblock, it's all DDEV users that use gitpod this way, and Drupalpod users.

I don't think there's a manual-user-should-go-fix-it path to sort this out, gitpod just needs to work as advertised.

BTW https://gitpod.io/t/drud/ddev/events is also 404.

[AlexTugarev]

It's not me that you need to unblock, it's all DDEV users that use gitpod this way, and Drupalpod users.

Absolutely. We understood the issue you're seeing and can reproduce. There is an inactivity detection for project which effectively disables prebuilds for your project incorrectly, see #12952. In addition to fixing the behavior behind the scene, we're going to implement a warning alert from #9232 with a helpful action to resume prebuilds after longer time of inactivity.

BTW https://gitpod.io/t/drud/ddev/events is also 404.

That was a shot in the dark, as I'm not in your team. Looking at the description above, your team slug is not drud, but ddrud, right? So the URL should contain the correct team slug: https://gitpod.io/t/ddrud/ddev/events

[rfay]

The slug is in fact drud, and in this case the project drud/ddev. If you describe how to navigate to the events URL instead of the link I'll find it and tell you what it is.

[shaal]

@rfay yeah, these URLs do not work.
Go to your Gitpod's user page.
Top-left corner, choose ddev as team (I think that is you and I there)

That would take you to https://gitpod.io/t/ddev/ddev
then, manually add /events at the end of URL (for me it's https://gitpod.io/t/ddev/ddev/events)

[shaal]

@AlexTugarev @geropl we're seeing a similar issue in this repo - https://github.com/drud/ddev-gitpod-launcher

I attached 3 screenshots:

1. prebuilds list for that project
2. events list for that project
3. The reason for cancellation (Prebuild has been cancelled. Either a newer commit was pushed to the same branch, a user cancelled it manually, or the prebuild rate limit has been exceeded.)

It is not clear why these prebuilds stopped working. Can you please help?

[geropl]

@jankeromnes Given our recent changes [1]: Is there a combination of config that you could recommend shaal to use to fix this problem? 🤔

[jankeromnes]

Thanks for the ping @geropl! Hi @shaal -- long time! 😁

So, to understand the problem at hand, I'm ignoring the issue title and all the previous comments before #9564 (comment) (but please let me know if I'm missing something).

1. prebuilds list for that project

Aha, I see that your prebuilds are getting cancelled. There are two recent changes that could help figure out why:

• If you go to the "Branches" tab (i.e. the "project overview"), you may see a warning at the top that explains what's wrong, with a direct button to resume your prebuilds

• If you open any of the cancelled prebuilds, the error message now contains the exact cancellation reason (this is something we removed by mistake, but I've added it back -- it's located at the end of the error message)

2. events list for that project

Oh, exciting, that must have happened while I was on holidays -- first time I'm hearing about this feature. 😅 👀

3. The reason for cancellation (Prebuild has been cancelled. Either a newer commit was pushed to the same branch, a user cancelled it manually, or the prebuild rate limit has been exceeded.)

Indeed, sorry that we accidentally removed the explicit cancellation reason from this message. It's back now, as mentioned above -- at the end of the message you've posted.

[geropl]

Closing this issue because I think we already mitigated this. Users now have more control over Prebuilds on the project level, and should be able to avoid the situations mentioned above: