Run prebuilds in the storage-bucket's region

[meysholdt]

Running a prebuild in the same geographical region of the storage bucket to which the prebuild will be uploaded saves us costs in two places:

• for all prebuilds, the compute egress costs from uploading the completed prebuild to its bucket.
• for incremental prebuilds, the storage egress of downloading a prebuild with the intention of updating it.

proposed solution: ws-manager-bridge/server should be aware of the geographical region of the prebuild's storage bucket and route the prebuild to a workspace cluster in the same region.

While we may not have capacity to fix this immediately it would be great to have this on the radar.

Related:

• https://github.com/gitpod-io/ops/issues/120
• Cache prebuilds locally (to a region) #6145
• https://github.com/gitpod-io/ops/issues/146
• https://github.com/gitpod-com/gitpod/issues/5773

[jankeromnes]

This sounds like a sensible and really good idea. 👍

(Unfortunately, it does not solve the egress due to popular prebuilds getting consumed in all regions -- for example, our starter templates typically have one prebuild that gets opened in many thousands of workspaces in all regions -- if a popular prebuild is stored in a single region, then all other regions will constantly transfer the same prebuild multiple times across oceans)

[JanKoehnlein]

We're a bit swamped at the moment. Moving this to our backlog to bring it up later

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaal]

If this issue becomes stale, will it be difficult to remember/find this feature-request in the future?

[jankeromnes]

I think we should raise the priority of this issue, because:

• We see a sharp uptake in Prebuilds, some of them with Incremental Prebuilds enabled
• Many projects with Incremental Prebuilds enabled do this:

1. Start new prebuild in US
2. Download old prebuild (base) from the EU bucket ($$$)
3. Build the new increment in US
4. Upload the new snapshot back to the EU bucket ($$$)

... repeated on every single push to such a repository (and some projects have a very high push-rate).

Also, is team WebApp the correct owner for this?

Cc @geropl

[geropl]

Ah, didn't look long enough to find this one, but created a fresh one: #9872

@jankeromnes Could you please replace the new issue with this one? And place in on top ot the "scheduled" column? 🙏

[csweichel]

Before this can work we need to solve #10134 - or make OTS region aware

[AlexTugarev]

Thanks for the sharing this!

@jldec, I think this issue here is not actionable at the moment. Let's pull out of scheduled, please!

[csweichel]

@AlexTugarev one path forward would be to make OTS region aware (i.e. introduce something like eu.gitpod.io or the likes) and use that to name the OTS. This sits directly within webapps space.

[jldec]

Removed from scheduled