Epic: Single Application Cluster

[svenefftinge]

Summary

We want to remove the multi-region application deployment we have in gitpod.io which includes removing the db-sync component and maintaining multiple databases, because it adds too much complexity and overhead for the value.

The target architecture is one application installation (in US) managing multiple workspace clusters in multiple regions.

Internal RFC

Preparation

Preview Give feedback

1. [ws-manager-api] Add region for cluster selection #16136
   release-note-none size/XL +2
2. Use location header from load balancer to select region #16271
   team: webapp +1
   
3. Move all jobs to US cluster #16268
   +0
   

Performing the switch

Preview Give feedback

1. Remove/disable EU application cluster from load balancer #16272
   team: webapp +1
   

Cleanup

Preview Give feedback

1. Disable db-sync, enable server deletion of records #16458
   +0
   
2. Remove slow-server, toxi-proxy #16447
   +0
   
3. Remove cloud-sql-proxy-global #16451
   team: webapp +1
   
4. Disable migrations in both regions #16459
   component: infrastructure team: webapp +2
   
5. https://github.com/gitpod-io/ops/issues/8562
6. Identify approach to removal of IDE components from EU #16457
   +0
   
7. Remove db-sync from wrapper chart #16581
   +0
8. Remove db-sync component #16582
   +0
9. Decomission staging EU cluster #16603
   +0
   
10. Decomission staging EU database #16604
    +0
    
11. Decomission production EU cluster #16606
    +0
    
12. Decomission production EU database #16605
    +0
    

Previous work and documents

• Epic: Get rid of db-sync #9198
• https://www.notion.so/gitpod/Get-rid-of-db-sync-Milestone-plan-dfd92243ab8940a280adf71f75039da2
• https://www.notion.so/gitpod/WebApp-Epic-Get-rid-of-db-sync-a790d1d453fa482aa5041a8e0be39c28

[kylos101]

👋 hey @svenefftinge,

Do you need help on the Workspace side?

When were y'all hoping to make these changes? @atduarte mentioned WebApp team was hoping to do the consolidation/move work this week.

There are a few things I can think of on Workspace side, can you think of anything else? For example, I was thinking the following would need to change:

• For existing registered clusters, can you set region in the database? Are there any downtime expectations?
• For new clusters, there's the cluster create job, it looks like we can reuse the existing region parameter values, and will need to (1) update the gpctl CLI version after this merges, (2) pass the new region arg on cluster register, (3) avoid registering with a non-governing meta cluster, and (4) remove the non-governing meta-cluster parameter from the job
• For the cluster delete job, we'll need to avoid non-governing meta clusters, and remove the related parameter in the job.
• The traffic shifting job isn't used, so we're all good there.

cc: @geropl @aledbf

[svenefftinge]

Handing over DRI role to @easyCZ so I can focus on #15980

[kylos101]

Thanks for the heads up, @svenefftinge ! @easyCZ lemme know if you'd like to plan later this week? As a heads up, I'm deploying for Workspace Team next week, too, and can help with related changes.

[kylos101]

We should plan on the following too:

1. For the initial leg of the consolidation, continue registering workspace clusters in the EU region, cordoning US, just incase we have to fallback
2. Once the EU infrastructure is removed, then we can remove the dual cluster registration
3. Writing a cluster selection test (if it doesn't exist) to include the new region field

[kylos101]

Timing wise we'd want to:

1. Make the database change to update existing clusters with the proper region
2. Flip the new feature flag, knowing the only requests making it to the US application cluster are for the US, and route to US workspace clusters
3. Uncordon and update score for EU workspace clusters in the US application cluster, still no traffic will be sent to them until
4. We finally update the GCP load balancer to remove the EU application cluster backend

[kylos101]

Before we do this, an internal test might look like:

1. Prepare an ephemeral cluster in the EU region, in the US application cluster, uncordon and update score and set admission constraint new-workspace-cluster
2. Make the database change to update existing clusters with the proper region
3. Flip the new feature flag, knowing the only requests making it to the US application cluster are for the US, and route to US workspace clusters...but it would be exclusive to few Gitpodders (us) working this change, still no traffic will be sent to it until
4. We finally update the GCP load balancer to remove the EU application cluster backend, we'd use a VPN to get ourselves to the US application cluster, if needed

[easyCZ]

Some additional details for the execution of the switch - Notion doc (internal)

[easyCZ]

This is now complete, including removal of unused code.

What remains, and is a potential candidate for removal, is the regional vs global message bus distinction. This will be further cleaned up with the PAYG transition so leaving that for now.