Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bridge] Refactor Bridge.controlInstances and add tracing (4/5) #10727

Merged
merged 1 commit into from
Jun 24, 2022

Conversation

geropl
Copy link
Member

@geropl geropl commented Jun 17, 2022

Description

This PR prepares for fixing a bug with prebuilds being stuck in queued indefinitely in rare cases. (4/5 in the chain)
To that end, if mostly refactors the method controlInstances in Bridge, and adds tracing to it.

Related Issue(s)

Context: #9395
Follow-up: #10882

How to test

  • start a workspace on this branch and ./dev/preview/install-k3s-kubeconfig.sh
  • start a workspace in the preview env
  • enable debug logs for ws-manager-bridge: gpctl debug log ws-manager-bridge
  • check the logs and notice this pattern:
    • "Controlling instances..."
    • "Controlling running instances..."
    • "Done controlling running instances."
    • "Done controlling instances."

Release Notes

fix prebuilds stuck in `queued` indefinitely

Documentation

  • /werft with-preview

@geropl geropl force-pushed the gpl/9395-stuck-prebuilds-4 branch from 59c02a3 to c22d66d Compare June 23, 2022 15:18
);

disconnectStarted = Number.MAX_SAFE_INTEGER; // Reset disconnect period
} catch (err) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This inner try-catch is preparation for a followup PR.


const runningInstances = await this.workspaceDB.trace(ctx).findRunningInstancesWithWorkspaces(installation);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are three changes in this method:

  • swapped local findRunningInstancesWithWorkspaces with external call
  • added span (and thus: try-catch to finish() it)
  • extraction of markWorkspaceInstanceAsStopped

@werft-gitpod-dev-com
Copy link

started the job as gitpod-build-gpl-9395-stuck-prebuilds-4.7 because the annotations in the pull request description changed
(with .werft/ from main)

@geropl geropl changed the title [bridge] Use PrebuildUpdater in MetaInstanceController when stopping workspace instances waiting for a imagebuild (3/3) [bridge] Refactor Bridge.controlInstances and add tracing (4/5) Jun 23, 2022
@geropl geropl marked this pull request as ready for review June 24, 2022 09:57
@geropl geropl requested a review from a team June 24, 2022 09:57
@github-actions github-actions bot added the team: webapp Issue belongs to the WebApp team label Jun 24, 2022
@roboquat roboquat merged commit 40b8202 into main Jun 24, 2022
@roboquat roboquat deleted the gpl/9395-stuck-prebuilds-4 branch June 24, 2022 10:33
@roboquat roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Jun 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: webapp Meta team change is running in production deployed Change is completely running in production release-note size/L team: webapp Issue belongs to the WebApp team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants