Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ws-manager-bridge] Skip stale prebuild events #9116

Merged
merged 1 commit into from
Apr 6, 2022

Conversation

easyCZ
Copy link
Member

@easyCZ easyCZ commented Apr 4, 2022

Description

Do not process stale WorkspaceStatus events when updating prebuilds.

Metrics show that we receive stale events relatively frequently on staging.

Screenshot 2022-04-05 at 21 38 10

This change will result in increased accuracy of our prebuild status.

Related Issue(s)

Relates to:

Fixes

How to test

Requires a degree of luck, as it depends on event ordering on the message bus, but:

  1. Start a prebuild
  2. port-forward ws-manager-bridge on port 9500 and fetch metrics with curl localhost:9500/metrics
  3. Check metrics contain gitpod_ws_manager_bridge_stale_prebuild_events_total with a positive count
  4. Check ws-manager-bridge logs contain Stale prebuild event received, skipping.

Release Notes

ws-manager-bridge skips stale prebuild events

/hold

@easyCZ easyCZ force-pushed the mp/prebuild-state-set-version branch 2 times, most recently from 894c265 to 92dbc78 Compare April 5, 2022 08:23
Base automatically changed from mp/prebuild-state-set-version to main April 5, 2022 10:57
@roboquat roboquat added size/M and removed size/XS labels Apr 5, 2022
@easyCZ easyCZ force-pushed the mp/prebuild-skip-stale-events branch from 51d261f to 5600e68 Compare April 5, 2022 19:34
Comment on lines 68 to 72
// We've gotten an event which is younger than one we've already processed. We shouldn't process the stale one.
span.setTag("updatePrebuiltWorkspace.staleEvent", true);
this.prometheusExporter.recordStalePrebuildEvent();
return;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only real logic change, rest is formatting from prettier.

@easyCZ easyCZ force-pushed the mp/prebuild-skip-stale-events branch from 5600e68 to b477b15 Compare April 5, 2022 19:37
@easyCZ easyCZ marked this pull request as ready for review April 5, 2022 19:45
@easyCZ easyCZ requested a review from a team April 5, 2022 19:45
@github-actions github-actions bot added the team: webapp Issue belongs to the WebApp team label Apr 5, 2022
Copy link
Member

@AlexTugarev AlexTugarev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@easyCZ
Copy link
Member Author

easyCZ commented Apr 6, 2022

Thanks for the review @AlexTugarev!

/unhold

@roboquat roboquat merged commit eb72c2f into main Apr 6, 2022
@roboquat roboquat deleted the mp/prebuild-skip-stale-events branch April 6, 2022 06:59
@roboquat roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: webapp Meta team change is running in production deployed Change is completely running in production release-note size/M team: webapp Issue belongs to the WebApp team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants