Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: fix mergify matrix integration tests #8267

Merged
merged 4 commits into from
Aug 29, 2023

Conversation

mhofman
Copy link
Member

@mhofman mhofman commented Aug 29, 2023

Description

Mergify can get stuck when matrix tests fail, not realizing that the PR cannot make forward progress anymore.

This change reports a matrix job result as normal job, and switches mergify to using that job's result instead

Security Considerations

None

Scaling Considerations

None

Documentation Considerations

None

Testing Considerations

Will manually test variations on this PR

Upgrade Considerations

None

@mhofman mhofman added the force:integration Force integration tests to run on PR label Aug 29, 2023
@mhofman mhofman force-pushed the mhofman/fix-mergify-integration-results branch from abbaacd to f327e26 Compare August 29, 2023 00:42
@mhofman mhofman added bypass:linear-history Allow merge and fixup commits in PR and removed force:integration Force integration tests to run on PR bypass:linear-history Allow merge and fixup commits in PR labels Aug 29, 2023
@mhofman mhofman marked this pull request as draft August 29, 2023 02:12
@mhofman mhofman marked this pull request as ready for review August 29, 2023 02:13
@mhofman mhofman added force:integration Force integration tests to run on PR and removed force:integration Force integration tests to run on PR labels Aug 29, 2023
@mhofman mhofman force-pushed the mhofman/fix-mergify-integration-results branch from d896682 to 13925e9 Compare August 29, 2023 03:06
@mhofman mhofman added bypass:automerge Allow manually merging a PR and removed force:integration Force integration tests to run on PR labels Aug 29, 2023
@mhofman mhofman force-pushed the mhofman/fix-mergify-integration-results branch from 13925e9 to ddd6443 Compare August 29, 2023 03:13
@mhofman mhofman added the force:integration Force integration tests to run on PR label Aug 29, 2023
@mhofman mhofman requested a review from michaelfig August 29, 2023 03:17
.mergify.yml Outdated
Comment on lines 10 to 11
- or:
- label=bypass:integration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not go all the way, and have a single integration-result? Is there something I'm missing?

Then most of this logic can collapse into just:

    - or:
          - label=bypass:integration
          - check-success=integration-result
          - check-neutral=integration-result
          - check-skipped=integration-result

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I guess. I had tried that before, but with the wrong if condition. This should work just as well

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I also removed the label=bypass:integration because that label being applied will caused the result job to be skipped, and makes the logic for mergify simpler to realize the condition has no other way to succeed.

@mhofman mhofman force-pushed the mhofman/fix-mergify-integration-results branch from ddd6443 to 98a408c Compare August 29, 2023 03:36
Copy link
Member

@michaelfig michaelfig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very pretty and squeaky clean! Just one question, but I'll approve assuming you have a good answer.

@@ -311,3 +317,22 @@ jobs:
timeout-minutes: 4
with:
datadog-token: ${{ secrets.DATADOG_API_KEY }}

integration-test-result:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love it! This is totally clear and maintainable.

Comment on lines 276 to 277
# Mark a failure of `test` bootstrap job as a successful result
continue-on-error: ${{ matrix.bootstrap-version == 'test' }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I don't really know the motivation behind having a separate 'test' bootstrap version (I think just that it was different than the 'main' mode for vaults), is there a reason its result should be ignored? Is this just codifying behaviour we already had, avoiding a flake, or something else?

@mhofman mhofman force-pushed the mhofman/fix-mergify-integration-results branch from 98a408c to 3e0bc6c Compare August 29, 2023 03:49
@mhofman mhofman merged commit eb2d030 into master Aug 29, 2023
@mhofman mhofman deleted the mhofman/fix-mergify-integration-results branch August 29, 2023 04:13
@mhofman mhofman linked an issue Aug 29, 2023 that may be closed by this pull request
@mhofman mhofman linked an issue Aug 29, 2023 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bypass:automerge Allow manually merging a PR force:integration Force integration tests to run on PR
Projects
None yet
2 participants