Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix tests to use dedicated bootstrapped service accounts instead of one shared account #10418

Merged
merged 1 commit into from
Apr 16, 2024

Conversation

roaks3
Copy link
Contributor

@roaks3 roaks3 commented Apr 11, 2024

Fixes hashicorp/terraform-provider-google#16247

Previously, one shared service account was bootstrapped and reused for at least 2 of these tests. During the bootstrap, which is called on every VCR run, the policy on the service account was designed to be overwritten (to control for drift). However, overwriting the policy is a read-modify-write operation that cannot be done concurrently. This creates a race condition when the bootstrap functions are run at the same time, which seems to be more common in the VCR case.

The solution chosen here is to create a separate service account per-test. Service accounts are cheap, there are only 5 such uses for them so far, and this allows us to enforce the policy on every run the way we do now. Since the accounts are isolated per-test, they can still be bootstrapped ahead of time, while not running into concurrency issues.

Release Note Template for Downstream PRs (will be copied)


@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 15 insertions(+), 8 deletions(-))
google-beta provider: Diff ( 4 files changed, 15 insertions(+), 8 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2811
Passed tests: 2452
Skipped tests: 356
Affected tests: 3

Click here to see the affected service packages
all service packages are affected

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccDataSourceGoogleServiceAccountAccessToken_basic|TestAccDataSourceGoogleServiceAccountIdToken_impersonation|TestAccSqlUser_postgresIAM

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataSourceGoogleServiceAccountAccessToken_basic[Error message] [Debug log]
TestAccDataSourceGoogleServiceAccountIdToken_impersonation[Error message] [Debug log]
TestAccSqlUser_postgresIAM[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@roaks3
Copy link
Contributor Author

roaks3 commented Apr 11, 2024

/gcbrun

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 15 insertions(+), 8 deletions(-))
google-beta provider: Diff ( 4 files changed, 15 insertions(+), 8 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3570
Passed tests: 3197
Skipped tests: 369
Affected tests: 4

Click here to see the affected service packages
all service packages are affected

Action taken

Found 4 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccDataSourceGoogleServiceAccountAccessToken_basic|TestAccDataSourceGoogleServiceAccountIdToken_impersonation|TestAccHealthcareDatasetIamPolicy|TestAccSqlUser_postgresIAM

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccHealthcareDatasetIamPolicy[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataSourceGoogleServiceAccountAccessToken_basic[Error message] [Debug log]
TestAccDataSourceGoogleServiceAccountIdToken_impersonation[Error message] [Debug log]
TestAccSqlUser_postgresIAM[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@roaks3 roaks3 force-pushed the fix-flaky-sa-tests branch from 04780b4 to 2612ae5 Compare April 11, 2024 21:40
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 15 insertions(+), 8 deletions(-))
google-beta provider: Diff ( 4 files changed, 15 insertions(+), 8 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3570
Passed tests: 3199
Skipped tests: 369
Affected tests: 2

Click here to see the affected service packages
all service packages are affected

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccDataSourceGoogleServiceAccountAccessToken_basic|TestAccSqlUser_postgresIAM

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccDataSourceGoogleServiceAccountAccessToken_basic[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccSqlUser_postgresIAM[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@roaks3 roaks3 requested review from a team and rileykarson and removed request for a team April 16, 2024 15:56
@roaks3 roaks3 merged commit 7041d80 into main Apr 16, 2024
9 of 10 checks passed
@roaks3 roaks3 deleted the fix-flaky-sa-tests branch April 16, 2024 21:05
balanaguharsha pushed a commit to balanaguharsha/magic-modules that referenced this pull request Apr 19, 2024
balanaguharsha pushed a commit to balanaguharsha/magic-modules that referenced this pull request May 2, 2024
balanaguharsha pushed a commit to balanaguharsha/magic-modules that referenced this pull request May 2, 2024
BBBmau pushed a commit to BBBmau/magic-modules that referenced this pull request May 8, 2024
pawelJas pushed a commit to pawelJas/magic-modules that referenced this pull request May 16, 2024
pengq-google pushed a commit to pengq-google/magic-modules that referenced this pull request May 21, 2024
Cheriit pushed a commit to Cheriit/magic-modules that referenced this pull request Jun 4, 2024
pcostell pushed a commit to pcostell/magic-modules that referenced this pull request Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failing VCR test(s): TestAccDataSourceGoogleServiceAccount*Token_*
3 participants