Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add func to wait for fleet condition without Fatalf #1065

Merged

Conversation

aLekSer
Copy link
Collaborator

@aLekSer aLekSer commented Sep 24, 2019

Fix stress test race condition.
Rename function WaitForFleetCondition() -> AssertFleetCondition() as it not only wait but also asserts.
It is forbidden to call t.Fatalf() from go routine other than main test routine.

Closes #1055.

@aLekSer aLekSer changed the title Add func to wait for fleet condition without Failf Add func to wait for fleet condition without Fatalf Sep 24, 2019
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 53def889-abed-48c3-8192-49923abb5846

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1065/head:pr_1065 && git checkout pr_1065
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.1.0-b5cad0c

@aLekSer aLekSer force-pushed the wait-for-fleet-cond-parallel branch 2 times, most recently from f57c319 to e8692b2 Compare September 25, 2019 13:11
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: c6a16e58-623a-4236-907f-f5bd22475a93

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1065/head:pr_1065 && git checkout pr_1065
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.1.0-f57c319

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: ace0f078-7a4c-465d-b118-f1cfaef5395e

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1065/head:pr_1065 && git checkout pr_1065
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.1.0-e8692b2

@aLekSer aLekSer marked this pull request as ready for review September 26, 2019 14:50
err := f.WaitForFleetConditionParallel(t, flt, condition)
if err != nil {
// Do not call Fatalf() from go routine other than main test go routine, because it could cause a race
t.Fatalf("error waiting for fleet condition on fleet %v", flt.Name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I'm looking at this, should this function instead return an error if it fails, much like we do for other Wait* commands (example). That would make it simpler no? (and we no longer need to pass through testing.T either.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a 73 occurrences of WaitForFleetCondition() so it is handy to enclose this Fatalf() function inside this helper ( if it called from main test routine).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I would argue should really be an AssertFleetCondition - rather than a wait. Because it's making an assertion, rather than waiting for a result.

@aLekSer aLekSer force-pushed the wait-for-fleet-cond-parallel branch from e8692b2 to e221a98 Compare September 30, 2019 12:15
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 25a4b2f1-a469-48ab-be7c-eae4a66e8343

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1065/head:pr_1065 && git checkout pr_1065
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.1.0-e221a98

Copy link
Member

@markmandel markmandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent!

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aLekSer, markmandel
To complete the pull request process, please assign
You can assign the PR to them by writing /assign in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-robot
Copy link

New changes are detected. LGTM label has been removed.

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: d5147298-3f07-4279-8ee8-a2a4ddb4f7bf

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@roberthbailey
Copy link
Member

Looks like this failed a lint check.

@aLekSer
Copy link
Collaborator Author

aLekSer commented Oct 1, 2019

Will fix that in a moment.

@aLekSer aLekSer force-pushed the wait-for-fleet-cond-parallel branch from 2871be4 to 30469cc Compare October 1, 2019 08:52
@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: 98914077-3778-4329-bb5e-6f11884c3b40

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

Add a function which would return error instead failing the test.
Add error stream to finish stress test gracefully in case there is no
capacity in a cluster for a specific test.
It is forbidden to call t.Fatalf() from go routine other than main test
routine.
@aLekSer aLekSer force-pushed the wait-for-fleet-cond-parallel branch from 30469cc to 4804c62 Compare October 1, 2019 10:03
@aLekSer
Copy link
Collaborator Author

aLekSer commented Oct 1, 2019

Oops, added but forgot to do git commit --amend

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 8b7b2c8e-d0cb-4c38-b824-dda4be754eeb

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1065/head:pr_1065 && git checkout pr_1065
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.1.0-4804c62

@markmandel markmandel added this to the 1.1.0 milestone Oct 1, 2019
@markmandel markmandel added area/tests Unit tests, e2e tests, anything to make sure things don't break kind/cleanup Refactoring code, fixing up documentation, etc labels Oct 1, 2019
@markmandel markmandel merged commit 72a3ddd into googleforgames:master Oct 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tests Unit tests, e2e tests, anything to make sure things don't break kind/cleanup Refactoring code, fixing up documentation, etc size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

make stress-test-e2e run detects a race condition in test framework.go
5 participants