Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test: fleetautoscaler_test.go #2296

Closed
roberthbailey opened this issue Oct 6, 2021 · 3 comments
Closed

Flaky test: fleetautoscaler_test.go #2296

roberthbailey opened this issue Oct 6, 2021 · 3 comments
Labels
good first issue These are great first issues. If you are looking for a place to start, start here! help wanted We would love help on these issues. Please come help us! kind/bug These are bugs.
Milestone

Comments

@roberthbailey
Copy link
Member

A spelling fix for the website failed e2e tests

--- FAIL: TestAutoscalerBasicFunctions (350.90s)
    fleetautoscaler_test.go:88: error waiting for fleet condition on fleet simple-fleet-5fxr4
FAIL

Build log

@roberthbailey roberthbailey added kind/bug These are bugs. good first issue These are great first issues. If you are looking for a place to start, start here! help wanted We would love help on these issues. Please come help us! labels Oct 6, 2021
@markmandel markmandel changed the title Faky test: fleetautoscaler_test.go Flaky test: fleetautoscaler_test.go Oct 6, 2021
@roberthbailey
Copy link
Member Author

Another instance (from a different PR).

markmandel added a commit to markmandel/agones that referenced this issue Oct 8, 2021
I noticed in a few flaky end-to-end tests, I kept seeing this in the
logs:

```
time="2021-10-08 19:02:01.419" level=info msg="sent UDP packet" address="35.247.94.25:7682" test=TestGameServerUnhealthyAfterReadyCrash
```

Over and over again, and also noting that it was happening _after_ the
e2e test had completed.

See:
https://console.cloud.google.com/cloud-build/builds/9ca5715a-443c-4693-bbd5-2879e61f2aaa;step=21?project=agones-images
https://console.cloud.google.com/cloud-build/builds/84cb8db2-1a11-4db3-a9e1-d9d51b9baf14;step=21?project=agones-images

My theory: The go routine had nothing in it that forced it to stop once
the test was complete - so depending on order of tests, it might keep
continuing for a while, while other tests ran.

If it did that, and a GameServer spun up on the same node and port as
the originally crashed GameServer, it would crash it - likely breaking
whatever test it ran into!

Work on googleforgames#2296
markmandel added a commit to markmandel/agones that referenced this issue Oct 8, 2021
Added the test name to the e2e framework logging around Fleets, so it
should hopefully make things a bit easier to debug flaky tests.

Work on googleforgames#2296
roberthbailey pushed a commit that referenced this issue Oct 8, 2021
Added the test name to the e2e framework logging around Fleets, so it
should hopefully make things a bit easier to debug flaky tests.

Work on #2296
roberthbailey added a commit that referenced this issue Oct 11, 2021
I noticed in a few flaky end-to-end tests, I kept seeing this in the
logs:

```
time="2021-10-08 19:02:01.419" level=info msg="sent UDP packet" address="35.247.94.25:7682" test=TestGameServerUnhealthyAfterReadyCrash
```

Over and over again, and also noting that it was happening _after_ the
e2e test had completed.

See:
https://console.cloud.google.com/cloud-build/builds/9ca5715a-443c-4693-bbd5-2879e61f2aaa;step=21?project=agones-images
https://console.cloud.google.com/cloud-build/builds/84cb8db2-1a11-4db3-a9e1-d9d51b9baf14;step=21?project=agones-images

My theory: The go routine had nothing in it that forced it to stop once
the test was complete - so depending on order of tests, it might keep
continuing for a while, while other tests ran.

If it did that, and a GameServer spun up on the same node and port as
the originally crashed GameServer, it would crash it - likely breaking
whatever test it ran into!

Work on #2296

Co-authored-by: Robert Bailey <[email protected]>
@markmandel
Copy link
Member

I think we can close this now? Seems like we've fixed it!

@roberthbailey
Copy link
Member Author

Looks like it.

@SaitejaTamma SaitejaTamma added this to the 1.21.0 milestone Feb 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue These are great first issues. If you are looking for a place to start, start here! help wanted We would love help on these issues. Please come help us! kind/bug These are bugs.
Projects
None yet
Development

No branches or pull requests

3 participants