Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nightly testing flakyness #190

Closed
alexander-fenster opened this issue May 5, 2018 · 14 comments · Fixed by #193
Closed

nightly testing flakyness #190

alexander-fenster opened this issue May 5, 2018 · 14 comments · Fixed by #193
Assignees
@callmehiphop
Labels
api: spanner Issues related to the googleapis/nodejs-spanner API. nightly-failure 🚨 This issue needs some love. triage me I really want to be triaged.

Comments

@alexander-fenster
Copy link
Contributor

alexander-fenster commented May 5, 2018 •
edited
Loading

We haven't had a clean nightly run for spanner for a long time. The fact that we kind of ignore test failures is a sign that something is going wrong. If tests are bad they should be fixed or at least marked to skip, otherwise there is no need to run them at all.

I suggest that we spend some time the following week on our engineering debt and make sure we have several green nightly runs.

Red page that makes me sad: https://circleci.com/gh/googleapis/workflows/nodejs-spanner/tree/master

cc: @crwilcox
@alexander-fenster alexander-fenster added priority: p0 Highest priority. Critical issue. P0 implies highest priority. nightly-failure labels May 5, 2018
@stephenplusplus
Copy link
Contributor

@callmehiphop did we have a theory for why sample tests fail so frequently? I recall that we did some investigating, but I can't remember if we ever figured anything out?

@alexander-fenster
Copy link
Contributor Author

My only theory for now is that the 'instance does not exist' error may be caused by system and samples tests running at the same time if the cleanup in any test is implemented incorrectly. I cannot prove it though.

@callmehiphop
Copy link
Contributor

Truthfully I have no idea! I remember we caught a bug with system test failures, but I don't recall ever looking into the sample tests.

@stephenplusplus stephenplusplus mentioned this issue May 10, 2018
Merged
@stephenplusplus
Copy link
Contributor

I think we're good: #193

@ghost ghost removed the priority: p0 Highest priority. Critical issue. P0 implies highest priority. label May 10, 2018
@stephenplusplus
Copy link
Contributor

After #193, the sample tests seem to be fixed, but there's still an issue with system tests: https://circleci.com/gh/googleapis/nodejs-spanner/4195

@stephenplusplus stephenplusplus added the priority: p0 Highest priority. Critical issue. P0 implies highest priority. label May 11, 2018
@stephenplusplus
Copy link
Contributor

Here's a potential fix that I originally tried last year for the same CI failures: #16

@stephenplusplus stephenplusplus mentioned this issue May 11, 2018
Merged
@JustinBeckwith JustinBeckwith added priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. priority: p0 Highest priority. Critical issue. P0 implies highest priority. labels May 31, 2018
@JustinBeckwith
Copy link
Contributor

nudges @stephenplusplus is this still on your radar? It looks like it's been clean for a while. Can we close this out?

@stephenplusplus
Copy link
Contributor

Just need some Spanner-y input on this PR: #16

@JustinBeckwith JustinBeckwith added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. labels Jun 13, 2018
@stephenplusplus stephenplusplus mentioned this issue Jul 5, 2018
Closed
@JustinBeckwith JustinBeckwith mentioned this issue Sep 14, 2018
Closed
@JustinBeckwith
Copy link
Contributor

The current problem with these tests appears to be a replaceProjectIdToken issue:
https://circleci.com/gh/googleapis/nodejs-spanner/7462

@stephenplusplus if I recall, you had previously fixed a few of these elsewhere right?

@stephenplusplus
Copy link
Contributor

Does Circle have a GCLOUD_PROJECT env var set to long-door-651?

@JustinBeckwith
Copy link
Contributor

It looks like they don't. @kinwa91 I thought this was a change you made all up in the Circle CI config?

@jkwlui jkwlui mentioned this issue Sep 17, 2018
Closed
@ghost ghost assigned jkwlui Sep 17, 2018
@jkwlui
Copy link
Member

jkwlui commented Sep 17, 2018

After fixing the GCLOUD_PROJECT env var, we still have a system-test failure:

https://circleci.com/gh/googleapis/nodejs-spanner/7498

@jkwlui
Copy link
Member

jkwlui commented Sep 17, 2018

screen shot 2018-09-17 at 12 28 20 pm

@jkwlui
Copy link
Member

jkwlui commented Sep 17, 2018

cc @stephenplusplus

@JustinBeckwith JustinBeckwith added priority: p0 Highest priority. Critical issue. P0 implies highest priority. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels Sep 19, 2018
@ghost ghost removed the priority: p0 Highest priority. Critical issue. P0 implies highest priority. label Sep 27, 2018
@callmehiphop callmehiphop mentioned this issue Oct 2, 2018
Merged
3 tasks
@google-cloud-label-sync google-cloud-label-sync bot added the api: spanner Issues related to the googleapis/nodejs-spanner API. label Jan 31, 2020
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Apr 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@callmehiphop callmehiphop

Labels
api: spanner Issues related to the googleapis/nodejs-spanner API. nightly-failure 🚨 This issue needs some love. triage me I really want to be triaged.
Projects
None yet
Milestone
No milestone
7 participants
@JustinBeckwith @crwilcox @stephenplusplus @callmehiphop @jkwlui @alexander-fenster @yoshi-automation