Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: release-20.1: double init due to restart of node 1 #51965

Closed
irfansharif opened this issue Jul 28, 2020 · 1 comment · Fixed by #52040
Closed

roachtest: release-20.1: double init due to restart of node 1 #51965

irfansharif opened this issue Jul 28, 2020 · 1 comment · Fixed by #52040
Assignees
@irfansharif
Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@irfansharif
Copy link
Contributor

This is the tracking issue for the set of failures in provisional_202007271721_v20.1.4

#51952
#51953
#51954
#51955
#51956
#51957
#51958
#51960

They all fail with "has already been initialized" errors, and are all attributable to recent changes made to roachprod starting from #51329, where we changed it to explicitly initialize the cluster when starting node 1, when there's no cluster-bootstrapped file found on disk. For all these failing tests we seem to always be starting node 1 mid-test, where upon not finding the cluster-bootstrapped file on disk it expectedly failing to initialize the cluster and thus errors out. The tests that fail in this pattern are "mixed-version" tests, and given we're targeting the release-20.1 branch here, they all start off with nodes running 19.2 (never actually persisting the cluster-bootstrapped file on disk during bootstrap, which only happens when initializing a 20.1+ cluster).
@irfansharif irfansharif added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Jul 28, 2020
@irfansharif irfansharif self-assigned this Jul 28, 2020
@irfansharif irfansharif changed the title roachtest: double init due to restart of node 1 roachtest: release-20.1: double init due to restart of node 1 Jul 28, 2020
@irfansharif
Copy link
Contributor Author

irfansharif commented Jul 28, 2020 •
edited
Loading

Note to self: This will affect release-20.1 roachtests going forward, so close them out as well when addressing this. This is also another reason to have #51897, it's a bit surprising that CI lets you land roach{prod,test} changes without testing how they affect tests against 20.1 and earlier. Blocking #51919.

@ajwerner ajwerner mentioned this issue Jul 28, 2020
Closed
@thoszhang thoszhang mentioned this issue Jul 28, 2020
Closed
This was referenced Jul 28, 2020
Closed
Closed
Closed
Closed
Closed
Closed
irfansharif added a commit to irfansharif/cockroach that referenced this issue Jul 28, 2020
@irfansharif
roachtest: fix release-20.1 roachtests failing due to double-init
4638e8f 
Fixes cockroachdb#51965 (and all referencing issues).

Roachprod clusters running v20.1+ crdb nodes persist this
`cluster-bootstrapped` file on disk after explicitly bootstrapping the
cluster. Roachprod then uses the existence of this file to avoid doubly
bootstrapping the cluster.

Given cockroachdb#51897 remains unresolved, master-built roachprod is used to run
roachtests against the 20.1 branch. Some of those roachtests test
mixed-version clusters that start off at 19.2. Consequently, we manually
add this file where roachprod expects to find it for already-initialized
clusters. (This is a pretty gross hack, that we should address by
addressing cockroachdb#51897.)

Release note: None
@irfansharif irfansharif mentioned this issue Jul 28, 2020
Merged
craig bot pushed a commit that referenced this issue Jul 29, 2020
@irfansharif
Merge #52040
605a7f2 
52040: roachtest: fix release-20.1 roachtests failing due to double-init r=RaduBerinde a=irfansharif

Fixes #51965 (and all referencing issues).

Roachprod clusters running v20.1+ crdb nodes persist this
`cluster-bootstrapped` file on disk after explicitly bootstrapping the
cluster. Roachprod then uses the existence of this file to avoid doubly
bootstrapping the cluster.

Given #51897 remains unresolved, master-built roachprod is used to run
roachtests against the 20.1 branch. Some of those roachtests test
mixed-version clusters that start off at 19.2. Consequently, we manually
add this file where roachprod expects to find it for already-initialized
clusters. (This is a pretty gross hack, that we should address by
addressing #51897.)

Release note: None

Co-authored-by: irfan sharif <[email protected]>
craig bot pushed a commit that referenced this issue Jul 29, 2020
@ajwerner @irfansharif
Merge #52015 #52040
11c5913 
52015: sql: use a structured error to detect roachpb.BatchTimestampBeforeGCE… r=ajwerner a=ajwerner

…rror

Fixes #50198.

See the issue for more details. This issue is so obscure it does not deserve
a release note.

Release note: None

52040: roachtest: fix release-20.1 roachtests failing due to double-init r=irfansharif a=irfansharif

Fixes #51965 (and all referencing issues).

Roachprod clusters running v20.1+ crdb nodes persist this
`cluster-bootstrapped` file on disk after explicitly bootstrapping the
cluster. Roachprod then uses the existence of this file to avoid doubly
bootstrapping the cluster.

Given #51897 remains unresolved, master-built roachprod is used to run
roachtests against the 20.1 branch. Some of those roachtests test
mixed-version clusters that start off at 19.2. Consequently, we manually
add this file where roachprod expects to find it for already-initialized
clusters. (This is a pretty gross hack, that we should address by
addressing #51897.)

Release note: None

Co-authored-by: Andrew Werner <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
@craig craig bot closed this as completed in d1ddaf5 Jul 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachtest: fix release-20.1 roachtests failing due to double-init irfansharif/cockroach
1 participant
@irfansharif