roachtest: decommission/mixed-versions failed #55798

cockroach-teamcity · 2020-10-21T06:24:49Z

(roachtest).decommission/mixed-versions failed on master@1d46df77dbd8721cccf508fb5ed498f3de78022c:

		  |   | * - Intruders can log in without password and read or write any data in the cluster.
		  |   | * - Intruders can consume all your server's resources and cause unavailability.
		  |   | *
		  |   | *
		  |   | * INFO: To start a secure server without mandating TLS for clients,
		  |   | * consider --accept-sql-without-tls instead. For other options, see:
		  |   | * 
		  |   | * - https://go.crdb.dev/issue-v/53404/v20.2
		  |   | * - https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html
		  |   | *
		  |   | *
		  |   | * ERROR: ERROR: cockroach server exited with error: store <no-attributes>=/mnt/data1/cockroach, last used with cockroach version v19.1-1, is too old for running version v20.2-1 (which requires data from v20.1 or later)
		  |   | *
		  |   | ERROR: cockroach server exited with error: store <no-attributes>=/mnt/data1/cockroach, last used with cockroach version v19.1-1, is too old for running version v20.2-1 (which requires data from v20.1 or later)
		  |   | Failed running "start"
		  |   | E201021 06:24:31.456015 1 cli/error.go:398  ERROR: exit status 1
		  |   | ERROR: exit status 1
		  |   | Failed running "start"
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I201021 06:24:31.492862 1 (gostd) cluster_synced.go:1733  command failed
		Wraps: (2) exit status 1
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

	cluster.go:1657,context.go:140,cluster.go:1646,test_runner.go:836: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2381197-1603261251-05-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 4730
		2: 4402
		3: 4642
		4: dead
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /decommission/mixed-versions
Related:

roachtest: decommission/mixed-versions failed #54908 roachtest: decommission/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

tbg · 2020-10-27T14:55:09Z

This is expected. ccd12ed needs to be picked up by release-20.1 (and used in roachtest) for this test to pass.

cockroach-teamcity · 2020-11-07T07:14:43Z

(roachtest).decommission/mixed-versions failed on master@61c96aaca632dfba5154d55d82c5af0732053b72:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions
Related:

roachtest: decommission/mixed-versions failed #54908 roachtest: decommission/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

thoszhang · 2020-11-07T16:18:23Z

mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0

Since this is a mixed-version test, the new failure is probably fallout from the version on master now being 21.1.

cockroach-teamcity · 2020-11-08T07:12:15Z

(roachtest).decommission/mixed-versions failed on master@be84ae4a59e6a5e50694c1756e05b0ba7c3a54d8:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions
Related:

roachtest: decommission/mixed-versions failed #54908 roachtest: decommission/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-10T07:12:17Z

(roachtest).decommission/mixed-versions failed on master@f2b53d1080f1ccc503263dd178ee31b4878d2360:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions
Related:

roachtest: decommission/mixed-versions failed #54908 roachtest: decommission/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-10T16:20:20Z

(roachtest).decommission/mixed-versions failed on master@f2b53d1080f1ccc503263dd178ee31b4878d2360:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions
Related:

roachtest: decommission/mixed-versions failed #54908 roachtest: decommission/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-11T07:11:00Z

(roachtest).decommission/mixed-versions failed on master@df21e5065a79d1dd5b0fd76018724dd26660beb2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions
Related:

roachtest: decommission/mixed-versions failed #54908 roachtest: decommission/mixed-versions failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The test was fully decommissioning and then recommissioning nodes, which does no longer work now that the master branch is 21.1 and thus the predecessor version in this test is 20.2, which does know about fully decommissioning (decommission attempts run via 20.1 nodes never set the status to fully decommissioned, so we got away with it so far). Pare down the test so that it carries out one random partial and full decommission cycle alongside a partial version upgrade, which is reasonably all we can do unless we want to get into the business of wiping and re-adding nodes to the cluster in this test, which I don't think we do as it doesn't work well with the step-driven format this test is using. Fixes cockroachdb#55798. Release note: None

cockroach-teamcity · 2020-11-13T07:08:58Z

(roachtest).decommission/mixed-versions failed on master@2d34be73030a0d114b84fc2785672eda072c5a50:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-14T07:10:09Z

(roachtest).decommission/mixed-versions failed on master@6b254f2d4a8e1470c1a7d5bd7c3f746d69de0797:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-15T07:25:37Z

(roachtest).decommission/mixed-versions failed on master@5d8f1d3dbc356573a1fb2d1d6bf564bf0370daff:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-16T07:03:30Z

(roachtest).decommission/mixed-versions failed on master@b4bf42c4fe10155717ac48f2e68ad8b501ca0b1a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/decommission/mixed-versions/run_1
	mixed_version_decommission.go:248,versionupgrade.go:189,mixed_version_decommission.go:122,decommission.go:73,test_runner.go:755: expected to find 1 node with membership=decommissioning, found 0
		(1) attached stack trace
		  -- stack trace:
		  | main.checkOneMembership.func1.1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:237
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | main.checkOneMembership.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:228
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:189
		  | main.runDecommissionMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_decommission.go:122
		  | main.registerDecommission.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/decommission.go:73
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) expected to find 1 node with membership=decommissioning, found 0
		Error types: (1) *withstack.withStack (2) *errutil.leafError

More

Artifacts: /decommission/mixed-versions

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

56568: roachtest: de-flake decommission/mixed-versions r=andreimatei a=tbg The test was fully decommissioning and then recommissioning nodes, which does no longer work now that the master branch is 21.1 and thus the predecessor version in this test is 20.2, which does know about fully decommissioning (decommission attempts run via 20.1 nodes never set the status to fully decommissioned, so we got away with it so far). Pare down the test so that it carries out one random partial and full decommission cycle alongside a partial version upgrade, which is reasonably all we can do unless we want to get into the business of wiping and re-adding nodes to the cluster in this test, which I don't think we do as it doesn't work well with the step-driven format this test is using. Fixes #55798. Release note: None Co-authored-by: Tobias Grieger <[email protected]>

cockroach-teamcity added this to the 20.2 milestone Oct 21, 2020

cockroach-teamcity mentioned this issue Nov 7, 2020

roachtest: decommission/mixed-versions failed #54908

Closed

andreimatei assigned tbg Nov 10, 2020

tbg mentioned this issue Nov 11, 2020

roachtest: de-flake decommission/mixed-versions #56568

Merged

craig bot closed this as completed in a8c0758 Nov 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: decommission/mixed-versions failed #55798

roachtest: decommission/mixed-versions failed #55798

cockroach-teamcity commented Oct 21, 2020

tbg commented Oct 27, 2020 •

edited

Loading

cockroach-teamcity commented Nov 7, 2020

thoszhang commented Nov 7, 2020

cockroach-teamcity commented Nov 8, 2020

cockroach-teamcity commented Nov 10, 2020

cockroach-teamcity commented Nov 10, 2020

cockroach-teamcity commented Nov 11, 2020

cockroach-teamcity commented Nov 13, 2020

cockroach-teamcity commented Nov 14, 2020

cockroach-teamcity commented Nov 15, 2020

cockroach-teamcity commented Nov 16, 2020

roachtest: decommission/mixed-versions failed #55798

roachtest: decommission/mixed-versions failed #55798

Comments

cockroach-teamcity commented Oct 21, 2020

tbg commented Oct 27, 2020 • edited Loading

cockroach-teamcity commented Nov 7, 2020

thoszhang commented Nov 7, 2020

cockroach-teamcity commented Nov 8, 2020

cockroach-teamcity commented Nov 10, 2020

cockroach-teamcity commented Nov 10, 2020

cockroach-teamcity commented Nov 11, 2020

cockroach-teamcity commented Nov 13, 2020

cockroach-teamcity commented Nov 14, 2020

cockroach-teamcity commented Nov 15, 2020

cockroach-teamcity commented Nov 16, 2020

tbg commented Oct 27, 2020 •

edited

Loading