roachtest: acceptance/version-upgrade failed

[cockroach-teamcity]

(roachtest).acceptance/version-upgrade failed on release-19.2@ce736a5705d3a16e793cd4adc88cc87c15c97c9a:

		  |  1589.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1590.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1591.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1592.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1593.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1594.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1595.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1596.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1597.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1598.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1599.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1600.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1601.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1602.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1603.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1604.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1605.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1606.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1607.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1608.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1609.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1610.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1611.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1612.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1613.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1614.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1615.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1616.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1617.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1618.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1619.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1620.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1621.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1622.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1623.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1624.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1625.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1626.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1627.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1628.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1629.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  |  1630.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[tbg]

Looks like the schema change step timed out

05:54:08 mixed_version_schemachange.go:54: Workload step run: 2
05:54:08 cluster.go:2261: [w13] > ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 10 --concurrency 2 {pgurl:1-4}
06:21:19 test_runner.go:774: [w13] dumped stacks to __stacks
06:21:19 test.go:189: test status: stopping cluster
06:21:19 cluster.go:382: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=11 teamcity-2240609-1599025771-10-n4cpu4:1-4
teamcity-2240609-1599025771-10-n4cpu4: stopping
06:21:20 test.go:337: test failure: 	test_runner.go:801: test timed out (30m0s)
06:21:20 test.go:337: [not the first failure] test failure: 
	cluster.go:2197,mixed_version_schemachange.go:67,versionupgrade.go:190,versionupgrade.go:178,acceptance.go:54,acceptance.go:91,test_runner.go:754: output in run_055408.214_n3_workload_run_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2240609-1599025771-10-n4cpu4:3 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 10 --concurrency 2 {pgurl:1-4} returned: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE

Its logs are not helpful: https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_WorkloadNightly/2240609:id/acceptance/version-upgrade/run_1/run_055408.214_n3_workload_run_schemachange.log

[andreimatei]

release-19.2 and old report. closing