roachtest: acceptance/version-upgrade failed #44732

cockroach-teamcity · 2020-02-05T00:00:07Z

(roachtest).acceptance/version-upgrade failed on master@bc31cf16d80dd1820488479666261586a22170e6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	upgrade.go:296,upgrade.go:333,upgrade.go:618,acceptance.go:84,test_runner.go:741: net/http: request canceled (Client.Timeout exceeded while reading body)

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-02-15T08:05:53Z

(roachtest).acceptance/version-upgrade failed on master@5c37cf2d12bafd4a01d4ac7f1725a0ae6e498c9c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	upgrade.go:595,upgrade.go:619,acceptance.go:84,test_runner.go:741: pq: value differs between gossip (2.1) and KV (2.0); try again later (<nil> after 1m58.725632525s)

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-17T16:11:04Z

(roachtest).acceptance/version-upgrade failed on master@1674c8aa4d955738365087d33ff707525e8a2b94:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: dead
		2: 32349
		Error:  1: dead, 3: dead

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-02T18:47:21Z

(roachtest).acceptance/version-upgrade failed on master@003fa437fcce1038085bcc314aa3be61084bbbbe:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	upgrade.go:595,upgrade.go:619,acceptance.go:84,test_runner.go:741: dial tcp 127.0.0.1:26257: connect: connection refused

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 30148
		2: 30261
		Error:  1: dead

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2020-03-03T11:53:50Z

^- v1.0.6 crashed with a consistency failure. We should change this test to not run versions that have reached end-of-life.

cockroach-teamcity · 2020-03-13T07:25:39Z

(roachtest).acceptance/version-upgrade failed on master@69dc87d68addedf2fabfb2b14c098cfb35b5f3d0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200313-1806460/acceptance/version-upgrade/run_1
	upgrade.go:540,upgrade.go:621,acceptance.go:84,test_runner.go:753: expected JSONB to succeed on cluster version 19.2, got pq: schema change cannot be initiated in this version until the version upgrade is finalized

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-19T15:12:37Z

(roachtest).acceptance/version-upgrade failed on master@49bf810c1ad531ddc75c6a9ce60e81b7dd810726:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	upgrade.go:595,upgrade.go:619,acceptance.go:84,test_runner.go:753: pq: value differs between gossip (2.1) and KV (2.0); try again later (<nil> after 1m59.515662779s)

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-29T03:15:29Z

(roachtest).acceptance/version-upgrade failed on master@92495ffb2182671b3338a8c6cd2734cb6de5d392:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/version-upgrade/run_1
	upgrade.go:540,upgrade.go:621,acceptance.go:84,test_runner.go:753: expected JSONB to succeed on cluster version 19.2, got pq: schema change cannot be initiated in this version until the version upgrade is finalized

More

Artifacts: /acceptance/version-upgrade

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2020-03-29T06:19:08Z

Cc @lucy-zhang

thoszhang · 2020-04-01T00:53:54Z

I'm not sure what the best way is to fix this. (For context, we're banning almost all schema changes (e.g., CREATE TABLE is allowed; DROP TABLE is not allowed) from running on pre-20.1 cluster versions, as part of dealing with migrating to using the new schema change job.) We could take advantage of the fact that CREATE TABLE is allowed, to just create tables with the features being tested without dropping them, but that seems hacky. What is the goal of this acceptance test?

tbg · 2020-04-01T07:41:04Z

The cluster has fully migrated to 19.2 when the error occurs. I think the check is probably just using version.IsActive which is updated asynchronously. These kinds of errors are awkward to fix, we expect them to go away in 20.2 as we implement #39182. For now, I would suggest adding code to that particular test that retries for up to 30s (that should be plenty - Gossip propagation is async but is reactive, so usually we're talking ~latency) when it gets this error.

thoszhang · 2020-04-01T15:54:17Z

The cluster has fully migrated to 19.2 when the error occurs.

So the cluster version is 19.2, but we're running a 20.1 binary, right? I think the issue is that the schema change can't proceed because the version gate present in the 20.1 binary is for 20.1.

tbg · 2020-04-01T16:08:49Z

I think I understand what the problem is. The test runs these steps

cockroach/pkg/cmd/roachtest/upgrade.go

Lines 466 to 477 in 8e4d7b2

    
           binaryVersionUpgrade("v19.1.5", nodes), 
        
           clusterVersionUpgrade(""), 
        
           binaryVersionUpgrade("v19.2.1", nodes), 
        
           clusterVersionUpgrade(""), 
        
           // Each new release has to be added here. When adding a new release, you'll 
        
           // probably need to use a release candidate binary. 
        
           // HEAD gives us the main binary for this roachtest run. 
        
           binaryVersionUpgrade("HEAD", nodes), 
        
           clusterVersionUpgrade(""),

and between the steps it "tests the features":

cockroach/pkg/cmd/roachtest/upgrade.go

Lines 514 to 544 in 8e4d7b2

    
           testFeature := func(f feature) { 
        
           	db := c.Conn(ctx, 1) 
        
           	defer db.Close() 
        
           	var cv string 
        
           	if err := db.QueryRowContext(ctx, `SHOW CLUSTER SETTING version`).Scan(&cv); err != nil { 
        
           		t.Fatal(err) 
        
           	} 
        
           	minAllowedVersion, err := roachpb.ParseVersion(f.minAllowedVersion) 
        
           	if err != nil { 
        
           		t.Fatal(err) 
        
           	} 
        
           	actualVersion, err := roachpb.ParseVersion(cv) 
        
           	if err != nil { 
        
           		t.Fatal(err) 
        
           	} 
        
           	_, err = db.Exec(f.query) 
        
           	if actualVersion.Less(minAllowedVersion) { 
        
           		if err == nil { 
        
           			t.Fatalf("expected %s to fail on cluster version %s", f.name, cv) 
        
           		} 
        
           		t.l.Printf("%s: %s fails expected\n", cv, f.name) 
        
           	} else { 
        
           		if err != nil { 
        
           			t.Fatalf("expected %s to succeed on cluster version %s, got %s", f.name, cv, err) 
        
           		} 
        
           		t.l.Printf("%s: %s works as expected\n", cv, f.name) 
        
           	} 
        
           }

When we upgrade the binaries, we rely on auto-upgrades, that is, the nodes automatically bump to the latest version. If this hasn't happened yet by the time we run DDL after starting the cluster into 20.1, we see the error here.

This test hasn't aged very well. I think originally it was intentionally targeting these steps at the mixed clusters, but thanks to the auto-upgrade, it mostly hits non-mixed clusters (though if it hits one now, it will fail with the issue we're discussing).

I have to touch this test anyway. Will take a look at making this work better.

tbg · 2020-04-02T13:33:56Z

Hmm I looked at the test now (and mostly rewrote it) but I am back to square one. We check that the auto-update completed and is visible on all nodes. This means we should never run into the version gate.
I think it's still better if I merge my rewrite first and then we see if the problem persists.

This test used to start a v1.0 cluster and migrated it all the way to the the master branch. As a result, the test would take longer to run with each release, but additionally it would sometimes flake in ancient releases that we don't support any more. Instead of just ditching the old releases altogether, we now start the test up from store directories that were initiated at v1.0 and were upgraded all the way through the lowest version that we care to run in this test. This is different from bootstrapping a new cluster at that version since older versions may have left old data around, or never gotten rid of idiosyncracies that we have since fixed. Touches cockroachdb#44732. Release note: None

tbg · 2020-04-02T14:01:56Z

Heh, no. My previous comment was wrong. We definitely do this thing and it will continue to flake. Once #46924 is merged, I know how to fix it.

We weren't checking whether n4 had actually upgraded. Fixes the failure cockroachdb#44732 (comment) (but there are others, so not closing). Release note: None

tbg · 2020-04-08T15:45:27Z

^- this last failure fixed in https://github.com/cockroachdb/cockroach/pull/47201/files (problem in the test)

47164: kvserver,roachtest/hotspotsplits: adjust constants to ensure backpressure r=nvanbenschoten a=ajwerner See individual commits. Fixes #46957. 47201: roachtest: fix one-off in version-upgrade r=nvanbenschoten,andreimatei a=tbg We weren't checking whether n4 had actually upgraded. Fixes the failure #44732 (comment) (but there are others, so not closing). Release note: None Co-authored-by: Andrew Werner <[email protected]> Co-authored-by: Tobias Schottdorf <[email protected]>

tbg · 2020-04-09T11:46:21Z

Turns out there was more than a one-off, so keeping this open also for #47235 (comment).

The recent changes to this test seem to have uncovered a latent [bug], as a result of which the test was skipped. This commit lets the test start from v19.2 fixtures (instead of v19.1 before) which experimentally were verified to not exhibit the bug, so the test is unskipped. While I was there, I automated the base version discovery of this test. On a branch at version X, the test will now use the fixtures for the last major release preceding X. We'll occasionally have to bake new fixtures and update the mapping in PredecessorVersion() but this is better than having the test rot on older branches whenever we update it for a new dev cycle. The root cause of the bug will be fixed separately; this is tracked in issue cockroachdb#44732. [bug]: cockroachdb#47235 (comment) Release note: None

47232: opt: corrects function doc for canMaybeConstrainIndex r=mgartner a=mgartner #### opt: corrects function doc for canMaybeConstrainIndex This commit corrects a the function documentation for canMaybeConstrainIndex. Prior to this change, it did not mention a third check, for tight filter constraints, that is performed to determinte the possibility that an index could be constrained by a filer. Also, the documentation now correctly maps to the logic of the function. Previously it falsly claimed that if any of the checks were false then the function would return false. Now it correctly states that if any of the checks are true, then the fucntion returns true. Release note: None 47268: roachtest: improve and de-flake version-upgrade r=spaskob a=tbg roachtest: automate and de-flake version-upgrade The recent changes to this test seem to have uncovered a latent [bug], as a result of which the test was skipped. This commit lets the test start from v19.2 fixtures (instead of v19.1 before) which experimentally were verified to not exhibit the bug, so the test is unskipped. While I was there, I automated the base version discovery of this test. On a branch at version X, the test will now use the fixtures for the last major release preceding X. We'll occasionally have to bake new fixtures and update the mapping in PredecessorVersion() but this is better than having the test rot on older branches whenever we update it for a new dev cycle. The root cause of the bug will be fixed separately; this is tracked in issue #44732. [bug]: #47235 (comment) Release note: None Co-authored-by: Marcus Gartner <[email protected]> Co-authored-by: Tobias Schottdorf <[email protected]>

I thought the [bug] would not repro after I bumped the base fixture, but it still does (though maybe one in fifteen only) and I'm running out of work day, so skip the test again. Tomorrow's a latter day. [bug]: cockroachdb#44732 (comment) Release note: None

cockroach-teamcity · 2020-04-17T07:30:22Z

(roachtest).acceptance/version-upgrade failed on master@a63a670b1a591be2ed38af7f98c1f85e32546596:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:1745,versionupgrade.go:225,versionupgrade.go:304,versionupgrade.go:166,versionupgrade.go:155,acceptance.go:55,acceptance.go:91,test_runner.go:753: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1881824-1587108483-14-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned: exit status 1
		(1) attached stack trace
		  | main.(*cluster).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1760
		  | main.(*cluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1744
		  | main.(*versionUpgradeTest).uploadVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:225
		  | main.binaryUpgradeStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:304
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:166
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:155
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:55
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:91
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) cluster.Put
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1881824-1587108483-14-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-1881824-1587108483-14-n4cpu4: putting (dist) /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6
		  | ...
		  |    1: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    3: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  | I200417 07:30:05.762956 1 cluster_synced.go:1178  put /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 failed
		Wraps: (4) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /acceptance/version-upgrade
Related:

roachtest: acceptance/version-upgrade failed #47024 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-18T07:45:19Z

(roachtest).acceptance/version-upgrade failed on master@b5b4a9a55f1122ef9c82b968aa5c8cc137c7e281:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:1766,versionupgrade.go:225,versionupgrade.go:304,versionupgrade.go:166,versionupgrade.go:155,acceptance.go:55,acceptance.go:91,test_runner.go:753: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1883648-1587195731-16-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned: exit status 1
		(1) attached stack trace
		  | main.(*cluster).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1781
		  | main.(*cluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1765
		  | main.(*versionUpgradeTest).uploadVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:225
		  | main.binaryUpgradeStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:304
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:166
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:155
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:55
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:91
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) cluster.Put
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1883648-1587195731-16-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-1883648-1587195731-16-n4cpu4: putting (dist) /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6
		  | ...
		  |    1: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    3: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  | I200418 07:44:57.697653 1 cluster_synced.go:1178  put /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 failed
		Wraps: (4) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /acceptance/version-upgrade
Related:

roachtest: acceptance/version-upgrade failed #47654 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: acceptance/version-upgrade failed #47632 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-provisional_202004171855_v20.1.0-rc.2 release-blocker
roachtest: acceptance/version-upgrade failed #47024 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-19T07:53:41Z

(roachtest).acceptance/version-upgrade failed on master@ce5d511368b741555344ea28efa40fb9facfa577:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:1766,versionupgrade.go:225,versionupgrade.go:304,versionupgrade.go:166,versionupgrade.go:155,acceptance.go:55,acceptance.go:91,test_runner.go:753: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1884693-1587282644-16-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned: exit status 1
		(1) attached stack trace
		  | main.(*cluster).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1781
		  | main.(*cluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1765
		  | main.(*versionUpgradeTest).uploadVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:225
		  | main.binaryUpgradeStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:304
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:166
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:155
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:55
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:91
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) cluster.Put
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1884693-1587282644-16-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-1884693-1587282644-16-n4cpu4: putting (dist) /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6
		  | ...
		  |    1: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    3: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  | I200419 07:53:19.488939 1 cluster_synced.go:1178  put /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 failed
		Wraps: (4) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /acceptance/version-upgrade
Related:

roachtest: acceptance/version-upgrade failed #47654 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: acceptance/version-upgrade failed #47632 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-provisional_202004171855_v20.1.0-rc.2 release-blocker
roachtest: acceptance/version-upgrade failed #47024 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-20T07:38:31Z

(roachtest).acceptance/version-upgrade failed on master@666a0ac62832b6884bc6b039b4c944dbb42924aa:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:1766,versionupgrade.go:225,versionupgrade.go:304,versionupgrade.go:166,versionupgrade.go:155,acceptance.go:55,acceptance.go:91,test_runner.go:753: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1885863-1587368152-08-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned: exit status 1
		(1) attached stack trace
		  | main.(*cluster).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1781
		  | main.(*cluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1765
		  | main.(*versionUpgradeTest).uploadVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:225
		  | main.binaryUpgradeStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:304
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:166
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:155
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:55
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:91
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) cluster.Put
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1885863-1587368152-08-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-1885863-1587368152-08-n4cpu4: putting (dist) /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6
		  | ...
		  |    1: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    3: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  | I200420 07:38:12.083039 1 cluster_synced.go:1178  put /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 failed
		Wraps: (4) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /acceptance/version-upgrade
Related:

roachtest: acceptance/version-upgrade failed #47654 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: acceptance/version-upgrade failed #47632 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-provisional_202004171855_v20.1.0-rc.2 release-blocker
roachtest: acceptance/version-upgrade failed #47024 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-04-21T07:33:14Z

(roachtest).acceptance/version-upgrade failed on master@5ad50d887ad95e6b17b0d20fa98bc30319a67c0d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/version-upgrade/run_1
	cluster.go:1766,versionupgrade.go:225,versionupgrade.go:304,versionupgrade.go:166,versionupgrade.go:155,acceptance.go:55,acceptance.go:91,test_runner.go:753: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1888205-1587454059-16-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned: exit status 1
		(1) attached stack trace
		  | main.(*cluster).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1781
		  | main.(*cluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1765
		  | main.(*versionUpgradeTest).uploadVersion
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:225
		  | main.binaryUpgradeStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:304
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:166
		  | main.runVersionUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:155
		  | main.registerAcceptance.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:55
		  | main.registerAcceptance.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/acceptance.go:91
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) cluster.Put
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1888205-1587454059-16-n4cpu4:1-4 /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6 returned
		  | stderr:
		  |
		  | stdout:
		  | teamcity-1888205-1587454059-16-n4cpu4: putting (dist) /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 ./cockroach-19.2.6
		  | ...
		  |    1: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    3: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  |    4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 [email protected]:./cockroach-19.2.6
		  | scp: ./cockroach-19.2.6: Text file busy
		  | : exit status 1
		  | I200421 07:32:52.252473 1 cluster_synced.go:1178  put /home/agent/temp/buildTmp/cockroach-v19.2.6.linux-amd64 failed
		Wraps: (4) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /acceptance/version-upgrade
Related:

roachtest: acceptance/version-upgrade failed #47654 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: acceptance/version-upgrade failed #47632 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-provisional_202004171855_v20.1.0-rc.2 release-blocker
roachtest: acceptance/version-upgrade failed #47024 roachtest: acceptance/version-upgrade failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

47690: roachtest: deflake acceptance/version-upgrade r=spaskob a=tbg Remove a workaround that wasn't necessary any more (since I regenerated the fixtures last week) but which caused flakes of its own because it was re-uploading the binary for predecessorVersion while processes were running using that binary (resulting in occasional 'text file busy' on linux). Closes #44732. Touches #47024. Release note: None Co-authored-by: Tobias Schottdorf <[email protected]>

cockroach-teamcity added this to the 20.1 milestone Feb 5, 2020

cockroach-teamcity assigned andreimatei Feb 5, 2020

tbg removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 11, 2020

nvanbenschoten mentioned this issue Mar 27, 2020

kv/kvserver: mv testdata/lock_table/add_discovered out of pkg/storage #46631

Merged

tbg assigned tbg and thoszhang and unassigned andreimatei Mar 30, 2020

cockroach-teamcity mentioned this issue Apr 1, 2020

roachtest: acceptance/version-upgrade failed #46825

Closed

This comment has been minimized.

Sign in to view

nvanbenschoten unassigned thoszhang Apr 7, 2020

tbg assigned asubiotto and unassigned tbg Apr 8, 2020

tbg added a commit to tbg/cockroach that referenced this issue Apr 8, 2020

roachtest: fix one-off in version-upgrade

66e25c8

We weren't checking whether n4 had actually upgraded. Fixes the failure cockroachdb#44732 (comment) (but there are others, so not closing). Release note: None

tbg mentioned this issue Apr 8, 2020

roachtest: fix one-off in version-upgrade #47201

Merged

tbg mentioned this issue Apr 9, 2020

roachtest: improve and de-flake version-upgrade #47268

Merged

This was referenced Apr 17, 2020

roachtest: acceptance/version-upgrade failed #47632

Closed

roachtest: acceptance/version-upgrade failed #47654

Closed

tbg mentioned this issue Apr 20, 2020

roachtest: deflake acceptance/version-upgrade #47690

Merged

craig bot closed this as completed in a218953 Apr 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: acceptance/version-upgrade failed #44732

roachtest: acceptance/version-upgrade failed #44732

cockroach-teamcity commented Feb 5, 2020

cockroach-teamcity commented Feb 15, 2020

cockroach-teamcity commented Feb 17, 2020

cockroach-teamcity commented Mar 2, 2020

tbg commented Mar 3, 2020

cockroach-teamcity commented Mar 13, 2020

cockroach-teamcity commented Mar 19, 2020

cockroach-teamcity commented Mar 29, 2020

tbg commented Mar 29, 2020

thoszhang commented Apr 1, 2020

tbg commented Apr 1, 2020

thoszhang commented Apr 1, 2020

tbg commented Apr 1, 2020

tbg commented Apr 2, 2020

tbg commented Apr 2, 2020

This comment has been minimized.

tbg commented Apr 8, 2020

tbg commented Apr 9, 2020

cockroach-teamcity commented Apr 17, 2020

cockroach-teamcity commented Apr 18, 2020

cockroach-teamcity commented Apr 19, 2020

cockroach-teamcity commented Apr 20, 2020

cockroach-teamcity commented Apr 21, 2020

roachtest: acceptance/version-upgrade failed #44732

roachtest: acceptance/version-upgrade failed #44732

Comments

cockroach-teamcity commented Feb 5, 2020

cockroach-teamcity commented Feb 15, 2020

cockroach-teamcity commented Feb 17, 2020

cockroach-teamcity commented Mar 2, 2020

tbg commented Mar 3, 2020

cockroach-teamcity commented Mar 13, 2020

cockroach-teamcity commented Mar 19, 2020

cockroach-teamcity commented Mar 29, 2020

tbg commented Mar 29, 2020

thoszhang commented Apr 1, 2020

tbg commented Apr 1, 2020

thoszhang commented Apr 1, 2020

tbg commented Apr 1, 2020

tbg commented Apr 2, 2020

tbg commented Apr 2, 2020

This comment has been minimized.

tbg commented Apr 8, 2020

tbg commented Apr 9, 2020

cockroach-teamcity commented Apr 17, 2020

cockroach-teamcity commented Apr 18, 2020

cockroach-teamcity commented Apr 19, 2020

cockroach-teamcity commented Apr 20, 2020

cockroach-teamcity commented Apr 21, 2020