roachtest: replicate/wide failed #50729

cockroach-teamcity · 2020-06-27T11:40:05Z

(roachtest).replicate/wide failed on master@8f768ad14cfb3f514db6d40465b2dd60ee1f2890:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:804: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

darinpp · 2020-07-07T15:28:07Z

This timed out in the last step - decrease the replication factor to 5 and verify the replicas per range again falls (128e2dc)

irfansharif · 2020-07-13T16:32:51Z

Darin, mind if I assign it to you? Very briefly glancing at the logs, and the area of the codebase I would suspect responsible, it looks to be the allocator.

E200627 11:26:20.984817 161942 kv/kvserver/queue.go:1087  [n1,replicate,s1,r4/1:/System{/tsd-tse}] could not select an appropriate replica to be removed
E200627 11:26:20.989962 161952 kv/kvserver/queue.go:1087  [n1,replicate,s1,r8/14:/Table/1{2-3}] could not select an appropriate replica to be removed
E200627 11:26:21.010495 161989 kv/kvserver/queue.go:1087  [n1,replicate,s1,r7/11:/Table/1{1-2}] could not select an appropriate replica to be removed
E200627 11:26:21.015722 161999 kv/kvserver/queue.go:1087  [n1,replicate,s1,r31/12:/Table/3{5-6}] could not select an appropriate replica to be removed
E200627 11:26:21.020770 162009 kv/kvserver/queue.go:1087  [n1,replicate,s1,r21/6:/Table/2{5-6}] could not select an appropriate replica to be removed

Also consider #50865, which is a bug around the split queue being wedged for some reason. I'm not sure that it has any bearing here, but just to keep on your radar.

cockroach-teamcity · 2020-07-25T12:29:05Z

(roachtest).replicate/wide failed on master@a16eb55ed96239dcd288aa1c2f80f306559f0f0b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-26T11:54:15Z

(roachtest).replicate/wide failed on master@3edbe4aeb3c7300e6690cb2222a8d5c01e920bf4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-19T11:30:24Z

(roachtest).replicate/wide failed on master@ebd5c732f83009acc9c6f5859ca95e74e5453a1c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-21T11:51:36Z

(roachtest).replicate/wide failed on master@bbbedabbf6ea0b1ff6fc799a0c04a75295a9f4c2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-22T11:22:10Z

(roachtest).replicate/wide failed on master@69ffd78d5bbab0d8f77cf1f2254e8a5fcbdf902f:

		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (2) 3 safe details enclosed
		  | Wraps: (3) 8: dead
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errors.errorString
		Wraps: (5) secondary error attachment
		  | 1: dead
		  | (1) attached stack trace
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (2) 3 safe details enclosed
		  | Wraps: (3) 1: dead
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errors.errorString
		Wraps: (6) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (7) 3 safe details enclosed
		Wraps: (8) 9: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *withstack.withStack (7) *safedetails.withSafeDetails (8) *errors.errorString

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-23T11:50:03Z

(roachtest).replicate/wide failed on master@7425e857e62fe4280f614f9076f310322cc78649:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-29T19:04:44Z

(roachtest).replicate/wide failed on master@8b91062f9351d18f9104aff567cb152df162021e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

knz · 2020-08-31T13:37:00Z

Symptom: replication fails to converge after 10mn

irfansharif · 2020-08-31T14:46:05Z

+cc @nvanbenschoten, @tbg for triage/routing.

cockroach-teamcity · 2020-09-02T11:02:51Z

(roachtest).replicate/wide failed on master@57e160b1fcc41dd12b595953729728007fd3fbda:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-13T11:47:52Z

(roachtest).replicate/wide failed on master@38115d0cc366243bcbae1658057cb0438e23565e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-14T11:24:21Z

(roachtest).replicate/wide failed on master@dc5544839735faaa04075e0d9e021ddba721f3bb:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicate/wide/run_1
	test_runner.go:801: test timed out (10m0s)

More

Artifacts: /replicate/wide

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

This used to live in the replicate queue, but there are other entry points to replication changes, notably the store rebalancer which caused cockroachdb#54444. Move the check in the guts of replication changes where it is guaranteed to be invoked. Fixes cockroachdb#50729 Touches cockroachdb#54444 (release-20.2) Release note (bug fix): in rare situations, an automated replication change could result in a loss of quorum. This would require down nodes and a simultaneous change in the replication factor. Note that a change in the replication factor can occur automatically if the cluster is comprised of less than five available nodes. Experimentally the likeli- hood of encountering this issue, even under contrived conditions, was small.

@aayushshah15

56735: kvserverpb: move quorum safeguard into execChangeReplicasTxn r=aayushshah15 a=tbg This used to live in the replicate queue, but there are other entry points to replication changes, notably the store rebalancer which caused #54444. Move the check in the guts of replication changes where it is guaranteed to be invoked. Fixes #50729 Touches #54444 (release-20.2) @aayushshah15 only requesting your review since you're in the area. Feel free to opt out. Release note (bug fix): in rare situations, an automated replication change could result in a loss of quorum. This would require down nodes and a simultaneous change in the replication factor. Note that a change in the replication factor can occur automatically if the cluster is comprised of less than five available nodes. Experimentally the likeli- hood of encountering this issue, even under contrived conditions, was small. Co-authored-by: Tobias Grieger <[email protected]>

cockroach-teamcity added this to the 20.2 milestone Jun 27, 2020

cockroach-teamcity assigned andreimatei Jun 27, 2020

irfansharif assigned darinpp and unassigned andreimatei Jul 13, 2020

irfansharif unassigned darinpp Aug 31, 2020

knz removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 31, 2020

tbg assigned darinpp Sep 1, 2020

nvanbenschoten added the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 2, 2020

tbg assigned tbg and unassigned darinpp Sep 8, 2020

tbg removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 8, 2020

cockroach-teamcity mentioned this issue Sep 16, 2020

roachtest: replicate/wide failed #54444

Closed

tbg mentioned this issue Nov 16, 2020

kvserverpb: move quorum safeguard into execChangeReplicasTxn #56735

Merged

craig bot closed this as completed in 5178559 Dec 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: replicate/wide failed #50729

roachtest: replicate/wide failed #50729

cockroach-teamcity commented Jun 27, 2020

darinpp commented Jul 7, 2020

irfansharif commented Jul 13, 2020

cockroach-teamcity commented Jul 25, 2020

cockroach-teamcity commented Jul 26, 2020

cockroach-teamcity commented Aug 19, 2020

cockroach-teamcity commented Aug 21, 2020

cockroach-teamcity commented Aug 22, 2020

cockroach-teamcity commented Aug 23, 2020

cockroach-teamcity commented Aug 29, 2020

knz commented Aug 31, 2020

irfansharif commented Aug 31, 2020

cockroach-teamcity commented Sep 2, 2020

cockroach-teamcity commented Sep 13, 2020

cockroach-teamcity commented Sep 14, 2020

roachtest: replicate/wide failed #50729

roachtest: replicate/wide failed #50729

Comments

cockroach-teamcity commented Jun 27, 2020

darinpp commented Jul 7, 2020

irfansharif commented Jul 13, 2020

cockroach-teamcity commented Jul 25, 2020

cockroach-teamcity commented Jul 26, 2020

cockroach-teamcity commented Aug 19, 2020

cockroach-teamcity commented Aug 21, 2020

cockroach-teamcity commented Aug 22, 2020

cockroach-teamcity commented Aug 23, 2020

cockroach-teamcity commented Aug 29, 2020

knz commented Aug 31, 2020

irfansharif commented Aug 31, 2020

cockroach-teamcity commented Sep 2, 2020

cockroach-teamcity commented Sep 13, 2020

cockroach-teamcity commented Sep 14, 2020