roachtest: kv50/enc=false/nodes=1/conc=8192 failed #84258

cockroach-teamcity · 2022-07-12T10:22:50Z

roachtest.kv50/enc=false/nodes=1/conc=8192 failed with artifacts on release-21.2 @ 97c5b1cbd23c109798597e9873544b91da651802:

		  |    18.0s        0            0.0         2349.8      0.0      0.0      0.0      0.0 write
		  |    19.0s        0           17.9         2518.6  10200.5  10737.4  11274.3  11274.3 read
		  |    19.0s        0          480.2         2251.5  10737.4  11274.3  11811.2  11811.2 write
		  |    20.0s        0         2376.6         2514.5    218.1  10737.4  11274.3  11811.2 read
		  |    20.0s        0         4464.5         2367.8  10737.4  11811.2  11811.2  12348.0 write
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |    21.0s        0         6597.8         2708.1    176.2    402.7  10737.4  12884.9 read
		  |    21.0s        0         7044.0         2589.6    704.6  11811.2  11811.2  12348.0 write
		  |    22.0s        0         4185.0         2773.1    184.5    520.1    771.8  12884.9 read
		  |    22.0s        0         4792.8         2687.3    637.5    939.5   1140.9  11274.3 write
		  |    23.0s        0         6766.5         2947.4    109.1   1073.7   1677.7  15032.4 read
		  |    23.0s        0         5626.9         2815.6    771.8   1409.3   2080.4  12884.9 write
		  |    24.0s        0         4888.9         3028.4    125.8    486.5   1610.6  15569.3 read
		  |    24.0s        0         5506.3         2927.9    671.1   1811.9   2281.7  14495.5 write
		  |    25.0s        0         4151.5         3073.8    109.1    637.5   1342.2  16643.0 read
		  |    25.0s        0         5362.8         3026.0   1073.7   1476.4   2281.7  15569.3 write
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,kv.go:137,kv.go:251,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:137
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:251
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-17554}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-07-15T10:01:37Z

roachtest.kv50/enc=false/nodes=1/conc=8192 failed with artifacts on release-21.2 @ 28873f5636faa3e7a1f00c03f3a6d28f9cb19cda:

		  |    15.0s        0           35.1         2676.2   6174.0   6710.9   7247.8   7247.8 write
		  |    16.0s        0           31.0         2775.5   6710.9   6710.9   6710.9   6710.9 read
		  |    16.0s        0            1.0         2509.0   8589.9   8589.9   8589.9   8589.9 write
		  |    17.0s        0         1149.1         2680.1   7516.2   8053.1   8321.5   9663.7 read
		  |    17.0s        0          538.3         2393.2   8589.9   9663.7  10200.5  10200.5 write
		  |    18.0s        0            0.0         2531.2      0.0      0.0      0.0      0.0 read
		  |    18.0s        0            0.0         2260.2      0.0      0.0      0.0      0.0 write
		  |    19.0s        0         1516.3         2479.0     96.5  10200.5  10737.4  11274.3 read
		  |    19.0s        0         2971.5         2300.0  10200.5  11274.3  11811.2  11811.2 write
		  |    20.0s        0         3952.1         2553.6    134.2   2281.7  10737.4  11811.2 read
		  |    20.0s        0         4238.8         2398.0   9663.7  11274.3  11274.3  12348.0 write
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |    21.0s        0         6880.9         2763.3    100.7    570.4   1275.1  12348.0 read
		  |    21.0s        0         7248.8         2632.8    771.8   1543.5  11274.3  12348.0 write
		  |    22.0s        0         4704.9         2848.4    209.7    671.1   4563.4  13421.8 read
		  |    22.0s        0         4503.9         2714.8    604.0   1040.2   1610.6  12884.9 write
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,kv.go:137,kv.go:251,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:137
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:251
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

nvanbenschoten · 2022-07-20T22:00:47Z

This most recent test failed with:

Error: ERROR: operation "storage.pendingLeaseRequest: requesting lease" timed out after 6s (given timeout 6s): aborted during Replica.Send: context deadline exceeded (SQLSTATE XXUUU)

This is fallout from #81816. That PR already made it into v21.2.13, so there's no point in treating this as a release blocker so I'll remove the label.

However, I think this is a surprising error to see all the way up in the client. @erikgrinaker did we consider returning a NotLeaseHolderError back to the client in these cases so that the DistSender would retry on another node, instead of returning the context deadline exceeded error which will propagate back to a client?

erikgrinaker · 2022-07-20T22:08:02Z

Oof, yeah -- I seem to recall there was some context error handling involved there, but don't remember the specifics. Seems reasonable to have the DistSender retry these on a different replica. I'll have a closer look tomorrow.

cockroach-teamcity added branch-release-21.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jul 12, 2022

cockroach-teamcity added this to the 21.2 milestone Jul 12, 2022

blathers-crl bot added the T-kv KV Team label Jul 12, 2022

nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jul 20, 2022

erikgrinaker self-assigned this Jul 20, 2022

cockroach-teamcity mentioned this issue Jul 21, 2022

roachtest: kv50/enc=false/nodes=1/conc=8192 failed #84818

Closed

This was referenced Jul 21, 2022

kvserver: always return NLHE on lease acquisition timeouts #84865

Merged

kvserver: lease acquisition timeouts mask NotLeaseHolderError #85115

Closed

craig bot closed this as completed in 314baa5 Aug 1, 2022

This was referenced Aug 1, 2022

release-22.1: kvserver: always return NLHE on lease acquisition timeouts #85428

Merged

release-21.2: kvserver: always return NLHE on lease acquisition timeouts #85429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: kv50/enc=false/nodes=1/conc=8192 failed #84258

roachtest: kv50/enc=false/nodes=1/conc=8192 failed #84258

cockroach-teamcity commented Jul 12, 2022 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jul 15, 2022

nvanbenschoten commented Jul 20, 2022

erikgrinaker commented Jul 20, 2022

roachtest: kv50/enc=false/nodes=1/conc=8192 failed #84258

roachtest: kv50/enc=false/nodes=1/conc=8192 failed #84258

Comments

cockroach-teamcity commented Jul 12, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jul 15, 2022

nvanbenschoten commented Jul 20, 2022

erikgrinaker commented Jul 20, 2022

cockroach-teamcity commented Jul 12, 2022 •

edited by cockroach-jira-scripts

Loading