Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: cdc/initial-scan failed #69742

Closed
cockroach-teamcity opened this issue Sep 2, 2021 · 0 comments · Fixed by #69747
Closed

roachtest: cdc/initial-scan failed #69742

cockroach-teamcity opened this issue Sep 2, 2021 · 0 comments · Fixed by #69747
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc

Comments

@cockroach-teamcity
Copy link
Member

roachtest.cdc/initial-scan failed with artifacts on master @ 63679e7bf4a0244c6e46381751543cfe3f32388c:

		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc001aaffd8 sp=0xc001aaffd0 pc=0x49d281
		  | created by golang.org/x/sync/errgroup.(*Group).Go
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=100 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Initializing 200 connections...
		  | Initializing 0 idle connections...
		  | Initializing 1000 workers and preparing statements...
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,cdc.go:293,cdc.go:715,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:293
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:715
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6309
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:208
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 2, 2021
@blathers-crl blathers-crl bot added the T-cdc label Sep 2, 2021
craig bot pushed a commit that referenced this issue Sep 2, 2021
69747: workload: synchronise concurrent access to preparedStatements map r=[otan,tbg,erikgrinaker] a=stevendanna

After 60dd572, the TPCC workload started to fail with

```
fatal error: concurrent map writes
```

because it attempts to start multiple sq runners using the same
MultiConnPool. This fix allows for that by putting a mutex around the
map accesses.

Fixes #69758
Fixes #69757
Fixes #69755
Fixes #69751
Fixes #69750
Fixes #69749
Fixes #69748
Fixes #69745
Fixes #69744
Fixes #69743
Fixes #69742
Fixes #69741
Fixes #69740
Fixes #69739
Fixes #69738
Fixes #69735
Fixes #69734

Release justification: Fixes critical bug in TPCC workload.

Release note: None

Co-authored-by: Steven Danna <[email protected]>
@craig craig bot closed this as completed in cb84f32 Sep 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant