Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: backupTPCC failed #69738

Closed
cockroach-teamcity opened this issue Sep 2, 2021 · 0 comments · Fixed by #69747
Closed

roachtest: backupTPCC failed #69738

cockroach-teamcity opened this issue Sep 2, 2021 · 0 comments · Fixed by #69747
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery

Comments

@cockroach-teamcity
Copy link
Member

roachtest.backupTPCC failed with artifacts on master @ 63679e7bf4a0244c6e46381751543cfe3f32388c:

		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000d04fd8 sp=0xc000d04fd0 pc=0x49d281
		  | created by golang.org/x/sync/errgroup.(*Group).Go
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
		  |
		  | goroutine 222 [runnable]:
		  | golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000c94d20, 0xc000cf5da0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000d057d0 sp=0xc000d057c8 pc=0x10eb400
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000d057d8 sp=0xc000d057d0 pc=0x49d281
		  | created by golang.org/x/sync/errgroup.(*Group).Go
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
		  |
		  | goroutine 223 [runnable]:
		  | golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000c94d20, 0xc000cf5e00)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000d05fd0 sp=0xc000d05fc8 pc=0x10eb400
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000d05fd8 sp=0xc000d05fd0 pc=0x49d281
		  | created by golang.org/x/sync/errgroup.(*Group).Go
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
		  |
		  | goroutine 224 [runnable]:
		  | golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000c94d20, 0xc000cf5e60)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000d067d0 sp=0xc000d067c8 pc=0x10eb400
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000d067d8 sp=0xc000d067d0 pc=0x49d281
		  | created by golang.org/x/sync/errgroup.(*Group).Go
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
		  |
		  | goroutine 225 [runnable]:
		  | golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000c94d20, 0xc000cf5ec0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000d06fd0 sp=0xc000d06fc8 pc=0x10eb400
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000d06fd8 sp=0xc000d06fd0 pc=0x49d281
		  | created by golang.org/x/sync/errgroup.(*Group).Go
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=10 {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Initializing 20 connections...
		  | Initializing 0 idle connections...
		  | Initializing 100 workers and preparing statements...
		Wraps: (8) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 2, 2021
craig bot pushed a commit that referenced this issue Sep 2, 2021
69747: workload: synchronise concurrent access to preparedStatements map r=[otan,tbg,erikgrinaker] a=stevendanna

After 60dd572, the TPCC workload started to fail with

```
fatal error: concurrent map writes
```

because it attempts to start multiple sq runners using the same
MultiConnPool. This fix allows for that by putting a mutex around the
map accesses.

Fixes #69758
Fixes #69757
Fixes #69755
Fixes #69751
Fixes #69750
Fixes #69749
Fixes #69748
Fixes #69745
Fixes #69744
Fixes #69743
Fixes #69742
Fixes #69741
Fixes #69740
Fixes #69739
Fixes #69738
Fixes #69735
Fixes #69734

Release justification: Fixes critical bug in TPCC workload.

Release note: None

Co-authored-by: Steven Danna <[email protected]>
@craig craig bot closed this as completed in cb84f32 Sep 2, 2021
@craig craig bot closed this as completed in #69747 Sep 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant