Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: clearrange/checks=false failed #44329

Closed
cockroach-teamcity opened this issue Jan 24, 2020 · 7 comments
Closed

roachtest: clearrange/checks=false failed #44329

cockroach-teamcity opened this issue Jan 24, 2020 · 7 comments
Assignees
@andreimatei
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).clearrange/checks=false failed on release-19.1@2878820bc2f9ccc5f005c6ef9128ed58282b176f:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200124-1708457/clearrange/checks=false/run_1
	cluster.go:1935,clearrange.go:46,clearrange.go:32,test_runner.go:719: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1969
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runClearRange
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:46
		    main.registerClearRange.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:32
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:719
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1708457-1579849715-24-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned:
		    stderr:
		    
		    stdout:
		    I200124 07:59:38.187808 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		    I200124 09:21:31.011478 15 ccl/workloadccl/fixture.go:516  imported bank (1h21m53s, 0 rows, 0 index entries, 0 B)
		    Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		    Error:  exit status 1:
		  - exit status 1
More

Artifacts: /clearrange/checks=false
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-release-19.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 24, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Jan 24, 2020
@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 3, 2020
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 11, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).clearrange/checks=false failed on release-19.1@cfdc5f37cad427dcc5e36aa630f9d49e588f5cf3:

		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_070556.042_n1_cockroach_workload_fixtures_import_bank:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runClearRange
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:47
		    main.registerClearRange.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:33
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1819536-1584684878-25-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned:
		    stderr:
		    I200320 07:05:56.797405 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		    I200320 08:37:10.461048 15 ccl/workloadccl/fixture.go:516  imported bank (1h31m14s, 0 rows, 0 index entries, 0 B)
		    Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		    Error:  exit status 1
		    
		    stdout::
		  - exit status 1

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1819536-1584684878-25-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: dead
		2: 3931
		6: 4028
		1: 4721
		8: 3978
		9: 3819
		7: 4110
		4: 3764
		10: 4448
		3: 3781
		Error:  5: dead
More

Artifacts: /clearrange/checks=false
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).clearrange/checks=false failed on release-19.1@d2ca82a9f898bd8ef7bd2e9660e24605f2d0b25d:

		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1829279-1585203477-25-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned:
		    stderr:
		    I200326 07:04:51.405275 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		    I200326 08:26:36.946349 23 ccl/workloadccl/fixture.go:516  imported bank (1h21m46s, 0 rows, 0 index entries, 0 B)
		    Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		    Error: DEAD_ROACH_PROBLEM:
		      - error with user detail: Node 1. Command with error:
		        ```
		        ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		        ```
		      - exit status 1
		    
		    stdout::
		  - exit status 30

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1829279-1585203477-25-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 9: dead
		1: 3869
		10: 3740
		8: 3768
		3: 3697
		6: 4182
		2: 3781
		5: 3750
		7: 3729
		4: 3784
		Error: UNCLASSIFIED_PROBLEM:
		  - 9: dead
		    main.glob..func13
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		    main.wrap.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		    main.main
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:203
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: /clearrange/checks=false
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).clearrange/checks=false failed on release-19.1@86e982de240a87f16c896995a25171e95019bf9c:

		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1851691-1585980958-25-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned:
		    stderr:
		    I200404 07:07:43.922691 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		    I200404 08:36:03.627420 28 ccl/workloadccl/fixture.go:516  imported bank (1h28m20s, 0 rows, 0 index entries, 0 B)
		    Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		    Error: DEAD_ROACH_PROBLEM:
		      - error with user detail: Node 1. Command with error:
		        ```
		        ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		        ```
		      - exit status 1
		    
		    stdout::
		  - exit status 30

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1851691-1585980958-25-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: dead
		1: 4562
		4: 4453
		2: 4186
		7: 3726
		8: 4345
		9: 4323
		6: 3704
		5: 4374
		10: 4024
		Error: UNCLASSIFIED_PROBLEM:
		  - 3: dead
		    main.glob..func13
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		    main.wrap.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		    main.main
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:203
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: /clearrange/checks=false
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).clearrange/checks=false failed on release-19.1@c406bb10543ca97010c64cc230a3c45690a7eb6c:

		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1862644-1586413264-26-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned:
		    stderr:
		    I200409 07:18:48.945849 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		    I200409 08:44:49.848837 15 ccl/workloadccl/fixture.go:516  imported bank (1h26m1s, 0 rows, 0 index entries, 0 B)
		    Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		    Error: DEAD_ROACH_PROBLEM:
		      - error with user detail: Node 1. Command with error:
		        ```
		        ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		        ```
		      - exit status 1
		    
		    stdout::
		  - exit status 30

	cluster.go:1420,context.go:135,cluster.go:1409,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1862644-1586413264-26-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 9: dead
		2: 3679
		3: 3945
		1: 4659
		6: 3714
		5: 4189
		8: 4371
		10: 4183
		7: 3683
		4: 4642
		Error: UNCLASSIFIED_PROBLEM:
		  - 9: dead
		    main.glob..func13
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		    main.wrap.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		    main.main
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:203
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: /clearrange/checks=false
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue May 5, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).clearrange/checks=false failed on release-19.1@73a373fb8c138c8ef6e4a05d7c1757207efa0a8d:

		Wraps: (3) output in run_070538.376_n1_cockroach_workload_fixtures_import_bank
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1980374-1590905635-26-n10cpu4:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I200531 07:05:39.158504 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 1 tables
		  | I200531 08:36:50.682239 53 ccl/workloadccl/fixture.go:516  imported bank (1h31m12s, 0 rows, 0 index entries, 0 B)
		  | Error: importing fixture: importing table bank: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: DEAD_ROACH_PROBLEM: exit status 1
		  | (1) DEAD_ROACH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cockroach (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (5) exit status 30
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1980374-1590905635-26-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: dead
		2: 3822
		5: 4148
		3: 3755
		1: 3885
		6: 3774
		9: 3811
		7: 3750
		8: 3773
		4: 3774
		Error: UNCLASSIFIED_PROBLEM: 10: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 10: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString
More

Artifacts: /clearrange/checks=false
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz
Copy link
Contributor

knz commented Jun 4, 2020

"No space left"

F200531 08:36:10.670221 182 storage/replica_raft.go:927  [n10,s10,r20617/1:/Table/53/1/5948{2268-5540}] during sideloading: during sideloading: IO error: No space left on deviceWhile appending to file: /mnt/data1/cockroach/auxiliary/sideloading/r2XXXX/r20617/i22.t7: No space left on device

@knz
Copy link
Contributor

knz commented Jun 4, 2020

dup #44845

@knz knz closed this as completed Jun 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreimatei andreimatei

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
20.1
Development

No branches or pull requests
3 participants
@andreimatei @knz @cockroach-teamcity