Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: scaledata/filesystem_simulator/nodes=6 failed #43284

Closed
cockroach-teamcity opened this issue Dec 18, 2019 · 32 comments
Closed

roachtest: scaledata/filesystem_simulator/nodes=6 failed #43284

cockroach-teamcity opened this issue Dec 18, 2019 · 32 comments
Assignees
@irfansharif
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on release-2.1@251f373b9d05716d4d12f02f06d3669223774e45:

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191218-1649986/scaledata/filesystem_simulator/nodes=6/run_1
	test_runner.go:715: test timed out (20m0s)
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6

make stressrace TESTS=scaledata/filesystem_simulator/nodes=6 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Dec 18, 2019
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Dec 18, 2019
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on release-19.2@a84266e50207e2e9a27828991ec9fc2f62d74a88:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:700
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1656791-1576827721-69-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.2:26257,10.128.1.1:26257,10.128.0.75:26257,10.128.1.3:26257,10.128.1.0:26257,10.128.0.137:26257'  returned:
		    stderr:
		    42 Deleted stripes for uuid b85e9e57-a2d7-45fd-bf00-674b581ae975
		    2019/12/20 12:34:42 RobustDB.RandomDB chose DB at index 1
		    2019/12/20 12:34:42 Created file 8_2452 with uuid aad7a306-9299-40e6-8459-3ced4d27eb7e and parent /default
		    2019/12/20 12:34:42 Deleted child_relations for uuid 824a4dba-84a0-49ac-9bfd-f57ac5d505ae
		    2019/12/20 12:34:42 RobustDB.RandomDB chose DB at index 4
		    2019/12/20 12:34:42 Deleted child_relations for uuid cfe5e71f-ae5b-4bf8-9098-97019d51bbec
		    2019/12/20 12:34:42 Deleted child_relations for uuid b85e9e57-a2d7-45fd-bf00-674b581ae975
		    2019/12/20 12:34:42 Deleted &{824a4dba-84a0-49ac-9bfd-f57ac5d505ae 1 2 189 default}
		    2019/12/20 12:34:42 ExecuteTx retry attempt 1 failed, started at 2019-12-20 12:34:39.477777001 +0000 UTC m=+451.648450429, now = 2019-12-20 12:34:42.320124962 +0000 UTC m=+454.490798437, took 2.842348008s
		    2019/12/20 12:34:42 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2019/12/20 12:34:42 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6

make stressrace TESTS=scaledata/filesystem_simulator/nodes=6 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@f1f15250871d124178f40bf340ef48719b50e7e5:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2233
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1668189-1577777590-70-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.80:26257,10.128.0.94:26257,10.128.0.64:26257,10.128.0.17:26257,10.128.0.113:26257,10.128.0.88:26257'  returned:
		    stderr:
		    327738458 +0000 UTC m=+331.893454209, now = 2019-12-31 12:20:24.3561922 +0000 UTC m=+332.921908025, took 1.028453816s
		    2019/12/31 12:20:24 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/31 12:20:24 RobustDB.RandomDB chose DB at index 3
		    2019/12/31 12:20:24 ExecuteTx retry attempt 1 failed, started at 2019-12-31 12:20:22.424507152 +0000 UTC m=+330.990222900, now = 2019-12-31 12:20:24.356279014 +0000 UTC m=+332.921994829, took 1.931771929s
		    2019/12/31 12:20:24 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/31 12:20:24 RobustDB.RandomDB chose DB at index 1
		    2019/12/31 12:20:24 ExecuteTx retry attempt 1 failed, started at 2019-12-31 12:20:22.372906001 +0000 UTC m=+330.938621758, now = 2019-12-31 12:20:24.35629465 +0000 UTC m=+332.922010436, took 1.983388678s
		    2019/12/31 12:20:24 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2019/12/31 12:20:24 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6

make stressrace TESTS=scaledata/filesystem_simulator/nodes=6 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@3d98dae853edbde083cc23b60ad6696958909849:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2233
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1688019-1578988048-73-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.157:26257,10.128.0.162:26257,10.128.0.172:26257,10.128.0.117:26257,10.128.0.68:26257,10.128.0.66:26257'  returned:
		    stderr:
		    dRelations - 7105, stripes - 1164
		    2020/01/14 12:30:53 Deleted stripes for uuid 91e7c00b-1aaa-4f89-97f9-9c4793e5a3bc
		    2020/01/14 12:30:53 ExecuteTx retry attempt 1 failed, started at 2020-01-14 12:30:51.120982226 +0000 UTC m=+208.508812156, now = 2020-01-14 12:30:53.737286749 +0000 UTC m=+211.125116749, took 2.616304593s
		    2020/01/14 12:30:53 pq error - Error code : XX000, Error class : XX
		    2020/01/14 12:30:53 Attempt failed with error pq: internal error: unexpected error from the vectorized runtime: rpc error: code = Canceled desc = context canceled: ... Retrying after sleeping 5ns
		    2020/01/14 12:30:53 RobustDB.RandomDB chose DB at index 2
		    2020/01/14 12:30:53 ExecuteTx retry attempt 1 failed, started at 2020-01-14 12:30:50.291832223 +0000 UTC m=+207.679662158, now = 2020-01-14 12:30:53.742739275 +0000 UTC m=+211.130569278, took 3.45090712s
		    2020/01/14 12:30:53 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/14 12:30:53 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6

make stressrace TESTS=scaledata/filesystem_simulator/nodes=6 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@0046d62b00aa9e74489d1f44125948c9af729e2b:

		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1690426-1579074735-68-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.156:26257,10.128.1.147:26257,10.128.1.157:26257,10.128.1.39:26257,10.128.1.159:26257,10.128.1.158:26257'  returned:
		    stderr:
		    20/01/15 12:31:43 RobustDB.RandomDB chose DB at index 0
		    2020/01/15 12:31:43 Deleted &{8daecf45-f3ae-4524-bb2e-54d123267d7e 1 0 74 default}
		    2020/01/15 12:31:43 RobustDB.RandomDB chose DB at index 4
		    2020/01/15 12:31:43 RobustDB.RandomDB chose DB at index 0
		    2020/01/15 12:31:43 ExecuteTx retry attempt 1 failed, started at 2020-01-15 12:31:43.929494804 +0000 UTC m=+576.498155984, now = 2020-01-15 12:31:43.95378135 +0000 UTC m=+576.522442564, took 24.28658ms
		    2020/01/15 12:31:43 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/15 12:31:43 Deleted &{585bd0f3-d169-41fd-b819-b97fb7a0ef7d 1 0 25 default}
		    2020/01/15 12:31:43 ExecuteTx retry attempt 1 failed, started at 2020-01-15 12:31:41.382674075 +0000 UTC m=+573.951335247, now = 2020-01-15 12:31:43.954547781 +0000 UTC m=+576.523209039, took 2.571873792s
		    2020/01/15 12:31:43 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/15 12:31:43 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6

make stressrace TESTS=scaledata/filesystem_simulator/nodes=6 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1693614-1579160767-72-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.20:26257,10.128.0.19:26257,10.128.0.198:26257,10.128.0.231:26257,10.128.0.230:26257,10.128.0.3:26257'  returned:
		    stderr:
		    ... Retrying after sleeping 5ns
		    2020/01/16 13:56:00 ExecuteTx retry attempt 1 failed, started at 2020-01-16 13:56:00.517188021 +0000 UTC m=+332.729481432, now = 2020-01-16 13:56:00.540428603 +0000 UTC m=+332.752722045, took 23.240613ms
		    2020/01/16 13:56:00 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/16 13:56:00 ExecuteTx retry attempt 1 failed, started at 2020-01-16 13:56:00.47666814 +0000 UTC m=+332.688961546, now = 2020-01-16 13:56:00.540480716 +0000 UTC m=+332.752774136, took 63.81259ms
		    2020/01/16 13:56:00 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/16 13:56:00 ExecuteTx retry attempt 1 failed, started at 2020-01-16 13:55:56.821752901 +0000 UTC m=+329.034046303, now = 2020-01-16 13:56:00.540527888 +0000 UTC m=+332.752821350, took 3.718775047s
		    2020/01/16 13:56:00 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/16 13:56:00 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6

make stressrace TESTS=scaledata/filesystem_simulator/nodes=6 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on release-19.2@4b12dcfe61b6a7fb7fae5ab50208753cd3e08f46:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1698248-1579333204-64-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.88:26257,10.128.1.92:26257,10.128.1.83:26257,10.128.1.91:26257,10.128.1.84:26257,10.128.1.74:26257'  returned:
		    stderr:
		    x retry attempt 1 failed, started at 2020-01-18 12:13:46.430416965 +0000 UTC m=+210.956997671, now = 2020-01-18 12:13:46.467306372 +0000 UTC m=+210.993887168, took 36.889497ms
		    2020/01/18 12:13:46 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/18 12:13:46 RobustDB.RandomDB chose DB at index 4
		    2020/01/18 12:13:46 ExecuteTx retry attempt 1 failed, started at 2020-01-18 12:13:46.358125823 +0000 UTC m=+210.884706532, now = 2020-01-18 12:13:46.467322751 +0000 UTC m=+210.993903481, took 109.196949ms
		    2020/01/18 12:13:46 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/18 12:13:46 ExecuteTx retry attempt 1 failed, started at 2020-01-18 12:13:45.728081608 +0000 UTC m=+210.254662313, now = 2020-01-18 12:13:46.467321648 +0000 UTC m=+210.993902373, took 739.24006ms
		    2020/01/18 12:13:46 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/18 12:13:46 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6
roachdash

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on provisional_202001212141_v19.1.7@a8444e28d3e7bf07c8ee0c6d5809e7466b57f6f9:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1702949-1579652818-61-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.53:26257,10.128.0.125:26257,10.128.0.128:26257,10.128.0.109:26257,10.128.0.102:26257,10.128.0.166:26257'  returned:
		    stderr:
		     Retrying after sleeping 5ns
		    2020/01/22 05:05:24 ExecuteTx retry attempt 1 failed, started at 2020-01-22 05:05:23.872469055 +0000 UTC m=+454.382833501, now = 2020-01-22 05:05:24.317515848 +0000 UTC m=+454.827880344, took 445.046843ms
		    2020/01/22 05:05:24 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/22 05:05:24 ExecuteTx retry attempt 1 failed, started at 2020-01-22 05:05:23.652541171 +0000 UTC m=+454.162905626, now = 2020-01-22 05:05:24.318197791 +0000 UTC m=+454.828562276, took 665.65665ms
		    2020/01/22 05:05:24 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/22 05:05:24 ExecuteTx retry attempt 1 failed, started at 2020-01-22 05:05:22.825071484 +0000 UTC m=+453.335435942, now = 2020-01-22 05:05:24.318429023 +0000 UTC m=+454.828793510, took 1.493357568s
		    2020/01/22 05:05:24 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/22 05:05:24 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=6
roachdash

powered by pkg/cmd/internal/issues

@tbg tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020
This was referenced Jan 23, 2020
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@545c96573d36519c27115278187fdd441f54a64d:

		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:719
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1705428-1579766323-66-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.33:26257,10.128.1.30:26257,10.128.1.31:26257,10.128.0.52:26257,10.128.1.27:26257,10.128.1.35:26257'  returned:
		    stderr:
		    3:17:51.961702712 +0000 UTC m=+576.542455555, took 2.253914165s
		    2020/01/23 13:17:51 pq error - Error code : XX000, Error class : XX
		    2020/01/23 13:17:51 Attempt failed with error pq: internal error: unexpected error from the vectorized runtime: rpc error: code = Canceled desc = context canceled: ... Retrying after sleeping 5ns
		    2020/01/23 13:17:51 RobustDB.RandomDB chose DB at index 3
		    2020/01/23 13:17:51 RobustDB.RandomDB chose DB at index 4
		    2020/01/23 13:17:51 RobustDB.RandomDB chose DB at index 5
		    2020/01/23 13:17:51 Consistency Test 0_275 @ 1579785470329374263.0000000000: sizes :- files - 18976, childRelations - 18975, stripes - 3130
		    2020/01/23 13:17:51 ExecuteTx retry attempt 1 failed, started at 2020-01-23 13:17:49.694537147 +0000 UTC m=+574.275289972, now = 2020-01-23 13:17:51.963079299 +0000 UTC m=+576.543832156, took 2.268542184s
		    2020/01/23 13:17:51 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/23 13:17:51 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@72ce65798f55af352c6a38c306542f63dfba465f:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:49
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:719
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2359
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1969
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:118
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2294
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1710860-1579939958-69-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.64:26257,10.128.0.135:26257,10.128.0.78:26257,10.128.0.89:26257,10.128.0.148:26257,10.128.0.72:26257'  returned:
		    stderr:
		    leeping 10ns
		    2020/01/25 13:16:33 RobustDB.RandomDB chose DB at index 2
		    2020/01/25 13:16:33 ExecuteTx retry attempt 3 failed, started at 2020-01-25 13:16:33.697398784 +0000 UTC m=+332.958691490, now = 2020-01-25 13:16:33.697636303 +0000 UTC m=+332.958929084, took 237.594µs
		    2020/01/25 13:16:33 Attempt failed with error dial tcp 10.128.0.78:26257: connect: connection refused: ... Retrying after sleeping 20ns
		    2020/01/25 13:16:33 RobustDB.RandomDB chose DB at index 5
		    2020/01/25 13:16:33 RobustDB.RandomDB chose DB at index 1
		    2020/01/25 13:16:33 RobustDB.RandomDB chose DB at index 5
		    2020/01/25 13:16:33 RobustDB.RandomDB chose DB at index 1
		    2020/01/25 13:16:33 ExecuteTx retry attempt 1 failed, started at 2020-01-25 13:16:31.127521249 +0000 UTC m=+330.388813951, now = 2020-01-25 13:16:33.706005574 +0000 UTC m=+332.967298345, took 2.578484394s
		    2020/01/25 13:16:33 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/25 13:16:33 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten nvanbenschoten mentioned this issue Jan 27, 2020
Closed
19 tasks
@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 7, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@b67d7d14e22f227caa4a2bc3727e3baf815fb25e:

		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_130813.192_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1734669-1581063190-68-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.243:26257,10.128.1.254:26257,10.128.1.250:26257,10.128.1.249:26257,10.128.1.252:26257,10.128.1.253:26257'  returned:
		    stderr:
		    2-07 13:15:48.141546519 +0000 UTC m=+454.206670187, now = 2020-02-07 13:15:48.507811087 +0000 UTC m=+454.572934774, took 366.264587ms
		    2020/02/07 13:15:48 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 5
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 1
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 4
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 5
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 5
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 2
		    2020/02/07 13:15:48 RobustDB.RandomDB chose DB at index 1
		    2020/02/07 13:15:48 ExecuteTx retry attempt 1 failed, started at 2020-02-07 13:15:47.832663643 +0000 UTC m=+453.897787302, now = 2020-02-07 13:15:48.518534607 +0000 UTC m=+454.583658328, took 685.871026ms
		    2020/02/07 13:15:48 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/07 13:15:48 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 8, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@c4cf5585cbda55caadb1782f6705e52a83659538:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_125300.402_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1739656-1581235400-65-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.159:26257,10.128.0.162:26257,10.128.0.161:26257,10.128.0.158:26257,10.128.0.160:26257,10.128.0.140:26257'  returned:
		    stderr:
		    , now = 2020-02-09 13:02:37.651359703 +0000 UTC m=+576.510223514, took 315.205µs
		    2020/02/09 13:02:37 Attempt failed with error dial tcp 10.128.0.160:26257: connect: connection refused: ... Retrying after sleeping 20ns
		    2020/02/09 13:02:37 RobustDB.RandomDB chose DB at index 3
		    2020/02/09 13:02:37 ExecuteTx retry attempt 1 failed, started at 2020-02-09 13:02:35.934983837 +0000 UTC m=+574.793847597, now = 2020-02-09 13:02:37.657686006 +0000 UTC m=+576.516549824, took 1.722702227s
		    2020/02/09 13:02:37 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/09 13:02:37 RobustDB.RandomDB chose DB at index 0
		    2020/02/09 13:02:37 ExecuteTx retry attempt 1 failed, started at 2020-02-09 13:02:37.024343137 +0000 UTC m=+575.883206896, now = 2020-02-09 13:02:37.666689817 +0000 UTC m=+576.525553623, took 642.346727ms
		    2020/02/09 13:02:37 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/09 13:02:37 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@a51ddb8342ff54283baf5c7556cb1d4ff8c4e5da:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_131223.604_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1761088-1582358300-67-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.13:26257,10.128.1.12:26257,10.128.1.11:26257,10.128.0.190:26257,10.128.0.254:26257,10.128.1.9:26257'  returned:
		    stderr:
		    0/02/22 13:22:00 Consistency Test 13_525 @ 1582377720215447936.0000000000: sizes :- files - 15569, childRelations - 15568, stripes - 2639
		    2020/02/22 13:22:01 Created file 5_2257 with uuid 5fefd913-a17f-4612-b407-1e1c7b8c7078 and parent /default
		    2020/02/22 13:22:01 RobustDB.RandomDB chose DB at index 2
		    2020/02/22 13:22:01 RobustDB.RandomDB chose DB at index 0
		    2020/02/22 13:22:01 Consistency Test 12_506 @ 1582377720222508335.0000000000: sizes :- files - 15569, childRelations - 15568, stripes - 2639
		    2020/02/22 13:22:01 Consistency Test 9_523 @ 1582377720181923745.0000000000: sizes :- files - 15569, childRelations - 15568, stripes - 2639
		    2020/02/22 13:22:01 ExecuteTx retry attempt 1 failed, started at 2020-02-22 13:21:59.447403067 +0000 UTC m=+575.101771216, now = 2020-02-22 13:22:01.072133014 +0000 UTC m=+576.726501249, took 1.624730033s
		    2020/02/22 13:22:01 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/22 13:22:01 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 25, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@14094e3d5ea5f548dfada7bdb6e0e1158f53e168:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_133418.152_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1776248-1583050059-70-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.130:26257,10.128.0.50:26257,10.128.0.80:26257,10.128.0.55:26257,10.128.0.90:26257,10.128.0.31:26257'  returned:
		    stderr:
		    cuteTx retry attempt 1 failed, started at 2020-03-01 13:35:47.09997516 +0000 UTC m=+88.196338946, now = 2020-03-01 13:35:48.140849346 +0000 UTC m=+89.237213155, took 1.040874209s
		    2020/03/01 13:35:48 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/01 13:35:48 ExecuteTx retry attempt 1 failed, started at 2020-03-01 13:35:47.767853526 +0000 UTC m=+88.864217316, now = 2020-03-01 13:35:48.140852786 +0000 UTC m=+89.237216623, took 372.999307ms
		    2020/03/01 13:35:48 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/01 13:35:48 RobustDB.RandomDB chose DB at index 5
		    2020/03/01 13:35:48 ExecuteTx retry attempt 1 failed, started at 2020-03-01 13:35:47.506327555 +0000 UTC m=+88.602691329, now = 2020-03-01 13:35:48.140917205 +0000 UTC m=+89.237281014, took 634.589685ms
		    2020/03/01 13:35:48 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/03/01 13:35:48 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@irfansharif
Copy link
Contributor

Same as #43273, probably.

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 4, 2020
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 5, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@c9b189b3a2171c8da78c75fe80dd02933e18a5e5:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_125550.071_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1786843-1583394619-61-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.102:26257,10.128.0.87:26257,10.128.0.43:26257,10.128.0.54:26257,10.128.0.79:26257,10.128.0.173:26257'  returned:
		    stderr:
		    -70969471d49b and parent /default
		    2020/03/05 13:05:51 Created file 15_2346 with uuid 8b6adb56-f020-44b3-95fa-33aab6a6d5bc and parent /default
		    2020/03/05 13:05:51 Created file 14_2379 with uuid 72c34b4e-62d5-42c2-bf9a-666f9b4b95fd and parent /default
		    2020/03/05 13:05:51 Consistency Test 0_417 @ 1583413550662606720.0000000000: sizes :- files - 14766, childRelations - 14765, stripes - 2319
		    2020/03/05 13:05:51 Created file 0_2515 with uuid 6bff77f6-e47e-464d-90c4-dd6ffcd3f881 and parent /default
		    2020/03/05 13:05:51 Created file 1_2344 with uuid 49d1125a-00ae-4a05-98c4-584b241e00e7 and parent /default
		    2020/03/05 13:05:51 Consistency Test 8_404 @ 1583413550756969286.0000000000: sizes :- files - 14767, childRelations - 14766, stripes - 2319
		    2020/03/05 13:05:51 Consistency Test 3_422 @ 1583413551053324268.0000000000: sizes :- files - 14784, childRelations - 14783, stripes - 2322
		    2020/03/05 13:05:51 Consistency Test 5_440 @ 1583413551274041327.0000000000: sizes :- files - 14784, childRelations - 14783, stripes - 2322
		    
		    stdout::
		  - context canceled
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@72c4a1bd411f2f82bf9aaa22883821a946614148:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_135815.714_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1799071-1583824469-81-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.28:26257,10.128.0.16:26257,10.128.0.51:26257,10.128.0.46:26257,10.128.0.55:26257,10.128.0.54:26257'  returned:
		    stderr:
		    66158 +0000 UTC m=+454.505733419, took 7m34.472097747s
		    2020/03/10 14:05:50 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/10 14:05:50 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/10 14:05:50 ExecuteTx retry attempt 1 failed, started at 2020-03-10 13:58:16.464028813 +0000 UTC m=+0.030096044, now = 2020-03-10 14:05:50.939666221 +0000 UTC m=+454.505733497, took 7m34.475637453s
		    2020/03/10 14:05:50 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/10 14:05:50 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2020/03/10 14:05:50 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2020/03/10 14:05:50 driver: bad connection
		    2020/03/10 14:05:50 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@f5585e933a097b53242d8d5800127b821a9a4d41:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_140531.383_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1807675-1584170966-74-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.35:26257,10.128.0.50:26257,10.128.0.42:26257,10.128.0.48:26257,10.128.0.41:26257,10.128.0.37:26257'  returned:
		    stderr:
		     Retrying after sleeping 5ns
		    2020/03/14 14:11:04 ExecuteTx retry attempt 1 failed, started at 2020-03-14 14:11:04.671544405 +0000 UTC m=+332.559285018, now = 2020-03-14 14:11:04.909519859 +0000 UTC m=+332.797260481, took 237.975463ms
		    2020/03/14 14:11:04 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/14 14:11:04 ExecuteTx retry attempt 1 failed, started at 2020-03-14 14:11:04.718129991 +0000 UTC m=+332.605870588, now = 2020-03-14 14:11:04.909553451 +0000 UTC m=+332.797294067, took 191.423479ms
		    2020/03/14 14:11:04 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/14 14:11:04 ExecuteTx retry attempt 1 failed, started at 2020-03-14 14:11:04.653742223 +0000 UTC m=+332.541482823, now = 2020-03-14 14:11:04.909584742 +0000 UTC m=+332.797325363, took 255.84254ms
		    2020/03/14 14:11:04 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/03/14 14:11:04 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@33d71472dc01cbc5064b3c5e1fcd666a33f606de:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_141831.001_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1808695-1584256913-70-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.11:26257,10.128.0.18:26257,10.128.0.29:26257,10.128.0.15:26257,10.128.0.16:26257,10.128.0.10:26257'  returned:
		    stderr:
		    ailed with error driver: bad connection: ... Retrying after sleeping 10ns
		    2020/03/15 14:24:04 ExecuteTx retry attempt 1 failed, started at 2020-03-15 14:20:13.257621846 +0000 UTC m=+101.508343313, now = 2020-03-15 14:24:04.406600827 +0000 UTC m=+332.657322354, took 3m51.148979041s
		    2020/03/15 14:24:04 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/15 14:24:04 ExecuteTx retry attempt 1 failed, started at 2020-03-15 14:20:11.960896457 +0000 UTC m=+100.211617919, now = 2020-03-15 14:24:04.406598439 +0000 UTC m=+332.657319974, took 3m52.445702055s
		    2020/03/15 14:24:04 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/15 14:24:04 RobustDB.RandomDB chose DB at index 1
		    2020/03/15 14:24:04 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2020/03/15 14:24:04 RobustDB.RandomDB chose DB at index 4
		    2020/03/15 14:24:04 Database - 3_8 1 driver: bad connection
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@d8bdc938177ea4b5226b95ee6cc51af0951054fc:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_142036.103_n7_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1809758-1584345030-72-n7cpu4:7 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.176:26257,10.128.0.95:26257,10.128.0.165:26257,10.128.0.177:26257,10.128.0.175:26257,10.128.0.178:26257'  returned:
		    stderr:
		    ion: ... Retrying after sleeping 10ns
		    2020/03/16 14:26:09 ExecuteTx retry attempt 2 failed, started at 2020-03-16 14:22:05.983935047 +0000 UTC m=+89.115078982, now = 2020-03-16 14:26:09.48621564 +0000 UTC m=+332.617359640, took 4m3.502280658s
		    2020/03/16 14:26:09 Attempt failed with error driver: bad connection: ... Retrying after sleeping 10ns
		    2020/03/16 14:26:09 ExecuteTx retry attempt 1 failed, started at 2020-03-16 14:20:44.595880579 +0000 UTC m=+7.727024515, now = 2020-03-16 14:26:09.486351086 +0000 UTC m=+332.617495033, took 5m24.890470518s
		    2020/03/16 14:26:09 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/16 14:26:09 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2020/03/16 14:26:09 driver: bad connection
		    2020/03/16 14:26:09 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2020/03/16 14:26:09 driver: bad connection
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 19, 2020
Closed
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 20, 2020
Closed
@irfansharif irfansharif mentioned this issue Mar 20, 2020
Closed
24 tasks
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 24, 2020
Closed
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@irfansharif
Copy link
Contributor

Recent failures due to what was explained in #43273 (comment).

Dead node issues before those was fixed in #45769.

This leaves the various (probably benign) EOF errors.

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@c0855e9fcf979cca1094d7edeeacb9882ddc9f96:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200327-1832621/scaledata/filesystem_simulator/nodes=6/run_1
	cluster.go:2368,scaledata.go:112,scaledata.go:49,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:112
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:49
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 6: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1832621-1585293773-82-n7cpu4 --oneshot --ignore-empty-nodes: exit status 1 7: skipped
		6: dead
		2: 3721
		1: 4080
		3: 3791
		5: 3754
		4: 4269
		Error: UNCLASSIFIED_PROBLEM:
		  - 6: dead
		    main.glob..func13
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		    main.wrap.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		    main.main
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:203
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@0e9f07ae35ad39e8789bc51c49066c123968a666:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200328-1834610/scaledata/filesystem_simulator/nodes=6/run_1
	cluster.go:2368,scaledata.go:112,scaledata.go:49,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:112
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:49
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 6: dead
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=6 failed on master@92495ffb2182671b3338a8c6cd2734cb6de5d392:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200329-1836230/scaledata/filesystem_simulator/nodes=6/run_1
	cluster.go:2368,scaledata.go:112,scaledata.go:49,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:112
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:49
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 2: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1836230-1585466755-74-n7cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: dead
		4: 6673
		6: 7652
		7: skipped
		3: 7183
		1: 6962
		5: 7536
		Error: UNCLASSIFIED_PROBLEM:
		  - 2: dead
		    main.glob..func13
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		    main.wrap.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		    github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		    main.main
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:203
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: /scaledata/filesystem_simulator/nodes=6
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@irfansharif irfansharif mentioned this issue Mar 30, 2020
Merged
@irfansharif
Copy link
Contributor

#43273 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
20.1
Development

No branches or pull requests
4 participants
@andreimatei @tbg @irfansharif @cockroach-teamcity