Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: scaledata/filesystem_simulator/nodes=3 failed #43273

Closed
cockroach-teamcity opened this issue Dec 18, 2019 · 53 comments · Fixed by #46750
Closed

roachtest: scaledata/filesystem_simulator/nodes=3 failed #43273

cockroach-teamcity opened this issue Dec 18, 2019 · 53 comments · Fixed by #46750
Assignees
@irfansharif
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on release-19.1@f8a99239043416cf032b47fd3ce71e4e364a0af1:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:700
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1649950-1576652741-13-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.64:26257,10.128.0.76:26257,10.128.0.95:26257'  returned:
		    stderr:
		    . Retrying after sleeping 5ns
		    2019/12/18 07:19:02 ExecuteTx retry attempt 1 failed, started at 2019-12-18 07:19:02.015582039 +0000 UTC m=+576.179591800, now = 2019-12-18 07:19:02.693001638 +0000 UTC m=+576.857011446, took 677.419646ms
		    2019/12/18 07:19:02 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/18 07:19:02 ExecuteTx retry attempt 1 failed, started at 2019-12-18 07:19:01.344880133 +0000 UTC m=+575.508889915, now = 2019-12-18 07:19:02.693048442 +0000 UTC m=+576.857058229, took 1.348168314s
		    2019/12/18 07:19:02 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/18 07:19:02 ExecuteTx retry attempt 1 failed, started at 2019-12-18 07:19:01.485761492 +0000 UTC m=+575.649771266, now = 2019-12-18 07:19:02.6930567 +0000 UTC m=+576.857066499, took 1.207295233s
		    2019/12/18 07:19:02 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2019/12/18 07:19:02 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Dec 18, 2019
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on release-19.2@7e9d38c44864898657bcd095b21cebe4702fa2da:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:700
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1649968-1576654634-08-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.15.224:26257,10.128.15.248:26257,10.128.15.228:26257'  returned:
		    stderr:
		    0a16be13 and parent /default
		    2019/12/18 07:52:08 ExecuteTx retry attempt 1 failed, started at 2019-12-18 07:52:07.115083226 +0000 UTC m=+575.837553555, now = 2019-12-18 07:52:08.192564014 +0000 UTC m=+576.915034410, took 1.077480855s
		    2019/12/18 07:52:08 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/18 07:52:08 ExecuteTx retry attempt 1 failed, started at 2019-12-18 07:52:04.801538742 +0000 UTC m=+573.524009083, now = 2019-12-18 07:52:08.192573653 +0000 UTC m=+576.915044003, took 3.39103492s
		    2019/12/18 07:52:08 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/18 07:52:08 ExecuteTx retry attempt 1 failed, started at 2019-12-18 07:52:06.829733556 +0000 UTC m=+575.552203882, now = 2019-12-18 07:52:08.192599681 +0000 UTC m=+576.915070033, took 1.362866151s
		    2019/12/18 07:52:08 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2019/12/18 07:52:08 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on release-2.1@a32057733127620847d5121d54984f4b5be9f33a:

		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:700
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1337
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1653478-1576738307-10-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.57:26257,10.128.0.66:26257,10.128.0.45:26257'  returned:
		    stderr:
		    126781s
		    2019/12/19 07:04:42 Attempt failed with error driver: bad connection: ... Retrying after sleeping 10ns
		    2019/12/19 07:04:42 Attempt failed with error driver: bad connection: ... Retrying after sleeping 20ns
		    2019/12/19 07:04:42 ExecuteTx retry attempt 5 failed, started at 2019-12-19 07:00:39.345242298 +0000 UTC m=+332.904247073, now = 2019-12-19 07:04:42.892812165 +0000 UTC m=+576.451817001, took 4m3.547569928s
		    2019/12/19 07:04:42 Attempt failed with error driver: bad connection: ... Retrying after sleeping 80ns
		    2019/12/19 07:04:42 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2019/12/19 07:04:42 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2019/12/19 07:04:42 driver: bad connection
		    2019/12/19 07:04:42 Aborting Retries because retry duration of 300 seconds expired : *errors.errorString : driver: bad connection
		    2019/12/19 07:04:42 driver: bad connection
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on release-19.1@a85a9a4ee0440cccdfd3e314b8f01394e52bdf3f:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:700
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1658803-1576912021-03-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.20:26257,10.128.0.24:26257,10.128.0.57:26257'  returned:
		    stderr:
		     Retrying after sleeping 5ns
		    2019/12/21 07:18:20 ExecuteTx retry attempt 1 failed, started at 2019-12-21 07:18:19.890357135 +0000 UTC m=+454.029581842, now = 2019-12-21 07:18:20.929289744 +0000 UTC m=+455.068514483, took 1.038932641s
		    2019/12/21 07:18:20 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/21 07:18:20 ExecuteTx retry attempt 1 failed, started at 2019-12-21 07:18:20.31745683 +0000 UTC m=+454.456681535, now = 2019-12-21 07:18:20.929566804 +0000 UTC m=+455.068791569, took 612.110034ms
		    2019/12/21 07:18:20 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/21 07:18:20 ExecuteTx retry attempt 1 failed, started at 2019-12-21 07:18:18.964299895 +0000 UTC m=+453.103524601, now = 2019-12-21 07:18:20.929877878 +0000 UTC m=+455.069102608, took 1.965578007s
		    2019/12/21 07:18:20 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2019/12/21 07:18:20 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@dd9b1c1f40dda59ee9d446416106d311ae5ce1e6:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:700
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1658785-1576914291-14-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.249:26257,10.128.0.241:26257,10.128.0.230:26257'  returned:
		    stderr:
		     0
		    2019/12/21 07:59:22 &{f9b3199c-5eff-4375-873f-76d3cd53a1e0 0 default}
		    2019/12/21 07:59:22 Created file 15_2307 with uuid 01947340-6f5f-4ef0-a3b9-d8e93199ab51 and parent /default
		    2019/12/21 07:59:22 RobustDB.RandomDB chose DB at index 1
		    2019/12/21 07:59:22 ExecuteTx retry attempt 1 failed, started at 2019-12-21 07:59:22.101855796 +0000 UTC m=+471.918425407, now = 2019-12-21 07:59:22.104176915 +0000 UTC m=+471.920746555, took 2.321148ms
		    2019/12/21 07:59:22 Attempt failed with error dial tcp 10.128.0.241:26257: connect: connection refused: ... Retrying after sleeping 5ns
		    2019/12/21 07:59:22 RobustDB.RandomDB chose DB at index 2
		    2019/12/21 07:59:22 Consistency Test 14_303 @ 1576915160950631708.0000000000: sizes :- files - 13347, childRelations - 13347, stripes - 2086
		    2019/12/21 07:59:22 Consistency Test 14_303 @ 1576915160950631708.0000000000: ChildRelation {/default 0_2028 47101dbf-d9ff-4ec2-931b-e7687499d3d3 default}: 47101dbf-d9ff-4ec2-931b-e7687499d3d3 child does not exist in files
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on release-19.2@599c988773fa6fb4544369a50fe8d06b90c64171:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2210
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1664336-1577432048-06-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.158:26257,10.128.0.154:26257,10.128.0.180:26257'  returned:
		    stderr:
		    ed at 2019-12-27 07:45:41.02486776 +0000 UTC m=+454.126379629, now = 2019-12-27 07:45:41.558934907 +0000 UTC m=+454.660446800, took 534.067171ms
		    2019/12/27 07:45:41 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2019/12/27 07:45:41 ExecuteTx retry attempt 1 failed, started at 2019-12-27 07:45:39.671845233 +0000 UTC m=+452.773357089, now = 2019-12-27 07:45:41.558973882 +0000 UTC m=+454.660485797, took 1.887128708s
		    2019/12/27 07:45:41 Aborting Retries because this error of type *crdb.AmbiguousCommitError is not retryable : driver: bad connection
		    2019/12/27 07:45:41 RobustDB.RandomDB chose DB at index 2
		    2019/12/27 07:45:41 ExecuteTx retry attempt 1 failed, started at 2019-12-27 07:45:40.362519243 +0000 UTC m=+453.464031099, now = 2019-12-27 07:45:41.55898346 +0000 UTC m=+454.660495403, took 1.196464304s
		    2019/12/27 07:45:41 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2019/12/27 07:45:41 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on release-19.2@9aaf5fb6f7108858d200dd3d199deb11871a09b4:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2233
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1671769-1578037401-05-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.243:26257,10.128.0.223:26257,10.128.0.244:26257'  returned:
		    stderr:
		     Retrying after sleeping 5ns
		    2020/01/03 07:53:15 ExecuteTx retry attempt 1 failed, started at 2020-01-03 07:53:15.011865999 +0000 UTC m=+333.009692011, now = 2020-01-03 07:53:15.062818074 +0000 UTC m=+333.060644124, took 50.952113ms
		    2020/01/03 07:53:15 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/03 07:53:15 ExecuteTx retry attempt 1 failed, started at 2020-01-03 07:53:14.216892182 +0000 UTC m=+332.214718187, now = 2020-01-03 07:53:15.062822339 +0000 UTC m=+333.060648373, took 845.930186ms
		    2020/01/03 07:53:15 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/03 07:53:15 ExecuteTx retry attempt 1 failed, started at 2020-01-03 07:53:14.019766095 +0000 UTC m=+332.017592106, now = 2020-01-03 07:53:15.062870309 +0000 UTC m=+333.060696435, took 1.043104329s
		    2020/01/03 07:53:15 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/03 07:53:15 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@0046d62b00aa9e74489d1f44125948c9af729e2b:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1690426-1579074735-10-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.251:26257,10.128.1.34:26257,10.128.1.6:26257'  returned:
		    stderr:
		    08:00:10 RobustDB.RandomDB chose DB at index 1
		    2020/01/15 08:00:10 RobustDB.RandomDB chose DB at index 2
		    2020/01/15 08:00:10 RobustDB.RandomDB chose DB at index 0
		    2020/01/15 08:00:10 RobustDB.RandomDB chose DB at index 1
		    2020/01/15 08:00:10 RobustDB.RandomDB chose DB at index 0
		    2020/01/15 08:00:10 RobustDB.RandomDB chose DB at index 2
		    2020/01/15 08:00:10 ExecuteTx retry attempt 1 failed, started at 2020-01-15 08:00:10.84091488 +0000 UTC m=+211.049594810, now = 2020-01-15 08:00:10.900758179 +0000 UTC m=+211.109438129, took 59.843319ms
		    2020/01/15 08:00:10 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/15 08:00:10 ExecuteTx retry attempt 1 failed, started at 2020-01-15 08:00:09.154629915 +0000 UTC m=+209.363309856, now = 2020-01-15 08:00:10.900821643 +0000 UTC m=+211.109501635, took 1.746191779s
		    2020/01/15 08:00:10 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/15 08:00:10 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
Repro

Artifacts: /scaledata/filesystem_simulator/nodes=3

make stressrace TESTS=scaledata/filesystem_simulator/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@tbg tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020
@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 23, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@3c408a47707a15d4b5c92031c4eafcfa643d307c:

		    main.runSqlapp
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:121
		    main.registerScaleData.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:48
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:719
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2359
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1969
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:117
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2294
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1708438-1579854217-11-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.1.87:26257,10.128.1.86:26257,10.128.1.84:26257'  returned:
		    stderr:
		    Retrying after sleeping 5ns
		    2020/01/24 08:35:26 ExecuteTx retry attempt 1 failed, started at 2020-01-24 08:35:24.627186835 +0000 UTC m=+331.083154389, now = 2020-01-24 08:35:26.491411749 +0000 UTC m=+332.947379346, took 1.864224957s
		    2020/01/24 08:35:26 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/24 08:35:26 ExecuteTx retry attempt 1 failed, started at 2020-01-24 08:35:26.244981619 +0000 UTC m=+332.700949187, now = 2020-01-24 08:35:26.491450715 +0000 UTC m=+332.947418294, took 246.469107ms
		    2020/01/24 08:35:26 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/01/24 08:35:26 ExecuteTx retry attempt 1 failed, started at 2020-01-24 08:35:24.617081356 +0000 UTC m=+331.073048910, now = 2020-01-24 08:35:26.491357021 +0000 UTC m=+332.947324587, took 1.874275677s
		    2020/01/24 08:35:26 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/01/24 08:35:26 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten nvanbenschoten mentioned this issue Jan 27, 2020
Closed
19 tasks
@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 7, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@e671a4ef97cbc6cf5d22f8f322fd45733d302094:

		  - error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_081350.001_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1749385-1581667748-05-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.235:26257,10.128.0.243:26257,10.128.0.255:26257'  returned:
		    stderr:
		    0/02/14 08:21:25 ExecuteTx retry attempt 3 failed, started at 2020-02-14 08:21:25.529398344 +0000 UTC m=+454.771239625, now = 2020-02-14 08:21:25.529722853 +0000 UTC m=+454.771564195, took 324.57µs
		    2020/02/14 08:21:25 Attempt failed with error dial tcp 10.128.0.235:26257: connect: connection refused: ... Retrying after sleeping 20ns
		    2020/02/14 08:21:25 ExecuteTx retry attempt 1 failed, started at 2020-02-14 08:21:23.77443652 +0000 UTC m=+453.016277790, now = 2020-02-14 08:21:25.533243667 +0000 UTC m=+454.775084971, took 1.758807181s
		    2020/02/14 08:21:25 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/14 08:21:25 ExecuteTx retry attempt 1 failed, started at 2020-02-14 08:21:24.750342628 +0000 UTC m=+453.992183902, now = 2020-02-14 08:21:25.537383697 +0000 UTC m=+454.779224999, took 787.041097ms
		    2020/02/14 08:21:25 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/14 08:21:25 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@bb14400cdd1397f47c3781b4830199f169cef497:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_081649.466_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1751913-1581840727-06-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.17:26257,10.128.0.20:26257,10.128.0.100:26257'  returned:
		    stderr:
		    +0000 UTC m=+576.534939971, now = 2020-02-16 08:26:26.843256466 +0000 UTC m=+576.540924456, took 5.984485ms
		    2020/02/16 08:26:26 Attempt failed with error dial tcp 10.128.0.100:26257: connect: connection refused: ... Retrying after sleeping 10ns
		    2020/02/16 08:26:26 ExecuteTx retry attempt 1 failed, started at 2020-02-16 08:26:26.841080393 +0000 UTC m=+576.538748315, now = 2020-02-16 08:26:26.843336874 +0000 UTC m=+576.541005042, took 2.256727ms
		    2020/02/16 08:26:26 Attempt failed with error dial tcp 10.128.0.100:26257: connect: connection refused: ... Retrying after sleeping 5ns
		    2020/02/16 08:26:26 RobustDB.RandomDB chose DB at index 1
		    2020/02/16 08:26:26 ExecuteTx retry attempt 1 failed, started at 2020-02-16 08:26:25.963545079 +0000 UTC m=+575.661213017, now = 2020-02-16 08:26:26.846651345 +0000 UTC m=+576.544319609, took 883.106592ms
		    2020/02/16 08:26:26 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/16 08:26:26 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@36273d3daa0bb659b58e826c67e6079f0747aac6:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_080615.960_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1755262-1582099313-08-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.176:26257,10.128.0.179:26257,10.128.0.175:26257'  returned:
		    stderr:
		     stripe 0
		    2020/02/19 08:13:51 &{b608307c-305a-4eb4-9f19-e81744bdd076 0 default}
		    2020/02/19 08:13:51 Created file 12_1612 with uuid e46f2f86-b92c-4fbf-8e48-fc78014012c5 and parent /default
		    2020/02/19 08:13:51 RobustDB.RandomDB chose DB at index 1
		    2020/02/19 08:13:51 Consistency Test 4_381 @ 1582100030821617644.0000000000: sizes :- files - 10697, childRelations - 10696, stripes - 1673
		    2020/02/19 08:13:51 RobustDB.RandomDB chose DB at index 0
		    2020/02/19 08:13:51 Consistency Test 13_391 @ 1582100030817675782.0000000000: sizes :- files - 10697, childRelations - 10696, stripes - 1673
		    2020/02/19 08:13:51 RobustDB.RandomDB chose DB at index 0
		    2020/02/19 08:13:51 ExecuteTx retry attempt 1 failed, started at 2020-02-19 08:13:50.01140027 +0000 UTC m=+453.336214389, now = 2020-02-19 08:13:51.630807224 +0000 UTC m=+454.955621374, took 1.619406985s
		    2020/02/19 08:13:51 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/19 08:13:51 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@b797cad6d130714748983bc53d4611ddc6151153:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_080829.460_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1759645-1582272286-11-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.168:26257,10.128.0.167:26257,10.128.0.165:26257'  returned:
		    stderr:
		     Retrying after sleeping 40ns
		    2020/02/21 08:16:04 ExecuteTx retry attempt 1 failed, started at 2020-02-21 08:16:03.082306921 +0000 UTC m=+452.886056818, now = 2020-02-21 08:16:04.86376675 +0000 UTC m=+454.667516679, took 1.781459861s
		    2020/02/21 08:16:04 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/21 08:16:04 ExecuteTx retry attempt 1 failed, started at 2020-02-21 08:16:04.272706707 +0000 UTC m=+454.076456601, now = 2020-02-21 08:16:04.870617341 +0000 UTC m=+454.674367258, took 597.910657ms
		    2020/02/21 08:16:04 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/21 08:16:04 ExecuteTx retry attempt 1 failed, started at 2020-02-21 08:16:04.283084197 +0000 UTC m=+454.086834100, now = 2020-02-21 08:16:04.875360808 +0000 UTC m=+454.679110730, took 592.27663ms
		    2020/02/21 08:16:04 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/21 08:16:04 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@fc5c7f093bf1e86852c3b839bc0f6710d9902729:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_081050.735_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1772892-1582877180-11-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.81:26257,10.128.0.32:26257,10.128.0.70:26257'  returned:
		    stderr:
		    . Retrying after sleeping 5ns
		    2020/02/28 08:20:28 ExecuteTx retry attempt 1 failed, started at 2020-02-28 08:20:27.81500682 +0000 UTC m=+576.366611379, now = 2020-02-28 08:20:28.12374768 +0000 UTC m=+576.675352283, took 308.740904ms
		    2020/02/28 08:20:28 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/28 08:20:28 ExecuteTx retry attempt 1 failed, started at 2020-02-28 08:20:26.343026485 +0000 UTC m=+574.894631044, now = 2020-02-28 08:20:28.123969554 +0000 UTC m=+576.675574155, took 1.780943111s
		    2020/02/28 08:20:28 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/02/28 08:20:28 ExecuteTx retry attempt 1 failed, started at 2020-02-28 08:20:27.330850819 +0000 UTC m=+575.882455375, now = 2020-02-28 08:20:28.124172891 +0000 UTC m=+576.675777492, took 793.322117ms
		    2020/02/28 08:20:28 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/28 08:20:28 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@3e2334ded18d2f18c08f2f912be085df563ca6ff:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_080850.894_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1775192-1582963470-01-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.56:26257,10.128.0.28:26257,10.128.0.4:26257'  returned:
		    stderr:
		     pq error - Error code : XX000, Error class : XX
		    2020/02/29 08:18:28 Attempt failed with error pq: internal error: unexpected error from the vectorized runtime: rpc error: code = Canceled desc = context canceled: ... Retrying after sleeping 5ns
		    2020/02/29 08:18:28 RobustDB.RandomDB chose DB at index 1
		    2020/02/29 08:18:28 ExecuteTx retry attempt 2 failed, started at 2020-02-29 08:18:28.278358798 +0000 UTC m=+576.660074952, now = 2020-02-29 08:18:28.278653294 +0000 UTC m=+576.660369464, took 294.512µs
		    2020/02/29 08:18:28 Attempt failed with error dial tcp 10.128.0.28:26257: connect: connection refused: ... Retrying after sleeping 10ns
		    2020/02/29 08:18:28 ExecuteTx retry attempt 1 failed, started at 2020-02-29 08:18:26.647215253 +0000 UTC m=+575.028931390, now = 2020-02-29 08:18:28.284571226 +0000 UTC m=+576.666287446, took 1.637356056s
		    2020/02/29 08:18:28 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/02/29 08:18:28 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@5f9a71adb995837bcff27b9456188018434be4b8:

		  - error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_081916.204_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1780216-1583223197-13-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.15.218:26257,10.128.15.203:26257,10.128.15.195:26257'  returned:
		    stderr:
		    03/03 08:28:53 ExecuteTx retry attempt 1 failed, started at 2020-03-03 08:28:53.09169688 +0000 UTC m=+576.170574539, now = 2020-03-03 08:28:53.496087567 +0000 UTC m=+576.574965250, took 404.390711ms
		    2020/03/03 08:28:53 Attempt failed with error driver: bad connection: ... Retrying after sleeping 5ns
		    2020/03/03 08:28:53 ExecuteTx retry attempt 2 failed, started at 2020-03-03 08:28:53.485243075 +0000 UTC m=+576.564120743, now = 2020-03-03 08:28:53.496282474 +0000 UTC m=+576.575160177, took 11.039434ms
		    2020/03/03 08:28:53 Attempt failed with error dial tcp 10.128.15.195:26257: connect: connection refused: ... Retrying after sleeping 10ns
		    2020/03/03 08:28:53 ExecuteTx retry attempt 1 failed, started at 2020-03-03 08:28:52.57049657 +0000 UTC m=+575.649374225, now = 2020-03-03 08:28:53.496122874 +0000 UTC m=+576.575000553, took 925.626328ms
		    2020/03/03 08:28:53 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/03/03 08:28:53 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@irfansharif irfansharif mentioned this issue Mar 3, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@d6a52a8f6f07b3b6b32af2243075e5365fb21c45:

		  - error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_074938.944_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1783198-1583307906-05-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.15.221:26257,10.128.15.220:26257,10.128.15.228:26257'  returned:
		    stderr:
		    st 10_460 @ 1583308779538593818.0000000000: sizes :- files - 12781, childRelations - 12780, stripes - 1993
		    2020/03/04 07:59:40 Consistency Test 4_452 @ 1583308779654971787.0000000000: sizes :- files - 12783, childRelations - 12782, stripes - 1990
		    2020/03/04 07:59:40 Created file 3_2052 with uuid 2194c2fe-1591-4f00-948d-9ea89f924c87 and parent /default
		    2020/03/04 07:59:40 Created file 11_1971 with uuid 6d51b8c7-31f7-417b-9c6b-0b3c72d4ee2a and parent /default
		    2020/03/04 07:59:40 Consistency Test 12_556 @ 1583308779633950874.0000000000: sizes :- files - 12783, childRelations - 12782, stripes - 1990
		    2020/03/04 07:59:40 Consistency Test 8_464 @ 1583308779688440498.0000000000: sizes :- files - 12784, childRelations - 12783, stripes - 1990
		    2020/03/04 07:59:40 Consistency Test 5_449 @ 1583308779708141506.0000000000: sizes :- files - 12785, childRelations - 12784, stripes - 1990
		    2020/03/04 07:59:40 Consistency Test 14_506 @ 1583308779986309462.0000000000: sizes :- files - 12798, childRelations - 12797, stripes - 1990
		    
		    stdout::
		  - context canceled
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@954fe69d554162aec0fbc001aad1fe5103d8df13:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_080856.412_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1790595-1583481796-15-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.15.235:26257,10.128.0.250:26257,10.128.15.243:26257'  returned:
		    stderr:
		    r from the vectorized runtime: rpc error: code = Canceled desc = context canceled: ... Retrying after sleeping 5ns
		    2020/03/06 08:18:33 ExecuteTx retry attempt 1 failed, started at 2020-03-06 08:18:33.561300891 +0000 UTC m=+576.441128518, now = 2020-03-06 08:18:33.740586243 +0000 UTC m=+576.620413891, took 179.285373ms
		    2020/03/06 08:18:33 pq error - Error code : XX000, Error class : XX
		    2020/03/06 08:18:33 Attempt failed with error pq: internal error: unexpected error from the vectorized runtime: rpc error: code = Canceled desc = context canceled: ... Retrying after sleeping 5ns
		    2020/03/06 08:18:33 RobustDB.RandomDB chose DB at index 2
		    2020/03/06 08:18:33 ExecuteTx retry attempt 1 failed, started at 2020-03-06 08:18:33.158029429 +0000 UTC m=+576.037857058, now = 2020-03-06 08:18:33.740814798 +0000 UTC m=+576.620642471, took 582.785413ms
		    2020/03/06 08:18:33 Aborting Retries because this error of type *errors.errorString is not retryable : unexpected EOF
		    2020/03/06 08:18:33 unexpected EOF
		    Error:  exit status 255
		    
		    stdout::
		  - exit status 1
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).scaledata/filesystem_simulator/nodes=3 failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_072522.411_n4_filesystemsimulator_:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.runSqlapp.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scaledata.go:108
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1796240-1583738442-11-n4cpu4:4 -- ./filesystem_simulator  --duration_secs=600 --num_workers=16 --cockroach_ip_addresses_csv='10.128.0.135:26257,10.128.0.117:26257,10.128.0.88:26257'  returned:
		    stderr:
		    020/03/09 07:35:23 Deleted child_relations for uuid 0ed9db8f-ca22-4b97-b5b9-fe458a06660d
		    2020/03/09 07:35:23 Deleted child_relations for uuid c1b4160b-ad0e-4704-b5e8-2ed1e3e7a661
		    2020/03/09 07:35:23 Deleted &{0ed9db8f-ca22-4b97-b5b9-fe458a06660d 1 2 236 default}
		    2020/03/09 07:35:23 Deleted &{c1b4160b-ad0e-4704-b5e8-2ed1e3e7a661 1 2 53 default}
		    2020/03/09 07:35:28 Removing &{f22c1866-1025-43c2-be78-33528eb59063 1 0 244 default}
		    2020/03/09 07:35:28 Deleted stripes for uuid f22c1866-1025-43c2-be78-33528eb59063
		    2020/03/09 07:35:28 Deleted child_relations for uuid f22c1866-1025-43c2-be78-33528eb59063
		    2020/03/09 07:35:28 Deleted &{f22c1866-1025-43c2-be78-33528eb59063 1 0 244 default}
		    2020/03/09 07:35:28 Removing &{60a0c6a8-e28e-4490-8b40-3136cb979524 1 0 5 default}
		    2020/03/09 07:35:28 Deleted stripes for uuid 60a0c6a8-e28e-4490-8b40-3136cb979524
		    2020/03/09 07:35:28 Deleted child_relations for uuid 60a0c6a8-e28e-4490-8b40-3136cb979524
		    2020/03/09 07:35:28 Deleted &{60a0c6a8-e28e-4490-8b40-3136cb979524 1 0 5 default}
		    
		    stdout::
		  - context canceled
More

Artifacts: /scaledata/filesystem_simulator/nodes=3
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten
Copy link
Member

Also, @irfansharif do you know whether https://github.com/scaledata/rksql/pull/6 was included when you re-deployed these tests?

@irfansharif
Copy link
Contributor

Also, @irfansharif do you know whether scaledata/rksql#6 was included when you re-deployed these tests?

It wasn't. It's currently deployed at https://github.com/irfansharif/rksql/tree/aaa85833508aad281d9defe8b0edd63bc6103faa. I'll include that before tonight's runs.

@irfansharif
Copy link
Contributor

irfansharif commented Mar 26, 2020 •
edited
Loading

read timestamp: 1
write timestamp: 3
it finds a value with timestamp: 2

This is exactly what we're running into. What do we want the new write timestamp to be? Should it not stay at 3 (given that it got to 3 for a good reason)?

@irfansharif
Copy link
Contributor

We were informed to bump up our read and write timestamps, but we already have a higher write timestamp, so we should simply be pulling up our read timestamp to match. Option 2 makes sense to me too.

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@jordanlewis jordanlewis mentioned this issue Mar 30, 2020
Closed
83 tasks
@nvanbenschoten
Copy link
Member

We were informed to bump up our read and write timestamps, but we already have a higher write timestamp, so we should simply be pulling up our read timestamp to match. Option 2 makes sense to me too.

I just added this to the 20.1 release blocker list.

@irfansharif irfansharif mentioned this issue Mar 30, 2020
Merged
@irfansharif
Copy link
Contributor

Also, @irfansharif do you know whether scaledata/rksql#6 was included when you re-deployed these tests?

Picked it up on this new fork: https://github.com/cockroachdb/rksql

@irfansharif irfansharif mentioned this issue Mar 30, 2020
Merged
@otan otan mentioned this issue Mar 30, 2020
Closed
24 tasks
irfansharif added a commit to irfansharif/cockroach that referenced this issue Mar 30, 2020
@irfansharif
kvserver: disallow timestamp regression with locking scans
25c1000 
After cockroachdb#46004, locking scans can now hit WriteTooOld errors if they
encounter values at timestamps higher than their read timestamps. The
`ActualTimestamp` recorded is the `encountered ts + 1`. When determining
what the new timestamp for the txn should be, previously we blindly used
the generated `encountered ts + 1`. This was buggy, and could lead to a
timestamp regression in the case where a txn with (read_ts, write_ts) =
(1, 4) finds a value with `ts = 2`. If we try to "bump" the txn to
`ts = 3`, we're regressing the write ts.

Now, when determining what the new timestamp should be, we ensure we use
`max(encountered ts + 1, txn's current write ts)`.

Fixes cockroachdb#43273.

Release note: None
Release justification: High-priority bug fix (resulted in panics for
contended UPDATEs)
@irfansharif
Copy link
Contributor

I'm not able to repro the panic with #46750.

irfansharif added a commit to irfansharif/cockroach that referenced this issue Mar 30, 2020
@irfansharif
kvserver: disallow timestamp regression with locking scans
a37e984 
After cockroachdb#46004, locking scans can now hit WriteTooOld errors if they
encounter values at timestamps higher than their read timestamps. The
`ActualTimestamp` recorded is the `encountered ts + 1`. When determining
what the new timestamp for the txn should be, previously we blindly used
the generated `encountered ts + 1`. This was buggy, and could lead to a
timestamp regression in the case where a txn with (read_ts, write_ts) =
(1, 4) finds a value with `ts = 2`. If we try to "bump" the txn to
`ts = 3`, we're regressing the write ts.

Now, when determining what the new timestamp should be, we ensure we use
`max(encountered ts + 1, txn's current write ts)`.

Fixes cockroachdb#43273.

Release note: None
Release justification: High-priority bug fix (resulted in panics for
contended UPDATEs)
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@nvanbenschoten
Copy link
Member 

F200331 07:37:21.026224 233807 kv/kvserver/replica_batch_updates.go:227  [n2,s2,r33/2:/Table/5{3-4}] trying to bump to 1585640240.292762266,1 inconsistent with ba.Txn.ReadTimestamp: 1585640240.066514692,0, ba.Txn.WriteTimestamp: 1585640240.963916454,1

Same issue as #43273 (comment), which is fixed by #46750.

craig bot pushed a commit that referenced this issue Mar 31, 2020
@irfansharif @Amruta-Ranade
Merge #46750 #46762
98a351e 
46750: kvserver: disallow timestamp regression with locking scans r=irfansharif a=irfansharif

After #46004, locking scans can now hit WriteTooOld errors if they
encounter values at timestamps higher than their read timestamps. The
`ActualTimestamp` recorded is the `encountered ts + 1`. When determining
what the new timestamp for the txn should be, previously we blindly used
the generated `encountered ts + 1`. This was buggy, and could lead to a
timestamp regression in the case where a txn with (read_ts, write_ts) =
(1, 4) finds a value with `ts = 2`. If we try to "bump" the txn to
`ts = 3`, we're regressing the write ts.

Now, when determining what the new timestamp should be, we ensure we use
`max(encountered ts + 1, txn's current write ts)`.

Fixes #43273.

Release note: None

46762: RBAC SQL diagram updates r=richardjcai a=Amruta-Ranade

RBAC SQL diagram updates
Release note: None
Release justification: Docs change


Co-authored-by: irfan sharif <[email protected]>
Co-authored-by: Amruta <[email protected]>
@craig craig bot closed this as completed in 3a9d409 Mar 31, 2020
@irfansharif
Copy link
Contributor

This might still flake on retry errors that are not captured as retry errors, but those are innocuous, and now that we have https://github.com/cockroachdb/rksql, can fix them on the fly.

irfansharif added a commit to irfansharif/cockroach that referenced this issue Apr 1, 2020
@irfansharif
kvserver: disallow timestamp regression with locking scans
2dee6d7 
After cockroachdb#46004, locking scans can now hit WriteTooOld errors if they
encounter values at timestamps higher than their read timestamps. The
`ActualTimestamp` recorded is the `encountered ts + 1`. When determining
what the new timestamp for the txn should be, previously we blindly used
the generated `encountered ts + 1`. This was buggy, and could lead to a
timestamp regression in the case where a txn with (read_ts, write_ts) =
(1, 4) finds a value with `ts = 2`. If we try to "bump" the txn to
`ts = 3`, we're regressing the write ts.

Now, when determining what the new timestamp should be, we ensure we use
`max(encountered ts + 1, txn's current write ts)`.

Fixes cockroachdb#43273.

Release note: None
Release justification: High-priority bug fix (resulted in panics for
contended UPDATEs)
@irfansharif irfansharif mentioned this issue Apr 1, 2020
Merged
craig bot pushed a commit that referenced this issue Apr 1, 2020
@irfansharif
Merge #46858
01409a8 
46858: release-20.1: kvserver: disallow timestamp regression with locking scans r=irfansharif a=irfansharif

Backport 1/1 commits from #46750.

/cc @cockroachdb/release

---

After #46004, locking scans can now hit WriteTooOld errors if they
encounter values at timestamps higher than their read timestamps. The
`ActualTimestamp` recorded is the `encountered ts + 1`. When determining
what the new timestamp for the txn should be, previously we blindly used
the generated `encountered ts + 1`. This was buggy, and could lead to a
timestamp regression in the case where a txn with (read_ts, write_ts) =
(1, 4) finds a value with `ts = 2`. If we try to "bump" the txn to
`ts = 3`, we're regressing the write ts.

Now, when determining what the new timestamp should be, we ensure we use
`max(encountered ts + 1, txn's current write ts)`.

Fixes #43273.

Release note: None


Co-authored-by: irfan sharif <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: disallow timestamp regression with locking scans irfansharif/cockroach
5 participants
@andreimatei @tbg @nvanbenschoten @irfansharif @cockroach-teamcity