roachtest: kv50/enc=false/nodes=4/cpu=96/batch=64 failed #35785

cockroach-teamcity · 2019-03-15T10:52:27Z

SHA: https://github.com/cockroachdb/cockroach/commits/7ce9188c6e64465d9dcb9f0ca0f113dd0e584da0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1178908&tab=buildLog

The test failed on release-2.1:
	cluster.go:1267,kv.go:69,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1178908-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		 read
		     44s        0          232.9          194.4    671.1   2080.4   2550.1   3489.7 write
		     45s        0          210.1          201.8    167.8    838.9   1208.0   1610.6 read
		     45s        0          237.1          195.4    637.5   1811.9   2952.8   4026.5 write
		     46s        0          216.0          202.1    285.2    872.4   1275.1   1744.8 read
		     46s        0          261.9          196.8    637.5   2147.5   3221.2   5368.7 write
		     47s        0          219.1          202.4    302.0    805.3   1275.1   1610.6 read
		     47s        0          227.1          197.4    704.6   1677.7   3087.0   4563.4 write
		     48s        0          121.0          200.7    260.0    838.9    939.5   1040.2 read
		     48s        0          135.0          196.1    771.8   1744.8   2147.5   2684.4 write
		     49s        0            0.0          196.7      0.0      0.0      0.0      0.0 read
		     49s        0            0.0          192.1      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1626,kv.go:72,kv.go:137,test.go:1214: unexpected node event: 3: dead

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-03-20T10:22:08Z

SHA: https://github.com/cockroachdb/cockroach/commits/deb08aa82922ae081c043ff7a6e85960c09910e2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1187498&tab=buildLog

The test failed on release-2.1:
	cluster.go:1267,kv.go:69,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1187498-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		 read
		     55s        0          241.1          221.2    570.4   2013.3   3221.2   4831.8 write
		     56s        0          197.0          222.9    192.9   1006.6   1610.6   2147.5 read
		     56s        0          252.0          221.8    604.0   1946.2   3087.0   3892.3 write
		     57s        0          303.0          224.3    151.0    906.0   1543.5   2684.4 read
		     57s        0          232.0          222.0    604.0   2013.3   3221.2   3355.4 write
		     58s        0          278.0          225.2    125.8    771.8   1073.7   1476.4 read
		     58s        0          251.0          222.5    604.0   1946.2   2818.6   3489.7 write
		     59s        0          216.0          225.0    125.8    805.3    973.1   2281.7 read
		     59s        0          238.0          222.7    671.1   1811.9   2684.4   3087.0 write
		    1m0s        0            0.0          221.3      0.0      0.0      0.0      0.0 read
		    1m0s        0            0.0          219.0      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1626,kv.go:72,kv.go:137,test.go:1214: unexpected node event: 3: dead

cockroach-teamcity · 2019-03-22T10:06:56Z

SHA: https://github.com/cockroachdb/cockroach/commits/c11656058e4a36c0c62275d7c188ef8921e02928

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1191975&tab=buildLog

The test failed on release-2.1:
	cluster.go:1267,kv.go:69,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1191975-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		 read
		      3s        0          237.9          202.0    369.1    939.5   1610.6   2952.8 write
		      4s        0          117.1          176.0    771.8   1208.0   1409.3   1543.5 read
		      4s        0          122.1          182.0    570.4   1342.2   1744.8   1811.9 write
		      5s        0          171.0          175.0    906.0   1409.3   1610.6   1879.0 read
		      5s        0          188.0          183.2    503.3   2281.7   2684.4   3489.7 write
		      6s        0          246.0          186.8    570.4   1073.7   1677.7   2415.9 read
		      6s        0          196.0          185.3    318.8   1879.0   2818.6   3087.0 write
		      7s        0           42.0          166.1    536.9   1073.7   2013.3   2013.3 read
		      7s        0           55.0          166.7    285.2   1073.7   1275.1   3892.3 write
		      8s        0            0.0          145.4      0.0      0.0      0.0      0.0 read
		      8s        0            0.0          145.9      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1626,kv.go:72,kv.go:137,test.go:1214: unexpected node event: 1: dead

cockroach-teamcity · 2019-04-21T10:39:13Z

SHA: https://github.com/cockroachdb/cockroach/commits/df200cbf3f407dbf349aa601ff9036b4dff88e83

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1252822&tab=buildLog

The test failed on release-2.1:
	cluster.go:1329,kv.go:69,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1252822-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		744.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     11s        0          208.8          108.8   1409.3   1879.0   2080.4   2281.7 read
		     11s        0          199.8          110.6   1275.1   2550.1   3221.2   3489.7 write
		     12s        0          184.0          115.1    604.0   1073.7   1476.4   2147.5 read
		     12s        0          183.0          116.7    402.7   2281.7   3623.9   4295.0 write
		     13s        0          204.0          121.9    486.5   1409.3   1811.9   3355.4 read
		     13s        0          187.0          122.1    369.1   2684.4   4563.4   5905.6 write
		     14s        0          154.0          124.2    453.0   1409.3   1677.7   1811.9 read
		     14s        0          136.0          123.1    335.5   1879.0   3623.9   4831.8 write
		     15s        0            0.0          115.9      0.0      0.0      0.0      0.0 read
		     15s        0            0.0          114.9      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1688,kv.go:72,kv.go:137,test.go:1237: unexpected node event: 3: dead
	cluster.go:953,context.go:89,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,kv.go:72,kv.go:137,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1252822-kv50-enc-false-nodes-4-cpu-96-batch-64 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: 4681
		4: 5340
		3: dead
		1: 4552
		Error:  3: dead

tbg · 2019-04-23T12:43:57Z

Uh oh. @nvanbenschoten take a look at this. Are we maybe missing some backport for the data races you recently fixed? All failures are on 2.1.

E190421 10:37:11.545209 141697 util/log/crash_reporting.go:203  [n3,client=10.142.0.133:46312,user=root,txn=57ea2cc6] a panic has occurred!
panic: runtime error: index out of range [recovered]
	panic: runtime error: index out of range

goroutine 141697 [running]:
panic(0x2777680, 0x4260d80)
	/usr/local/go/src/runtime/panic.go:551 +0x3c1 fp=0xc42fd7e318 sp=0xc42fd7e278 pc=0x6de8e1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc4208f8090, 0x305d860, 0xc42be55890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:184 +0x11f fp=0xc42fd7e378 sp=0xc42fd7e318 pc=0xf0f58f
runtime.call32(0x0, 0x2c3c328, 0xc428bf63f0, 0x1800000018)
	/usr/local/go/src/runtime/asm_amd64.s:573 +0x3b fp=0xc42fd7e3a8 sp=0xc42fd7e378 pc=0x70bc7b
panic(0x2777680, 0x4260d80)
	/usr/local/go/src/runtime/panic.go:502 +0x229 fp=0xc42fd7e448 sp=0xc42fd7e3a8 pc=0x6de749
runtime.panicindex()
	/usr/local/go/src/runtime/panic.go:28 +0x5e fp=0xc42fd7e468 sp=0xc42fd7e448 pc=0x6dd58e
github.com/cockroachdb/cockroach/pkg/roachpb.(*Value).MarshalTo(0xc42d197378, 0xc42fcb7949, 0xa, 0xa, 0xc, 0x14, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/data.pb.go:1069 +0x21d fp=0xc42fd7e4c8 sp=0xc42fd7e468 pc=0xe6a17d
github.com/cockroachdb/cockroach/pkg/roachpb.(*PutRequest).MarshalTo(0xc42d197340, 0xc42fcb7935, 0x1e, 0x1e, 0x20, 0x2, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:6905 +0x1ba fp=0xc42fd7e528 sp=0xc42fd7e4c8 pc=0xdce06a
github.com/cockroachdb/cockroach/pkg/roachpb.(*RequestUnion_Put).MarshalTo(0xc42848a358, 0xc42fcb7933, 0x20, 0x20, 0xdf1409, 0xc42848a358, 0x22)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10346 +0xe1 fp=0xc42fd7e578 sp=0xc42fd7e528 pc=0xde27e1
github.com/cockroachdb/cockroach/pkg/roachpb.(*RequestUnion).MarshalTo(0xc42fd7e620, 0xc42fcb7933, 0x20, 0x20, 0x22, 0xb3, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10317 +0x73 fp=0xc42fd7e5c0 sp=0xc42fd7e578 pc=0xde2563
github.com/cockroachdb/cockroach/pkg/roachpb.(*BatchRequest).MarshalTo(0xc434875100, 0xc42fcb7880, 0xd3, 0xd3, 0xd3, 0xd3, 0x2aa8a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:11729 +0x245 fp=0xc42fd7e640 sp=0xc42fd7e5c0 pc=0xdea4f5
github.com/cockroachdb/cockroach/pkg/roachpb.(*BatchRequest).Marshal(0xc434875100, 0x2aa8a00, 0xc434875100, 0x7fe5af4a2650, 0xc434875100, 0xc42fd7e601)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:11704 +0x7f fp=0xc42fd7e698 sp=0xc42fd7e640 pc=0xdea22f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/encoding/proto.codec.Marshal(0x2aa8a00, 0xc434875100, 0x6b6f16, 0x3, 0xc4200cc070, 0xc4200cc000, 0xc434998f58)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/encoding/proto/proto.go:70 +0x19e fp=0xc42fd7e718 sp=0xc42fd7e698 pc=0xbd60be
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/encoding/proto.(*codec).Marshal(0x48833e8, 0x2aa8a00, 0xc434875100, 0x6de194, 0x2c3e220, 0xc42fd7e7d0, 0x10, 0xc42c768900)
	<autogenerated>:1 +0x48 fp=0xc42fd7e760 sp=0xc42fd7e718 pc=0xbd6838
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.encode(0x7fe5af43e678, 0x48833e8, 0x2aa8a00, 0xc434875100, 0xc429f423f0, 0x0, 0xc429f423f0, 0x0, 0xbf272fd5e07dffd2)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:487 +0x61 fp=0xc42fd7e7e8 sp=0xc42fd7e760 pc=0xbee9e1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).sendMsg(0xc429d47ad0, 0x2aa8a00, 0xc434875100, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:479 +0xc9 fp=0xc42fd7e908 sp=0xc42fd7e7e8 pc=0xbfbb99
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).SendMsg(0xc434875200, 0x2aa8a00, 0xc434875100, 0xc420a3cf00, 0x2b1f443)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:400 +0x43 fp=0xc42fd7e940 sp=0xc42fd7e908 pc=0xbfb763
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x305d860, 0xc42be55890, 0x2b1f443, 0x21, 0x2aa8a00, 0xc434875100, 0x2997ca0, 0xc420ef1d40, 0xc420a3cf00, 0xc4259d5200, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:75 +0xfe fp=0xc42fd7e9c8 sp=0xc42fd7e940 pc=0xbde54e
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x305d860, 0xc42be55890, 0x2b1f443, 0x21, 0x2aa8a00, 0xc434875100, 0x2997ca0, 0xc420ef1d40, 0xc420a3cf00, 0x2c3e210, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:44 +0xe28 fp=0xc42fd7ede8 sp=0xc42fd7e9c8 pc=0x14d1268
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc420a3cf00, 0x305d860, 0xc42be55890, 0x2b1f443, 0x21, 0x2aa8a00, 0xc434875100, 0x2997ca0, 0xc420ef1d40, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x109 fp=0xc42fd7eea0 sp=0xc42fd7ede8 pc=0xbde109
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.Invoke(0x305d860, 0xc42be55890, 0x2b1f443, 0x21, 0x2aa8a00, 0xc434875100, 0x2997ca0, 0xc420ef1d40, 0xc420a3cf00, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:60 +0xc1 fp=0xc42fd7ef20 sp=0xc42fd7eea0 pc=0xbde421
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc42592c7f8, 0x305d860, 0xc42be55890, 0xc434875100, 0x0, 0x0, 0x0, 0x305d860, 0xc42be55890, 0x100000002)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:6570 +0xd2 fp=0xc42fd7efa8 sp=0xc42fd7ef20 pc=0xdcc0d2
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc429f42390, 0x305d860, 0xc42be55890, 0x303a360, 0xc42592c7f8, 0x0, 0x0, 0x20000

nvanbenschoten · 2019-04-24T21:22:25Z

None of the fixes for the data races are on 2.1 yet. I'll backport them.

It looks like we picked this up at least twice on Sentry: #34674.

cockroach-teamcity · 2019-05-08T04:08:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/d554884a4e474cc06213230d5ba7d757a88e9e46

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279548&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1474,kv.go:69,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279548-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		563.4 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     11s        0          229.9          190.1    436.2    872.4   1073.7   1543.5 read
		     11s        0          242.9          189.7    369.1   1677.7   3087.0   4831.8 write
		     12s        0          241.0          194.3    436.2    838.9   1208.0   1543.5 read
		     12s        0          241.0          194.0    369.1   1946.2   3758.1   6979.3 write
		     13s        0          215.9          196.0    419.4   1040.2   1677.7   2013.3 read
		     13s        0          235.9          197.2    402.7   1811.9   3221.2   5905.6 write
		     14s        0           82.1          187.9    385.9    973.1   1275.1   1409.3 read
		     14s        0           64.1          187.7    335.5   1476.4   1946.2   3892.3 write
		     15s        0            0.0          175.3      0.0      0.0      0.0      0.0 read
		     15s        0            0.0          175.2      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1833,kv.go:72,kv.go:137,test.go:1251: unexpected node event: 3: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,kv.go:72,kv.go:137,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1279548-kv50-enc-false-nodes-4-cpu-96-batch-64 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: 4768
		4: 4796
		3: dead
		1: 4842
		Error:  3: dead

cockroach-teamcity · 2019-05-09T02:42:25Z

SHA: https://github.com/cockroachdb/cockroach/commits/d554884a4e474cc06213230d5ba7d757a88e9e46

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1281453&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1474,kv.go:69,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1281453-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		1000 splits
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      1s        0            4.5           43.0    906.0   1006.6   1006.6   1006.6 read
		      1s        0           16.7          159.0    402.7    906.0   1006.6   1006.6 write
		      2s        0          160.9          102.0   1476.4   1946.2   1946.2   2013.3 read
		      2s        0          102.9          131.0    453.0   1476.4   1610.6   1811.9 write
		      3s        0          261.9          155.3    604.0   1543.5   2080.4   2415.9 read
		      3s        0          210.9          157.6    285.2    704.6   2281.7   2415.9 write
		      4s        0          167.0          158.2    805.3   1006.6   1073.7   1140.9 read
		      4s        0          143.0          154.0    436.2    973.1   1073.7   1140.9 write
		      5s        0            0.0          126.6      0.0      0.0      0.0      0.0 read
		      5s        0            0.0          123.2      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1833,kv.go:72,kv.go:137,test.go:1251: unexpected node event: 2: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,kv.go:72,kv.go:137,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1281453-kv50-enc-false-nodes-4-cpu-96-batch-64 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: dead
		3: 4735
		1: 4854
		4: 4786
		Error:  2: dead

cockroach-teamcity · 2019-05-11T21:13:58Z

SHA: https://github.com/cockroachdb/cockroach/commits/d554884a4e474cc06213230d5ba7d757a88e9e46

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/enc=false/nodes=4/cpu=96/batch=64 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1286052&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1474,kv.go:69,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1286052-kv50-enc-false-nodes-4-cpu-96-batch-64:5 -- ./workload run kv --init --splits=1000 --histograms=logs/stats.json --concurrency=256 --duration=10m --read-percent=50 --batch=64 {pgurl:1-4} returned:
		stderr:
		
		stdout:
		 read
		     14s        0          271.0          190.9    251.7   1946.2   3221.2   7247.8 write
		     15s        0          254.0          201.7    268.4    872.4   1342.2   1409.3 read
		     15s        0          236.0          193.9    260.0   2013.3   3758.1   4831.8 write
		     16s        0          198.0          201.4    285.2   1610.6   2281.7   2281.7 read
		     16s        0          190.0          193.7    285.2   2415.9   2818.6   4295.0 write
		     17s        0          214.0          202.2    402.7   1879.0   2684.4   2684.4 read
		     17s        0          225.0          195.5    385.9   2550.1   3623.9   6710.9 write
		     18s        0          178.0          200.8    352.3   1610.6   2684.4   3355.4 read
		     18s        0          197.0          195.6    385.9   2415.9   4831.8  12348.0 write
		     19s        0            0.0          190.3      0.0      0.0      0.0      0.0 read
		     19s        0            0.0          185.3      0.0      0.0      0.0      0.0 write
		Error: EOF
		Error:  exit status 1
		: exit status 1
	cluster.go:1833,kv.go:72,kv.go:137,test.go:1251: unexpected node event: 3: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,kv.go:72,kv.go:137,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1286052-kv50-enc-false-nodes-4-cpu-96-batch-64 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: 4732
		3: dead
		1: 4928
		2: 4783
		Error:  3: dead

nvanbenschoten · 2019-05-14T17:50:30Z

Closed by #37495.

cockroach-teamcity added this to the 19.1 milestone Mar 15, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Mar 15, 2019

tbg assigned nvanbenschoten Apr 23, 2019

yuzefovich mentioned this issue May 9, 2019

release: v2.1.7 #37334

Closed

14 tasks

nvanbenschoten mentioned this issue May 13, 2019

release-2.1: roachpb: prevent data race on Transaction #37495

Merged

nvanbenschoten closed this as completed May 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: kv50/enc=false/nodes=4/cpu=96/batch=64 failed #35785

roachtest: kv50/enc=false/nodes=4/cpu=96/batch=64 failed #35785

cockroach-teamcity commented Mar 15, 2019

cockroach-teamcity commented Mar 20, 2019

cockroach-teamcity commented Mar 22, 2019

cockroach-teamcity commented Apr 21, 2019

tbg commented Apr 23, 2019

nvanbenschoten commented Apr 24, 2019

cockroach-teamcity commented May 8, 2019

cockroach-teamcity commented May 9, 2019

cockroach-teamcity commented May 11, 2019

nvanbenschoten commented May 14, 2019

roachtest: kv50/enc=false/nodes=4/cpu=96/batch=64 failed #35785

roachtest: kv50/enc=false/nodes=4/cpu=96/batch=64 failed #35785

Comments

cockroach-teamcity commented Mar 15, 2019

cockroach-teamcity commented Mar 20, 2019

cockroach-teamcity commented Mar 22, 2019

cockroach-teamcity commented Apr 21, 2019

tbg commented Apr 23, 2019

nvanbenschoten commented Apr 24, 2019

cockroach-teamcity commented May 8, 2019

cockroach-teamcity commented May 9, 2019

cockroach-teamcity commented May 11, 2019

nvanbenschoten commented May 14, 2019