roachtest: disk-full failed [skipped] #35328

cockroach-teamcity · 2019-03-02T08:21:08Z

SHA: https://github.com/cockroachdb/cockroach/commits/032c4980720abc1bdd71e4428e4111e6e6383297

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1158877&tab=buildLog

The test failed on master:
	cluster.go:1152,disk_full.go:84,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1158877-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp077746866, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp077746866/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190302 08:18:45.376333 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:396
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190302 08:18:45.381201 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1226,disk_full.go:50,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1158877-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.3     12.1     21.0     41.9 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2182.0         1961.8      3.1     12.1     18.9     30.4 write
		    1m2s        0         2181.0         1965.4      3.0     12.1     22.0     33.6 write
		    1m3s        0         2145.1         1968.2      3.4     11.5     16.8     33.6 write
		    1m4s        0         1222.9         1956.6      3.0     12.1     21.0     31.5 write
		    1m5s        0            0.0         1926.5      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1897.3      0.0      0.0      0.0      0.0 write
		    1m7s        0          595.0         1877.9      3.5     15.2   3221.2   3221.2 write
		    1m8s        0         1971.1         1879.2      3.5     13.1     18.9     32.5 write
		    1m9s        0         2065.9         1881.9      3.4     12.1     19.9     28.3 write
		   1m10s        0         2119.1         1885.3      3.4     11.5     18.9     33.6 write
		: signal: killed
	cluster.go:1585,disk_full.go:87,test.go:1211: Goexit() was called

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-03-03T08:12:55Z

SHA: https://github.com/cockroachdb/cockroach/commits/d888b76df319571afe4d5816f1a1f0f53905653f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1159641&tab=buildLog

The test failed on master:
	cluster.go:1152,disk_full.go:84,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1159641-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp180469990, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp180469990/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190303 08:11:36.227279 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:396
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190303 08:11:36.232411 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1226,disk_full.go:50,cluster.go:1564,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1159641-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.4     15.2     27.3     54.5 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2007.8         1834.9      3.3     13.1     24.1     35.7 write
		    1m2s        0         1989.0         1837.4      3.4     12.6     21.0     39.8 write
		    1m3s        0         2019.1         1840.3      3.3     13.6     19.9     37.7 write
		    1m4s        0         1603.0         1836.6      3.5     15.2     39.8    151.0 write
		    1m5s        0         1696.9         1834.4      3.4     14.7     27.3    142.6 write
		    1m6s        0         1663.0         1831.8      3.4     13.6     21.0    176.2 write
		    1m7s        0         1641.9         1829.0      3.3     15.7     39.8    167.8 write
		    1m8s        0         2031.1         1831.9      3.3     13.1     21.0     39.8 write
		    1m9s        0         2016.0         1834.6      3.3     13.1     21.0     30.4 write
		   1m10s        0         1985.9         1836.8      3.3     12.6     19.9     39.8 write
		: signal: killed
	cluster.go:1585,disk_full.go:87,test.go:1211: Goexit() was called

cockroach-teamcity · 2019-03-05T08:12:36Z

SHA: https://github.com/cockroachdb/cockroach/commits/047556f80399b7f70cb36c09345c6cd244210c2b

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1161867&tab=buildLog

The test failed on master:
	cluster.go:1170,disk_full.go:84,cluster.go:1582,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1161867-disk-full:1 returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp829778059, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp829778059/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190305 08:11:10.446207 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190305 08:11:10.451504 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1244,disk_full.go:50,cluster.go:1582,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1161867-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.4     14.7     23.1     39.8 write
		     59s        0         1933.0         1871.4      3.4     14.2     23.1     39.8 write
		    1m0s        0         1902.0         1871.9      3.3     14.7     26.2     60.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2072.0         1875.2      3.1     13.1     19.9     29.4 write
		    1m2s        0         2062.1         1878.2      3.1     12.6     23.1     41.9 write
		    1m3s        0         2038.9         1880.8      3.3     13.1     18.9     35.7 write
		    1m4s        0         2112.1         1884.4      3.1     12.6     19.9     41.9 write
		    1m5s        0         2137.9         1888.3      3.1     12.6     18.9     26.2 write
		    1m6s        0         2036.0         1890.5      3.3     13.6     21.0     37.7 write
		    1m7s        0         2005.1         1892.2      3.1     14.2     21.0     65.0 write
		    1m8s        0         2109.0         1895.4      3.1     12.1     19.9     37.7 write
		: signal: killed
	cluster.go:1603,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-06T08:07:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/959dcf7de0f94cfcfa0062387b109adebd1f11da

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1163702&tab=buildLog

The test failed on master:
	cluster.go:1170,disk_full.go:84,cluster.go:1582,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1163702-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp034674172, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp034674172/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190306 08:05:07.186330 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190306 08:05:07.190881 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1244,disk_full.go:50,cluster.go:1582,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1163702-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.3     14.2     24.1     52.4 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2015.9         1896.7      3.1     12.6     22.0     62.9 write
		    1m2s        0         2078.0         1899.6      3.1     12.6     17.8     52.4 write
		    1m3s        0         2065.9         1902.3      2.9     13.6     22.0     44.0 write
		    1m4s        0         1997.1         1903.8      3.3     13.6     22.0     60.8 write
		    1m5s        0         2065.9         1906.3      3.0     13.6     21.0     37.7 write
		    1m6s        0         2068.0         1908.7      3.0     13.1     19.9     32.5 write
		    1m7s        0         2062.1         1911.0      3.0     13.1     21.0     37.7 write
		    1m8s        0         2080.1         1913.5      3.3     13.1     19.9     33.6 write
		    1m9s        0         2133.8         1916.7      3.0     12.1     21.0     31.5 write
		   1m10s        0         2108.0         1919.4      3.1     12.6     19.9     30.4 write
		: signal: killed
	cluster.go:1603,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-12T07:24:13Z

SHA: https://github.com/cockroachdb/cockroach/commits/9d058d53c8a82fceb2205f1827c26f1bf36c32ba

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1172386&tab=buildLog

The test failed on master:
	cluster.go:1170,disk_full.go:84,cluster.go:1582,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1172386-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp510889544, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp510889544/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190312 07:22:20.976141 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190312 07:22:20.984369 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1244,disk_full.go:50,cluster.go:1582,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1172386-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.7     13.6     22.0     56.6 write
		    1m0s        0         1875.9         1864.2      3.8     13.6     22.0     33.6 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1920.1         1865.1      3.5     13.6     22.0     65.0 write
		    1m2s        0         2003.0         1867.3      3.4     12.6     18.9     32.5 write
		    1m3s        0          677.8         1848.4      3.1     12.6     21.0     33.6 write
		    1m4s        0            0.0         1819.6      0.0      0.0      0.0      0.0 write
		    1m5s        0            0.0         1791.6      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1764.4      0.0      0.0      0.0      0.0 write
		    1m7s        0         1792.0         1764.8      3.3     12.1     21.0   3892.3 write
		    1m8s        0         2090.4         1769.6      3.0     12.6     19.9     35.7 write
		    1m9s        0         2084.0         1774.2      3.3     12.6     18.9     29.4 write
		: signal: killed
	cluster.go:1603,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-13T05:54:32Z

SHA: https://github.com/cockroachdb/cockroach/commits/a512e390f7f2f2629f3f811bab5866c46e3e5713

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1174765&tab=buildLog

The test failed on provisional_201903122203_v19.1.0-beta.20190318:
	cluster.go:1172,disk_full.go:84,cluster.go:1584,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1174765-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp242721028, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp242721028/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190313 05:51:29.821858 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190313 05:51:29.853508 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1246,disk_full.go:50,cluster.go:1584,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1174765-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   2.9     12.6     18.9     33.6 write
		     59s        0         2388.0         2402.6      2.8     11.5     16.8     41.9 write
		    1m0s        0         2371.1         2402.1      2.9     11.0     16.8     33.6 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1832.7         2392.8      3.0     11.5     18.9     27.3 write
		    1m2s        0            0.0         2354.2      0.0      0.0      0.0      0.0 write
		    1m3s        0            0.0         2316.8      0.0      0.0      0.0      0.0 write
		    1m4s        0            0.0         2280.6      0.0      0.0      0.0      0.0 write
		    1m5s        0            0.0         2245.5      0.0      0.0      0.0      0.0 write
		    1m6s        0          853.0         2224.4      3.1     12.6   4831.8   4831.8 write
		    1m7s        0         2387.1         2226.9      2.6     11.5     16.3     26.2 write
		    1m8s        0         2283.0         2227.7      2.9     12.1     16.8     46.1 write
		: signal: killed
	cluster.go:1605,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-15T06:55:02Z

SHA: https://github.com/cockroachdb/cockroach/commits/70e3468e7ed5fb30b10eaaf972acbb0f16099d01

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1178890&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1178890-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp816771348, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp816771348/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190315 06:52:45.424297 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1301
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190315 06:52:45.436808 1 cluster_synced.go:1383  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1178890-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.1     11.5     17.8     46.1 write
		    1m0s        0         2142.8         1993.9      3.1     12.1     19.9     37.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2251.2         1998.1      3.3     11.0     15.2     28.3 write
		    1m2s        0         2246.1         2002.1      3.0     12.1     17.8     32.5 write
		    1m3s        0         1886.6         2000.3      3.1     12.1     17.8     33.6 write
		    1m4s        0            0.0         1969.0      0.0      0.0      0.0      0.0 write
		    1m5s        0            0.0         1938.7      0.0      0.0      0.0      0.0 write
		    1m6s        0            3.0         1909.4      2.9   2415.9   2415.9   2415.9 write
		    1m7s        0            0.0         1880.9      0.0      0.0      0.0      0.0 write
		    1m8s        0         2028.3         1883.1      3.1     12.6     19.9   4295.0 write
		    1m9s        0         2239.1         1888.2      3.0     12.1     17.8     33.6 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-16T07:07:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/04ef15974085e14f758b20c552a84052eac9fa2b

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1180753&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1180753-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp986828122, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp986828122/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190316 07:05:21.808974 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190316 07:05:21.814191 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1180753-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.1     13.1     21.0     39.8 write
		    1m0s        0         2116.9         1894.0      3.1     12.1     19.9     31.5 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2154.0         1898.3      3.0     12.6     19.9     48.2 write
		    1m2s        0         2196.0         1903.1      3.1     12.1     16.3     29.4 write
		    1m3s        0         2065.9         1905.7      3.1     13.6     21.0     35.7 write
		    1m4s        0         2118.1         1909.0      3.1     12.6     19.9     29.4 write
		    1m5s        0         2125.1         1912.3      3.1     12.6     19.9     33.6 write
		    1m6s        0         2118.0         1915.5      3.1     13.1     21.0     37.7 write
		    1m7s        0         2167.8         1919.2      3.1     12.6     18.9     28.3 write
		    1m8s        0         2140.1         1922.5      3.1     12.1     19.9     29.4 write
		    1m9s        0         2127.8         1925.4      3.1     11.5     17.8    167.8 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-18T07:26:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/c2939ec9a4f15b7fb8683a5805deeb241953e7aa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1182991&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1182991-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp150464588, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp150464588/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190318 07:24:42.476685 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190318 07:24:42.481393 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1182991-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.8     11.0     14.7     22.0 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2120.0         1953.5      3.5     11.0     17.8     35.7 write
		    1m2s        0         2142.9         1956.6      3.5     11.0     16.3     25.2 write
		    1m3s        0         2255.1         1961.3      3.3     10.5     14.7     25.2 write
		    1m4s        0         1999.9         1961.9      3.4     12.6     19.9     29.4 write
		    1m5s        0            0.0         1931.7      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1902.5      0.0      0.0      0.0      0.0 write
		    1m7s        0            0.0         1874.1      0.0      0.0      0.0      0.0 write
		    1m8s        0            0.0         1846.5      0.0      0.0      0.0      0.0 write
		    1m9s        0          296.0         1824.1      3.5     39.8   5100.3   5100.3 write
		   1m10s        0         2144.1         1828.6      3.3     11.0     17.8     27.3 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-18T14:08:48Z

SHA: https://github.com/cockroachdb/cockroach/commits/a512e390f7f2f2629f3f811bab5866c46e3e5713

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1183678&tab=buildLog

The test failed on provisional_201903122203_v19.1.0-beta.20190318:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1183678-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp849331758, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp849331758/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190318 14:05:40.514495 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190318 14:05:40.547286 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1183678-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.7     12.1     18.9     28.3 write
		     59s        0         2028.0         2057.2      3.4     11.5     21.0     65.0 write
		    1m0s        0         2011.0         2056.4      3.4     12.6     19.9     27.3 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0          866.9         2036.9      3.5     11.5     18.9     41.9 write
		    1m2s        0            0.0         2004.1      0.0      0.0      0.0      0.0 write
		    1m3s        0            0.0         1972.3      0.0      0.0      0.0      0.0 write
		    1m4s        0            0.0         1941.4      0.0      0.0      0.0      0.0 write
		    1m5s        0         1098.0         1928.5      3.4     12.6     26.2   4160.7 write
		    1m6s        0         2197.1         1932.5      3.3     11.5     15.7     25.2 write
		    1m7s        0         2198.2         1936.5      3.0     11.5     17.8     32.5 write
		    1m8s        0         2146.0         1939.6      3.0     13.1     21.0     32.5 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-19T07:29:35Z

SHA: https://github.com/cockroachdb/cockroach/commits/f95d45653df6be5587fb9887de241f50b6932000

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1185396&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1185396-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp402844805, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp402844805/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190319 07:27:27.636763 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190319 07:27:27.641711 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1185396-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.5     13.1     23.1     44.0 write
		     59s        0         1952.0         1944.0      3.5     13.6     21.0     44.0 write
		    1m0s        0         2080.1         1946.3      3.3     13.1     19.9     33.6 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2120.1         1949.1      3.3     11.5     17.8     32.5 write
		    1m2s        0         2150.8         1952.4      3.1     12.6     17.8     35.7 write
		    1m3s        0         2155.1         1955.6      3.1     12.6     19.9     30.4 write
		    1m4s        0         2137.9         1958.5      3.1     12.1     18.9     24.1 write
		    1m5s        0         2132.9         1961.1      3.0     13.1     21.0     31.5 write
		    1m6s        0         2091.1         1963.1      3.0     13.6     22.0     33.6 write
		    1m7s        0         2174.1         1966.3      3.0     12.1     18.9     30.4 write
		    1m8s        0         2086.0         1968.0      3.3     12.6     18.9     32.5 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-20T07:03:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/3a7ea2d8c9d4a3e0d97f8f106fcf95b3f03765ec

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1187480&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1187480-disk-full:1 returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp949853355, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp949853355/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190320 07:01:29.957136 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190320 07:01:29.962390 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1187480-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.5     14.2     21.0     39.8 write
		    1m0s        0         1994.7         1853.7      3.3     14.2     21.0     37.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1982.2         1855.8      3.1     13.6     23.1     44.0 write
		    1m2s        0         2041.1         1858.8      3.1     13.6     21.0     30.4 write
		    1m3s        0         1938.7         1860.1      3.3     14.2     25.2     37.7 write
		    1m4s        0         1924.0         1861.1      3.4     14.2     23.1     48.2 write
		    1m5s        0         1971.2         1862.8      3.1     13.6     24.1     46.1 write
		    1m6s        0         2066.1         1865.8      3.1     13.1     21.0     39.8 write
		    1m7s        0         2058.8         1868.7      3.1     13.1     21.0     37.7 write
		    1m8s        0         1994.0         1870.6      3.0     14.7     23.1     35.7 write
		    1m9s        0         2019.3         1872.7      3.1     12.6     19.9     44.0 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-22T07:14:19Z

SHA: https://github.com/cockroachdb/cockroach/commits/a200cea4368ec90aaee12337d7ab5f9ca555108f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1191939&tab=buildLog

The test failed on master:
	disk_full.go:73,cluster.go:1605,errgroup.go:57: node successfully started unexpectedly
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1191939-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		214         1929.9         1818.1      2.9     11.5     18.9     37.7 write
		E190322 07:11:44.269161 1 workload/cli/run.go:420  dial tcp 10.142.0.135:26257: connect: connection refused
		     57s      729         2060.9         1822.4      2.4      6.8     10.0     14.7 write
		E190322 07:11:45.274228 1 workload/cli/run.go:420  dial tcp 10.142.0.135:26257: connect: connection refused
		     58s     1244         2059.2         1826.5      2.4      7.1      9.4     15.2 write
		E190322 07:11:46.276859 1 workload/cli/run.go:420  dial tcp 10.142.0.135:26257: connect: connection refused
		     59s     1637         1590.9         1822.5      2.4      6.6     10.0     14.2 write
		    1m0s     1637            0.0         1792.1      0.0      0.0      0.0      0.0 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s     1637         1239.9         1783.1      3.1     13.6     56.6   4831.8 write
		    1m2s     1637         2064.2         1787.6      3.1     12.6     18.9     39.8 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-22T07:16:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/b5768aecd39461ab9a54e2e7db059a3fe8b00459

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1191957&tab=buildLog

The test failed on release-19.1:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1191957-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp756116129, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp756116129/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190322 07:14:13.903172 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190322 07:14:13.907853 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1191957-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.3     12.6     18.9     33.6 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2118.1         1985.4      3.3     12.1     17.8     29.4 write
		    1m2s        0         2169.0         1988.4      3.3     11.0     16.3     23.1 write
		    1m3s        0         2118.0         1990.5      3.3     12.1     17.8     41.9 write
		    1m4s        0         1703.8         1986.0      3.1     11.0     15.7     26.2 write
		    1m5s        0            0.0         1955.4      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1925.8      0.0      0.0      0.0      0.0 write
		    1m7s        0            0.0         1897.0      0.0      0.0      0.0      0.0 write
		    1m8s        0            0.0         1869.2      0.0      0.0      0.0      0.0 write
		    1m9s        0         1832.1         1868.6      3.5     12.1     27.3   4563.4 write
		   1m10s        0         2191.0         1873.2      3.1     11.0     16.8     31.5 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-23T07:12:43Z

SHA: https://github.com/cockroachdb/cockroach/commits/6cac063ae1cb578130afbafb2abf4035268a10c9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1194308&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1194308-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp829825313, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp829825313/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190323 07:10:35.224867 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190323 07:10:35.229631 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1194308-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.5     14.2     21.0     31.5 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1931.0         1827.0      3.4     13.6     19.9     37.7 write
		    1m2s        0         2013.0         1830.0      3.5     12.6     18.9     39.8 write
		    1m3s        0         2029.0         1833.1      3.4     12.6     17.8     29.4 write
		    1m4s        0         1731.7         1831.5      3.4     12.6     18.9     41.9 write
		    1m5s        0            0.0         1803.4      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1776.0      0.0      0.0      0.0      0.0 write
		    1m7s        0            0.0         1749.5      0.0      0.0      0.0      0.0 write
		    1m8s        0            0.0         1723.8      0.0      0.0      0.0      0.0 write
		    1m9s        0          362.1         1704.1      3.7     13.6   5100.3   5100.3 write
		   1m10s        0         2025.0         1708.6      3.1     13.1     19.9     41.9 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-24T07:09:34Z

SHA: https://github.com/cockroachdb/cockroach/commits/c59f5347d5424edb90575fb0fd50bad677953752

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1195732&tab=buildLog

The test failed on release-19.1:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1195732-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp586048745, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp586048745/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190324 07:07:09.829988 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190324 07:07:09.835116 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1195732-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.0     13.1     23.1    167.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2152.0         1886.9      3.0     12.6     18.9    838.9 write
		    1m2s        0         2054.0         1889.6      3.1     13.6     19.9     30.4 write
		    1m3s        0         2078.0         1892.6      3.1     12.6     22.0     44.0 write
		    1m4s        0         2039.9         1894.9      3.0     13.6     21.0     37.7 write
		    1m5s        0         2061.1         1897.5      3.0     13.1     19.9     30.4 write
		    1m6s        0         2125.8         1900.9      3.0     12.6     21.0     39.8 write
		    1m7s        0         2148.2         1904.6      3.1     12.1     19.9     39.8 write
		    1m8s        0         1989.0         1905.9      3.1     14.2     25.2     52.4 write
		    1m9s        0         2181.1         1909.9      3.3     11.5     17.8     28.3 write
		   1m10s        0         1994.9         1911.1      3.1     13.6     24.1    159.4 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-24T07:09:58Z

SHA: https://github.com/cockroachdb/cockroach/commits/5a746073c3f8ede851f37dd895cf1a91d6dcc3cf

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1195714&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1195714-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp236472365, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp236472365/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190324 07:07:55.803546 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190324 07:07:55.808053 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1195714-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.1     13.1     19.9     39.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2134.0         1914.9      3.1     12.1     19.9     37.7 write
		    1m2s        0         2123.1         1918.3      3.1     12.1     18.9     29.4 write
		    1m3s        0         2140.0         1921.8      3.1     12.1     18.9     30.4 write
		    1m4s        0         1838.7         1920.5      3.5     11.5     18.9     32.5 write
		    1m5s        0            0.0         1891.0      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         1862.3      0.0      0.0      0.0      0.0 write
		    1m7s        0            0.0         1834.5      0.0      0.0      0.0      0.0 write
		    1m8s        0            0.0         1807.5      0.0      0.0      0.0      0.0 write
		    1m9s        0         1960.1         1809.8      3.5     13.6     21.0   4160.7 write
		   1m10s        0         2031.9         1812.9      3.4     12.6     21.0     32.5 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-25T07:09:55Z

SHA: https://github.com/cockroachdb/cockroach/commits/7bc9ea5fbe0c0082fdcfd408245a79c62b00edd4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1197065&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1197065-disk-full:1 returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp509603001, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp509603001/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190325 07:07:42.223884 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190325 07:07:42.228925 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1197065-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.7     13.6     22.0     37.7 write
		    1m0s        0         1883.8         1832.6      3.5     14.2     23.1     37.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1963.2         1834.8      3.4     13.6     21.0     35.7 write
		    1m2s        0         2037.8         1838.1      3.3     12.6     19.9     41.9 write
		    1m3s        0         1989.3         1840.5      3.4     13.1     19.9     37.7 write
		    1m4s        0         2055.9         1843.8      3.3     12.6     21.0     31.5 write
		    1m5s        0         2019.0         1846.5      3.3     13.6     21.0     37.7 write
		    1m6s        0         2048.1         1849.6      3.4     12.6     19.9     41.9 write
		    1m7s        0         1990.9         1851.7      3.4     12.6     19.9     46.1 write
		    1m8s        0         2060.0         1854.7      3.4     12.6     18.9     26.2 write
		    1m9s        0         1992.9         1856.7      3.3     13.6     21.0     58.7 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-25T07:13:03Z

SHA: https://github.com/cockroachdb/cockroach/commits/ec89b45cea7e8a6dd92a9bfd60e0cc06842e06d8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1197083&tab=buildLog

The test failed on release-19.1:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1197083-disk-full:1 returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp039151246, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp039151246/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190325 07:10:39.274108 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190325 07:10:39.279994 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1197083-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.5     15.7     25.2     48.2 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1867.0         1782.4      3.4     16.3     24.1     35.7 write
		    1m2s        0         1861.9         1783.7      3.5     15.2     24.1     31.5 write
		    1m3s        0         1840.2         1784.6      3.5     15.7     25.2     52.4 write
		    1m4s        0          783.3         1769.0      2.9     41.9    192.9    385.9 write
		    1m5s        0         1422.1         1763.6      3.0     17.8    125.8    453.0 write
		    1m6s        0         1642.0         1761.8      2.9     15.2     52.4    151.0 write
		    1m7s        0         1234.0         1753.9      2.8     27.3    121.6    218.1 write
		    1m8s        0         1996.0         1757.5      3.0     14.7     23.1     41.9 write
		    1m9s        0         1917.1         1759.8      3.1     14.7     25.2     41.9 write
		   1m10s        0         1978.9         1762.9      3.0     14.2     25.2     56.6 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-26T07:00:27Z

SHA: https://github.com/cockroachdb/cockroach/commits/25398c010b2af75b11fed189680ea6b9645f0cf5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1199659&tab=buildLog

The test failed on master:
	disk_full.go:73,cluster.go:1605,errgroup.go:57: node successfully started unexpectedly
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1199659-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		255         1628.0         1752.4      2.8     11.0     17.8     39.8 write
		E190326 06:58:13.219205 1 workload/cli/run.go:420  dial tcp 10.142.0.124:26257: connect: connection refused
		     57s      610         1421.0         1746.6      2.5      7.9     10.5     18.9 write
		E190326 06:58:14.221348 1 workload/cli/run.go:420  dial tcp 10.142.0.124:26257: connect: connection refused
		     58s     1005         1580.1         1743.8      2.4      7.3     10.0     17.8 write
		E190326 06:58:15.223116 1 workload/cli/run.go:420  dial tcp 10.142.0.124:26257: connect: connection refused
		     59s     1390         1552.9         1740.5      2.5      7.1     10.0     21.0 write
		    1m0s     1390            0.0         1711.5      0.0      0.0      0.0      0.0 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s     1390         1757.0         1712.3      3.5     14.7     27.3   4831.8 write
		    1m2s     1390         1841.0         1714.3      3.5     14.2     22.0     37.7 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1214: Goexit() was called

cockroach-teamcity · 2019-03-27T07:09:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/3aadd20bbf0940ef65f8b2cdcda498401ba5d9c6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1201905&tab=buildLog

The test failed on release-19.1:
	disk_full.go:73,cluster.go:1605,errgroup.go:57: node successfully started unexpectedly
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1201905-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		4     12.6     21.0     32.5 write
		     52s        0         1882.0         1923.2      3.3     15.7     24.1     75.5 write
		     53s        0         1978.1         1924.2      3.3     13.6     21.0     44.0 write
		E190327 07:07:51.559314 1 workload/cli/run.go:420  EOF
		     54s      233         1850.8         1922.8      2.6     10.5     16.8     30.4 write
		E190327 07:07:52.560501 1 workload/cli/run.go:420  dial tcp 10.142.0.10:26257: connect: connection refused
		     55s      645         1647.1         1917.8      2.2      7.3     10.5     15.7 write
		E190327 07:07:53.561832 1 workload/cli/run.go:420  dial tcp 10.142.0.10:26257: connect: connection refused
		     56s      932         1161.9         1904.3      2.4      7.1      8.9     10.5 write
		     57s      932          193.0         1874.3      3.1     26.2   3489.7   3489.7 write
		     58s      932         1960.9         1875.8      3.3     14.2     23.1     33.6 write
		     59s      932         1891.9         1876.1      3.3     14.2     24.1     35.7 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1216: Goexit() was called

cockroach-teamcity · 2019-03-27T07:14:09Z

SHA: https://github.com/cockroachdb/cockroach/commits/ed21f6da64b022a3e1e550fad5850fdffe2a7d17

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1201887&tab=buildLog

The test failed on master:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1201887-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp860405672, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp860405672/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190327 07:11:45.171263 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190327 07:11:45.176265 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1201887-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2114.2         1962.6      3.4     12.1     17.8     31.5 write
		    1m2s        0         2240.9         1967.1      3.1     11.0     16.3     27.3 write
		    1m3s        0         2264.9         1971.8      3.0     11.5     16.3     26.2 write
		    1m4s        0         2163.9         1974.8      3.0     11.5     18.9     62.9 write
		    1m5s        0          598.9         1953.7      3.0     14.7     19.9     29.4 write
		    1m6s        0            0.0         1924.1      0.0      0.0      0.0      0.0 write
		    1m7s        0            0.0         1895.3      0.0      0.0      0.0      0.0 write
		    1m8s        0            0.0         1867.5      0.0      0.0      0.0      0.0 write
		    1m9s        0         1344.2         1859.9      3.3     14.2     35.7   4160.7 write
		   1m10s        0         2203.3         1864.8      3.0     11.5     17.8    117.4 write
		   1m11s        0         2126.8         1868.5      3.1     11.5     18.9     37.7 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1216: Goexit() was called

cockroach-teamcity · 2019-03-28T07:02:17Z

SHA: https://github.com/cockroachdb/cockroach/commits/17565100d1e7c66341e6db3e39bb66202958cb81

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1204567&tab=buildLog

The test failed on master:
	disk_full.go:73,cluster.go:1605,errgroup.go:57: node successfully started unexpectedly
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1204567-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   358         2458.6         1884.1      2.8     10.5     14.7     25.2 write
		E190328 06:59:53.057560 1 workload/cli/run.go:420  dial tcp 10.142.0.82:26257: connect: connection refused
		     57s      985         2506.2         1895.0      2.6      7.3     11.0     31.5 write
		E190328 06:59:54.060099 1 workload/cli/run.go:420  dial tcp 10.142.0.82:26257: connect: connection refused
		     58s     1628         2571.2         1906.7      2.5      8.1     12.1     24.1 write
		E190328 06:59:55.061171 1 workload/cli/run.go:420  dial tcp 10.142.0.82:26257: connect: connection refused
		     59s     2160         2153.8         1910.9      2.6      6.8     10.5     16.8 write
		    1m0s     2160          171.0         1881.9      3.5   1073.7   4563.4   4563.4 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s     2160         2009.1         1884.0      3.4     13.1     23.1     37.7 write
		    1m2s     2160         2050.9         1886.7      3.4     12.6     19.9     35.7 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1216: Goexit() was called

cockroach-teamcity · 2019-03-28T07:09:28Z

SHA: https://github.com/cockroachdb/cockroach/commits/a1e6e9decc9dec15a32bbb6d30efc67ca45a532a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1204585&tab=buildLog

The test failed on release-19.1:
	cluster.go:1193,disk_full.go:84,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1204585-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp870008538, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp870008538/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190328 07:07:51.065096 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1320
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190328 07:07:51.086931 1 cluster_synced.go:1402  command failed
		: exit status 1
	cluster.go:1267,disk_full.go:50,cluster.go:1605,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1204585-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.0     10.0     14.7     22.0 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2523.0         2279.7      2.9     10.0     14.7     27.3 write
		    1m2s        0         2335.2         2280.6      3.1     10.0     15.2     52.4 write
		    1m3s        0         2217.1         2279.5      3.5     10.5     16.8     28.3 write
		    1m4s        0         1322.7         2264.6      3.3     10.0     14.2     26.2 write
		    1m5s        0            0.0         2229.8      0.0      0.0      0.0      0.0 write
		    1m6s        0            0.0         2196.0      0.0      0.0      0.0      0.0 write
		    1m7s        0            0.0         2163.2      0.0      0.0      0.0      0.0 write
		    1m8s        0            0.0         2131.4      0.0      0.0      0.0      0.0 write
		    1m9s        0          566.1         2108.7      3.1     11.5   5368.7   5368.7 write
		   1m10s        0         1733.8         2103.3      3.0     11.5     19.9     60.8 write
		: signal: killed
	cluster.go:1626,disk_full.go:87,test.go:1216: Goexit() was called

cockroach-teamcity · 2019-03-28T19:35:30Z

SHA: https://github.com/cockroachdb/cockroach/commits/3aadd20bbf0940ef65f8b2cdcda498401ba5d9c6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1206925&tab=buildLog

The test failed on release-19.1:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1206925-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp065987847, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp065987847/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190328 19:31:55.683263 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190328 19:31:55.715653 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1206925-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		     14.2     24.1     37.7 write
		    1m2s        0         1998.0         1865.2      3.4     12.6     19.9     39.8 write
		    1m3s        0         1996.1         1867.3      3.4     13.6     21.0     31.5 write
		    1m4s        0         1968.0         1868.8      3.1     14.2     22.0     41.9 write
		    1m5s        0         1938.0         1869.9      3.1     14.7     23.1     39.8 write
		    1m6s        0         1988.0         1871.7      3.3     13.6     19.9     41.9 write
		    1m7s        0         2006.8         1873.7      3.0     14.7     23.1     37.7 write
		    1m8s        0         2001.2         1875.6      3.1     14.2     21.0     37.7 write
		    1m9s        0         1832.9         1875.0      3.4     15.7     24.1     41.9 write
		   1m10s        0         2066.1         1877.7      3.1     13.1     19.9     28.3 write
		   1m11s        0         1975.0         1879.1      3.3     13.6     21.0     33.6 write
		   1m12s        0         1948.0         1880.0      3.1     14.2     23.1     50.3 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

cockroach-teamcity · 2019-03-29T06:51:48Z

SHA: https://github.com/cockroachdb/cockroach/commits/d03a34e92d2ee558fb6aedb0709b733a1fab97f4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1207666&tab=buildLog

The test failed on master:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1207666-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp939507801, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp939507801/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190329 06:50:16.578999 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190329 06:50:16.583819 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1207666-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.4     13.6     22.0     41.9 write
		    1m0s        0         2031.1         1821.7      3.3     12.6     21.0     31.5 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2029.1         1825.1      3.3     13.1     19.9     35.7 write
		    1m2s        0         2011.8         1828.1      3.4     12.6     19.9     44.0 write
		    1m3s        0         2035.1         1831.4      3.3     12.1     19.9     33.6 write
		    1m4s        0         2096.0         1835.5      3.1     12.6     19.9     46.1 write
		    1m5s        0         2035.0         1838.6      3.3     12.6     21.0     37.7 write
		    1m6s        0         2103.9         1842.6      3.1     12.6     19.9     33.6 write
		    1m7s        0         2035.1         1845.5      3.3     13.1     19.9     28.3 write
		    1m8s        0         2046.9         1848.4      3.1     12.6     21.0     29.4 write
		    1m9s        0         1972.0         1850.2      3.1     14.2     24.1     46.1 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

tbg · 2019-03-29T11:35:04Z

Looks like this fails every night with the same error. I just looked at the test and... well I really think it's the test's fault, it plays fast and loose with starting processes and doesn't really track 100% well whether they actually stop

cockroach-teamcity · 2019-03-30T07:07:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/23f9707873abbd2de91a42055535529d7ff296ce

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1209900&tab=buildLog

The test failed on release-19.1:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1209900-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp836455097, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp836455097/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190330 07:06:07.487558 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190330 07:06:07.492544 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1209900-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.3     13.6     23.1     37.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         1879.0         1853.5      3.5     13.6     22.0     32.5 write
		    1m2s        0         1951.0         1855.1      3.4     13.1     22.0     32.5 write
		    1m3s        0         1960.2         1856.8      3.4     13.6     21.0     39.8 write
		    1m4s        0         1342.8         1848.7      3.5     15.2    159.4    167.8 write
		    1m5s        0         1104.0         1837.3      3.4     16.3    159.4    192.9 write
		    1m6s        0          895.0         1823.0      3.4     16.8    159.4    167.8 write
		    1m7s        0         1765.1         1822.1      3.4     15.7     71.3    302.0 write
		    1m8s        0         1989.1         1824.6      3.3     14.2     22.0     33.6 write
		    1m9s        0         1971.1         1826.7      3.3     13.6     21.0     39.8 write
		   1m10s        0         1886.0         1827.6      3.4     13.1     21.0    151.0 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

cockroach-teamcity · 2019-03-31T07:03:24Z

SHA: https://github.com/cockroachdb/cockroach/commits/a6b3c540b696002b2ed07036a657612995d6d1ab

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1211353&tab=buildLog

The test failed on master:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1211353-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp607111970, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp607111970/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190331 07:02:08.242182 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190331 07:02:08.255484 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1211353-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.4     16.3     24.1    218.1 write
		    1m0s        0         1971.9         1863.0      3.1     13.6     23.1     52.4 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2009.2         1865.4      3.3     13.1     21.0     35.7 write
		    1m2s        0         1974.9         1867.2      3.1     14.2     22.0     37.7 write
		    1m3s        0         1926.0         1868.1      3.1     14.2     21.0     39.8 write
		    1m4s        0         1945.0         1869.3      3.3     13.6     19.9     32.5 write
		    1m5s        0         1967.0         1870.8      3.3     13.6     19.9     39.8 write
		    1m6s        0         1688.6         1868.0      3.4     13.6     18.9     27.3 write
		    1m7s        0         1843.3         1867.7      3.1     15.7     28.3    176.2 write
		    1m8s        0         1999.8         1869.6      3.1     13.1     21.0     44.0 write
		    1m9s        0         2012.9         1871.7      3.3     12.6     21.0     31.5 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

cockroach-teamcity · 2019-04-01T04:36:54Z

SHA: https://github.com/cockroachdb/cockroach/commits/5921cf0dcc76548931cc85500c0fa2186a82142f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1212185&tab=buildLog

The test failed on release-19.1:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1212185-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp146857136, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp146857136/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190401 04:33:48.003237 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190401 04:33:48.036555 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1212185-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		     14.2     22.0     41.9 write
		    1m2s        0         2010.0         1860.5      3.1     13.1     21.0     58.7 write
		    1m3s        0         1985.0         1862.5      3.0     14.7     22.0     37.7 write
		    1m4s        0         1918.9         1863.4      3.0     15.7     26.2     56.6 write
		    1m5s        0         1879.0         1863.6      3.1     15.2     25.2     41.9 write
		    1m6s        0         1897.0         1864.1      3.0     15.7     26.2     37.7 write
		    1m7s        0         1854.1         1864.0      3.1     15.7     25.2     46.1 write
		    1m8s        0         1811.0         1863.2      3.1     16.3     30.4     71.3 write
		    1m9s        0         1913.0         1863.9      3.0     15.2     24.1     41.9 write
		   1m10s        0         1946.1         1865.1      3.0     15.2     25.2     41.9 write
		   1m11s        0         1936.0         1866.1      3.0     15.2     26.2     39.8 write
		   1m12s        0         2076.8         1869.0      2.9     13.1     19.9     35.7 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

cockroach-teamcity · 2019-04-01T06:51:15Z

SHA: https://github.com/cockroachdb/cockroach/commits/668162cc99e4f3198b663b1abfa51858eeb3ccb8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1212251&tab=buildLog

The test failed on master:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1212251-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp769438099, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp769438099/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190401 06:49:48.002216 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190401 06:49:48.006552 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1212251-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.4     12.1     21.0     32.5 write
		    1m0s        0         2228.0         1999.8      3.1     12.1     19.9     28.3 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2234.0         2003.6      3.1     11.5     17.8     37.7 write
		    1m2s        0         2228.9         2007.3      3.1     11.5     16.8     35.7 write
		    1m3s        0         2183.0         2010.0      3.1     11.5     17.8     33.6 write
		    1m4s        0         1779.0         2006.4      3.1     12.6     26.2    167.8 write
		    1m5s        0         1990.2         2006.2      3.0     13.6     24.1    159.4 write
		    1m6s        0         2088.8         2007.4      3.3     12.1     21.0    159.4 write
		    1m7s        0         2025.1         2007.7      3.1     12.6     23.1    151.0 write
		    1m8s        0         2284.0         2011.8      3.0     11.5     16.8     27.3 write
		    1m9s        0         2056.0         2012.4      3.3     12.6     19.9     41.9 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

cockroach-teamcity · 2019-04-01T07:06:55Z

SHA: https://github.com/cockroachdb/cockroach/commits/a039a93a5cc6eb3f395ceb6f7dc8030985dccc29

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=disk-full PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1212269&tab=buildLog

The test failed on release-19.1:
	cluster.go:1219,disk_full.go:84,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1212269-disk-full:1 --encrypt returned:
		stderr:
		
		stdout:
		t how to secure your cluster: https://www.cockroachlabs.com/docs/v19.1/secure-a-cluster.html
		*
		*
		* ERROR: could not cleanup temporary directories from record file: could not lock temporary directory /mnt/data1/cockroach/cockroach-temp375245150, may still be in use: IO error: While lock file: /mnt/data1/cockroach/cockroach-temp375245150/TEMP_DIR.LOCK: Resource temporarily unavailable
		*
		Failed running "start"
		E190401 07:05:21.963895 1 cli/error.go:229  exit status 1
		Error: exit status 1
		Failed running "start"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func7
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:397
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1382
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333: 
		I190401 07:05:21.969451 1 cluster_synced.go:1464  command failed
		: exit status 1
	cluster.go:1293,disk_full.go:50,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1212269-disk-full:5 -- ./workload run kv --tolerate-errors --init --read-percent=0 --concurrency=10 --duration=2m {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   3.1     15.2     26.2     46.1 write
		    1m0s        0         1943.3         1869.6      3.1     13.6     23.1    167.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1m1s        0         2018.9         1872.1      3.1     13.6     21.0     41.9 write
		    1m2s        0         2040.1         1874.8      3.0     13.1     21.0     32.5 write
		    1m3s        0         2083.9         1878.1      2.9     14.2     23.1     37.7 write
		    1m4s        0         2079.9         1881.3      2.9     13.1     22.0     44.0 write
		    1m5s        0         2120.2         1884.9      2.9     13.1     21.0     27.3 write
		    1m6s        0         2066.0         1887.7      2.9     14.2     23.1     39.8 write
		    1m7s        0         2011.9         1889.5      3.1     13.6     21.0     37.7 write
		    1m8s        0         2089.2         1892.5      2.9     13.1     22.0     37.7 write
		    1m9s        0         2088.8         1895.3      3.0     12.6     19.9     33.6 write
		: signal: killed
	cluster.go:1652,disk_full.go:87,test.go:1223: Goexit() was called

tbg · 2019-04-01T11:23:28Z

It looks like the process never dies when the out of disk condition is created. I don't know what it's doing. The debug zip has stacks but nothing there sticks out. Note that the log for n1 (the node running out of disk) has an error from the compactor noting an out of disk condition, so creating the ballast file must have worked. I can imagine that things just got "stuck" while out of disk, and that they recovered when the ballast was recreated.

disk-full.zip

This needs someone to catch this in action to take a closer look. I didn't see any commits on Mar 2nd that stuck out to me as being able to cause this. The test is also not verifying that the process actually crashes. It calls m.ExpectDeath() but that doesn't verify that there is a crash, it simply would tolerate one.

I will skip this test until it is investigated.

It fails constantly and is in need of an investigation. Initial notes here: cockroachdb#35328 (comment) Touches cockroachdb#35328. Release note: None

andreimatei · 2019-06-14T20:08:19Z

Some suspicion of a RocksDB bug around here: https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1560537597006900

petermattis · 2020-09-03T13:03:48Z

@itsbilal Can you take a look at this? Perhaps this test works under Pebble. And if it doesn't, it would be good to fix whatever is preventing it from working.

53800: server: defer gossip start to when absolutely needed r=irfansharif a=irfansharif This was pulled out of #52526 to keep the diff there focussed on the introduction of the RPC (and to see if that alone shaked out any failures). That change lets us make this one, were we can defer gossip start until right before we start up Node. Release justification: low risk, high benefit change to existing functionality Release note: None 53846: sql: fix inflated "overhead" in statement timings r=solongordon a=solongordon Fixes #40675 Fixes #50108 Release justification: Low-risk, high-reward fix to existing functionality. Release note (admin ui change): In some cases, the Execution Stats page would show a confusingly high Overhead latency for a statement. This could happen due to multiple statements being parsed together or due to statement execution being retried. To avoid this, we stop considering the time between when parsing ends and execution begins when determining service latency. 53906: roachtest: Unskip disk-full roachtest r=tbg a=itsbilal The disk-full roachtest no longer fails now that Pebble is the default storage engine. Remove the skipped marker. Fixes #35328. Release justification: Roachtest-only change. Release note: None. 53907: geos: bump to include seg fault fix r=sumeerbhola a=otan Resolves #53254. Release justification: bug fix to new functionality Release note: None Co-authored-by: irfan sharif <[email protected]> Co-authored-by: Solon Gordon <[email protected]> Co-authored-by: Bilal Akhtar <[email protected]> Co-authored-by: Oliver Tan <[email protected]>

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Mar 2, 2019

nvanbenschoten mentioned this issue Mar 21, 2019

teamcity: failed test: gossip/restart #35857

Closed

tbg added the A-disaster-recovery label Mar 29, 2019

tbg changed the title ~~roachtest: disk-full failed~~ roachtest: disk-full failed [skipped] Apr 1, 2019

tbg added a commit to tbg/cockroach that referenced this issue Apr 1, 2019

roachtest: skip disk_full

be793f7

It fails constantly and is in need of an investigation. Initial notes here: cockroachdb#35328 (comment) Touches cockroachdb#35328. Release note: None

nvanbenschoten added the E-starter Might be suitable for a starter project for new employees or team members. label May 1, 2019

darinpp self-assigned this May 28, 2019

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

petermattis assigned itsbilal Sep 3, 2020

itsbilal mentioned this issue Sep 3, 2020

roachtest: Unskip disk-full roachtest #53906

Merged

craig bot closed this as completed in d51519e Sep 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: disk-full failed [skipped] #35328

roachtest: disk-full failed [skipped] #35328

cockroach-teamcity commented Mar 2, 2019

cockroach-teamcity commented Mar 3, 2019

cockroach-teamcity commented Mar 5, 2019

cockroach-teamcity commented Mar 6, 2019

cockroach-teamcity commented Mar 12, 2019

cockroach-teamcity commented Mar 13, 2019

cockroach-teamcity commented Mar 15, 2019

cockroach-teamcity commented Mar 16, 2019

cockroach-teamcity commented Mar 18, 2019

cockroach-teamcity commented Mar 18, 2019

cockroach-teamcity commented Mar 19, 2019

cockroach-teamcity commented Mar 20, 2019

cockroach-teamcity commented Mar 22, 2019

cockroach-teamcity commented Mar 22, 2019

cockroach-teamcity commented Mar 23, 2019

cockroach-teamcity commented Mar 24, 2019

cockroach-teamcity commented Mar 24, 2019

cockroach-teamcity commented Mar 25, 2019

cockroach-teamcity commented Mar 25, 2019

cockroach-teamcity commented Mar 26, 2019

cockroach-teamcity commented Mar 27, 2019

cockroach-teamcity commented Mar 27, 2019

cockroach-teamcity commented Mar 28, 2019

cockroach-teamcity commented Mar 28, 2019

cockroach-teamcity commented Mar 28, 2019

cockroach-teamcity commented Mar 29, 2019

tbg commented Mar 29, 2019

cockroach-teamcity commented Mar 30, 2019

cockroach-teamcity commented Mar 31, 2019

cockroach-teamcity commented Apr 1, 2019

cockroach-teamcity commented Apr 1, 2019

cockroach-teamcity commented Apr 1, 2019

tbg commented Apr 1, 2019

andreimatei commented Jun 14, 2019

petermattis commented Sep 3, 2020

roachtest: disk-full failed [skipped] #35328

roachtest: disk-full failed [skipped] #35328

Comments

cockroach-teamcity commented Mar 2, 2019

cockroach-teamcity commented Mar 3, 2019

cockroach-teamcity commented Mar 5, 2019

cockroach-teamcity commented Mar 6, 2019

cockroach-teamcity commented Mar 12, 2019

cockroach-teamcity commented Mar 13, 2019

cockroach-teamcity commented Mar 15, 2019

cockroach-teamcity commented Mar 16, 2019

cockroach-teamcity commented Mar 18, 2019

cockroach-teamcity commented Mar 18, 2019

cockroach-teamcity commented Mar 19, 2019

cockroach-teamcity commented Mar 20, 2019

cockroach-teamcity commented Mar 22, 2019

cockroach-teamcity commented Mar 22, 2019

cockroach-teamcity commented Mar 23, 2019

cockroach-teamcity commented Mar 24, 2019

cockroach-teamcity commented Mar 24, 2019

cockroach-teamcity commented Mar 25, 2019

cockroach-teamcity commented Mar 25, 2019

cockroach-teamcity commented Mar 26, 2019

cockroach-teamcity commented Mar 27, 2019

cockroach-teamcity commented Mar 27, 2019

cockroach-teamcity commented Mar 28, 2019

cockroach-teamcity commented Mar 28, 2019

cockroach-teamcity commented Mar 28, 2019

cockroach-teamcity commented Mar 29, 2019

tbg commented Mar 29, 2019

cockroach-teamcity commented Mar 30, 2019

cockroach-teamcity commented Mar 31, 2019

cockroach-teamcity commented Apr 1, 2019

cockroach-teamcity commented Apr 1, 2019

cockroach-teamcity commented Apr 1, 2019

tbg commented Apr 1, 2019

andreimatei commented Jun 14, 2019

petermattis commented Sep 3, 2020