Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: splits/load/uniform/nodes=3 failed #42903

Closed
cockroach-teamcity opened this issue Dec 3, 2019 · 17 comments · Fixed by #43078
Closed

roachtest: splits/load/uniform/nodes=3 failed #42903

cockroach-teamcity opened this issue Dec 3, 2019 · 17 comments · Fixed by #43078
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/d3574ad671bd3631e780510235485681720c2b8f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1622074&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191203-1622074/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Dec 3, 2019
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Dec 3, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1da69d917105a0280aad10e86a7ee8eb2059cc92

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1623285&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=provisional_201912031738_v20.1.0-alpha20191209, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191203-1623285/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 2 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ed717cbaf741e3a32c76db25b16a59dc2a8221d7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1624103&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191204-1624103/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

@tbg
Copy link
Member

tbg commented Dec 4, 2019

There's reason to believe fad2024 broke this.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c5647e9ab0282e182805a9d1a44259790bea8558

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1626316&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191205-1626316/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 3 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c50e3f43a5ccb3ed2700af002a25c10f59e877a5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1626334&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191205-1626334/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 3 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e1c1a9ae765730782f3adc2298a76d6a07ad055d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1626298&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191205-1626298/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e4fa0b8b8e674d19c7957f03ca3a2d1f716f1f1d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1629591&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191206-1629591/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/28f216e1bd53da872a759a98779144c7f70f33a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1629573&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191206-1629573/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/60412eb85271ecd1539971fcc6ea3bf11f1ca7a6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1629609&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191206-1629609/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/9ca6d845aa87e6f2d629b06b884df2498f203d4f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1631719&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191207-1631719/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 3 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/5224df9c8bd9a36dfde24ae3abe7b7e42a5d9660

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1631737&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191207-1631737/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 2 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/edc18a6fc4bb4fbf118f4fc6c98e5642dc6adae7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1631701&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191207-1631701/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 2 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c85d435c2caa7a3066b1affdcc34b7ed6a6f3653

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1633173&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191208-1633173/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 2 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/9d697080f227324220c09d1b63da7bbf969133ac

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=splits/load/uniform/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1633532&tab=artifacts#/splits/load/uniform/nodes=3

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191209-1633532/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:697: kv.kv has 2 ranges, expected between 11 and 2147483647 splits

@cockroach-teamcity
Copy link
Member Author

(roachtest).splits/load/uniform/nodes=3 failed on provisional_201912092102_v19.2.2@c85d435c2caa7a3066b1affdcc34b7ed6a6f3653:

The test failed on branch=provisional_201912092102_v19.2.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191209-1634692/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:700: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

details

Artifacts: /splits/load/uniform/nodes=3

make stressrace TESTS=splits/load/uniform/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).splits/load/uniform/nodes=3 failed on release-19.1@6e95a8334dea98fdfa93fdc2e981935947089e32:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191210-1634885/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:700: kv.kv has 1 ranges, expected between 11 and 2147483647 splits

details

Artifacts: /splits/load/uniform/nodes=3

make stressrace TESTS=splits/load/uniform/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).splits/load/uniform/nodes=3 failed on master@734e357412dadafcd6084b8eab8e251e44e86b4a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191210-1634867/splits/load/uniform/nodes=3/run_1
	cluster.go:2163,split.go:205,split.go:50,test_runner.go:700: kv.kv has 3 ranges, expected between 11 and 2147483647 splits

details

Artifacts: /splits/load/uniform/nodes=3

make stressrace TESTS=splits/load/uniform/nodes=3 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues

tbg added a commit to tbg/cockroach that referenced this issue Dec 10, 2019
Fixes cockroachdb#42903.

Load based splitting would previously not split when EnsureSafeSplitKey
returns an error. However such an error only meant that the key in
question wasn't even a SQL key, which is the case most of the time,
at least in simple workloads such as kv (the SQL layer will send
single-element scans, but without specifying the column family).

No release note since not released.

Release note: None
craig bot pushed a commit that referenced this issue Dec 10, 2019
43078: decider: ignore EnsureSafeSplitKey on error r=nvanbenschoten a=tbg

Fixes #42903.

Load based splitting would previously not split when EnsureSafeSplitKey
returns an error. However such an error only meant that the key in
question wasn't even a SQL key, which is the case most of the time,
at least in simple workloads such as kv (the SQL layer will send
single-element scans, but without specifying the column family).

No release note since not released.

Release note: None

Co-authored-by: Tobias Schottdorf <[email protected]>
@craig craig bot closed this as completed in 1d5eb7c Dec 10, 2019
tbg added a commit to tbg/cockroach that referenced this issue Dec 10, 2019
Fixes cockroachdb#42903.

Load based splitting would previously not split when EnsureSafeSplitKey
returns an error. However such an error only meant that the key in
question wasn't even a SQL key, which is the case most of the time,
at least in simple workloads such as kv (the SQL layer will send
single-element scans, but without specifying the column family).

No release note since not released.

Release note: None
tbg added a commit to tbg/cockroach that referenced this issue Dec 10, 2019
Fixes cockroachdb#42903.

Load based splitting would previously not split when EnsureSafeSplitKey
returns an error. However such an error only meant that the key in
question wasn't even a SQL key, which is the case most of the time,
at least in simple workloads such as kv (the SQL layer will send
single-element scans, but without specifying the column family).

No release note since not released.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants