Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpccbench/nodes=3/cpu=4 failed #36325

Closed
cockroach-teamcity opened this issue Mar 29, 2019 · 3 comments
Closed

roachtest: tpccbench/nodes=3/cpu=4 failed #36325

cockroach-teamcity opened this issue Mar 29, 2019 · 3 comments
Assignees
@tbg
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/3aadd20bbf0940ef65f8b2cdcda498401ba5d9c6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1206925&tab=buildLog

The test failed on release-19.1:
	cluster.go:953,context.go:90,cluster.go:942,test.go:1224: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1206925-tpccbench-nodes-3-cpu-4 --oneshot --ignore-empty-nodes: exit status 1
@cockroach-teamcity cockroach-teamcity added this to the 19.1 milestone Mar 29, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Mar 29, 2019
@tbg
Copy link
Member

tbg commented Mar 29, 2019

This has fired for the other tpccbench tests too, gonna wait for tomorrow to see the output. Maybe it just gets confused. tpccbench probably doesn't have the cluster running at the end. Hmm. I looked and it does. Maybe I'm missing something.

tbg added a commit to tbg/cockroach that referenced this issue Mar 29, 2019
@tbg
roachtest: don't stop nodes at end of tpccbench
05d48e9 
Fixes cockroachdb#36325.
Touches cockroachdb#36277.
Touches cockroachdb#32135.

Release note: None
@tbg
Copy link
Member

tbg commented Mar 29, 2019

Test harness shuts down nodes. PR incoming

@tbg tbg mentioned this issue Mar 29, 2019
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/d03a34e92d2ee558fb6aedb0709b733a1fab97f4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1207666&tab=buildLog

The test failed on master:
	cluster.go:953,context.go:90,cluster.go:942,test.go:1224: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1207666-tpccbench-nodes-3-cpu-4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		2: dead
		3: dead
		Error:  3: dead

craig bot pushed a commit that referenced this issue Mar 29, 2019
@tbg
Merge #36328 #36329
377d1dc 
36328: storage: skip TestStoreRangeRemoveDead, TestLeaseNotUsedAfterRestart r=andreimatei,nvanbenschoten a=tbg

We need to investigate both (in particular the former), but for now
minimize disruption.

Release note: None

36329: roachtest: don't stop nodes at end of tpc{c,h}bench r=andreimatei,nvanbenschoten a=tbg

Fixes #36325.
Touches #36277.
Touches #32135.
Fixes #36022.

Release note: None

Co-authored-by: Tobias Schottdorf <[email protected]>
@craig craig bot closed this as completed in #36329 Mar 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tbg tbg

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.1
Development

No branches or pull requests
2 participants
@tbg @cockroach-teamcity