Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: drop/tpcc/w=100,nodes=9 failed #56040

Closed
cockroach-teamcity opened this issue Oct 28, 2020 · 4 comments · Fixed by #57274
Closed

roachtest: drop/tpcc/w=100,nodes=9 failed #56040

cockroach-teamcity opened this issue Oct 28, 2020 · 4 comments · Fixed by #57274
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).drop/tpcc/w=100,nodes=9 failed on master@70fb9a54d6c4d7b3c1d8ad93bb399eefef8f5d04:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/drop/tpcc/w=100_nodes=9/run_1
	drop.go:92,cluster.go:2626,errgroup.go:57: Node 4 space used: 176 MiB less than 256 MiB

	cluster.go:2648,drop.go:157,drop.go:178,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.registerDrop.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:157
		  | main.registerDrop.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:178
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
More

Artifacts: /drop/tpcc/w=100,nodes=9

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 28, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 28, 2020
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 31, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).drop/tpcc/w=100,nodes=9 failed on master@9ec0151e0079c51afeec2f3f75a4747ce80a615b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/drop/tpcc/w=100_nodes=9/run_1
	drop.go:92,cluster.go:2626,errgroup.go:57: Node 5 space used: 59 MiB less than 256 MiB

	cluster.go:2648,drop.go:157,drop.go:178,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.registerDrop.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:157
		  | main.registerDrop.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:178
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
More

Artifacts: /drop/tpcc/w=100,nodes=9
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).drop/tpcc/w=100,nodes=9 failed on master@25edeb8d6c2cefca0c206f86553bc75569f0f2d9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/drop/tpcc/w=100_nodes=9/run_1
	drop.go:92,cluster.go:2626,errgroup.go:57: Node 1 space used: 244 MiB less than 256 MiB

	cluster.go:2648,drop.go:157,drop.go:178,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.registerDrop.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:157
		  | main.registerDrop.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:178
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
More

Artifacts: /drop/tpcc/w=100,nodes=9
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten
Copy link
Member

This is almost certainly fallout from the switch from RESTORE to IMPORT for TPC-C. Now that we no longer restore a fixed-size fixture, this assertion is much harder to make. I'll weaken it.

@nvanbenschoten nvanbenschoten mentioned this issue Dec 1, 2020
Merged
@nvanbenschoten nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Dec 1, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).drop/tpcc/w=100,nodes=9 failed on master@677f6f89f97d492f5eb443fcdb326d5695ddc5d7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/drop/tpcc/w=100_nodes=9/run_1
	drop.go:92,cluster.go:2626,errgroup.go:57: Node 4 space used: 237 MiB less than 256 MiB

	cluster.go:2648,drop.go:157,drop.go:178,test_runner.go:760: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.registerDrop.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:157
		  | main.registerDrop.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:178
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5646
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
More

Artifacts: /drop/tpcc/w=100,nodes=9
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

craig bot pushed a commit that referenced this issue Dec 3, 2020
@jayshrivastava @TheSamHuang
@nvanbenschoten @knz
Merge #56379 #57034 #57272 #57274 #57454
ac6463f 
56379: workload/schemachange: improve error screening r=ajwerner a=jayshrivastava

workload/schemachange: improve error screening

Several ops will be updated to screen for errors (#56119):
- dropTable
- dropView
- createSequence
- renameSequence
- dropSequence
- insertRow

Sequences were also updated to have more interesting cases: non-existing sequences can randomly be returned and sequences can be owned by columns.

Furthermore, this change fixes a bug where non-existing tables were getting returned instead of existing ones.

Finally, error screening logic is updated to ignore transaction retry errors. This fixes one of the issues in #56230.

57034: cli: add reset-quorum command r=knz a=TheSamHuang

This adds `reset-quorum` as a command in the debug CLI. This command utilizes the functionality added in #56333 to recover unavailable ranges where all replicas are lost. 

57272: roachtest: Add missing gossiped StoreDescriptor to expected relocate errors r=nvanbenschoten a=nvanbenschoten

Closes #57191.

This seems possible shortly after the cluster starts up, if gossip hasn't propagated by the time the `ALTER TABLE _ EXPERIMENTAL_RELOCATE _` statement is run.

57274: roachtest: remove initial disk space check from drop test r=nvanbenschoten a=nvanbenschoten

Fixes #56040.

Will need to be backported.

This assertion seemed like a good idea, but it was repeatedly (dec148a)
a source of flakiness when fixtures changed. Now that we're using IMPORT
for TPC-C, the check is even harder to get right without making it so
small as to be useless. It doesn't seem to be worth the instability, so
remove it.

57454: util/log: de-flake TestSecondaryGC and some SQL tests r=irfansharif a=knz

Fixes #57442
Fixes #57444 

See individual commits for details.

Co-authored-by: Jayant Shrivastava <[email protected]>
Co-authored-by: Sam Huang <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Raphael 'kena' Poss <[email protected]>
@craig craig bot closed this as completed in 187198a Dec 3, 2020
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 3, 2020
@nvanbenschoten
roachtest: remove initial disk space check from drop test
f93fd05 
Fixes cockroachdb#56040.

Will need to be backported.

This assertion seemed like a good idea, but it was repeatedly (dec148a)
a source of flakiness when fixtures changed. Now that we're using IMPORT
for TPC-C, the check is even harder to get right without making it so
small as to be useless. It doesn't seem to be worth the instability, so
remove it.
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 3, 2020
@nvanbenschoten
roachtest: remove initial disk space check from drop test
6a8c69a 
Fixes cockroachdb#56040.

Will need to be backported.

This assertion seemed like a good idea, but it was repeatedly (dec148a)
a source of flakiness when fixtures changed. Now that we're using IMPORT
for TPC-C, the check is even harder to get right without making it so
small as to be useless. It doesn't seem to be worth the instability, so
remove it.
This was referenced Dec 3, 2020
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
20.2
Development

Successfully merging a pull request may close this issue.

roachtest: remove initial disk space check from drop test nvanbenschoten/cockroach
2 participants
@nvanbenschoten @cockroach-teamcity