Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: backupTPCC failed #86289

Closed
cockroach-teamcity opened this issue Aug 17, 2022 · 8 comments · Fixed by #86809
Closed

roachtest: backupTPCC failed #86289

cockroach-teamcity opened this issue Aug 17, 2022 · 8 comments · Fixed by #86809
Assignees
@msbutler
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 17, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.backupTPCC failed with artifacts on master @ b173a16715e71e94115820374da1eb350b3b459d:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf2\x88"/0,0#0,SET, "stats/\xf2\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf2\x88"/0,0#0,SET, "stats/\xf2\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

Jira issue: CRDB-18668
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 17, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Aug 17, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ 5c2c62ecf1bea60c807edc6b4da22d900ad4ae03:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf2\x88"/0,0#0,SET, "stats/\xf2\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf2\x88"/0,0#0,SET, "stats/\xf2\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ a0d8839aa6164af81a9ebb140147d3baf5321287:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ cb55144cdec54d2a70f074ad64b4eca5e6c6891a:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:1006,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf2\x88"/0,0#0,SET, "stats/\xf2\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:1006
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf2\x88"/0,0#0,SET, "stats/\xf2\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ e6a7dc2f8ee39549e186bd05626c4c375b76fd04:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ aaf50e920ceff3c2863ab96b9e3614b8434b70a8:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:1006,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:1006
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ 80c274877a917580af62be6eb0cd48c8c7ae9c08:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ 003c0360de8b64319b5f0f127b99be91dbdca8a3:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@msbutler msbutler added release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 24, 2022
@msbutler msbutler mentioned this issue Aug 24, 2022
Closed
@msbutler msbutler mentioned this issue Aug 24, 2022
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Aug 24, 2022
@msbutler
backupccl: set kv.bulkio.write_metadata_sst.enabled to default false
7ea3af4 
This patch sets write_metadata_sst cluster setting to false in prep for the
22.2 branch cut, as there's additional worked required before this feature gets
used in production. Unit tests may continue to write the MetadataSST because of
a new MetamorphicTestBool.

Setting this to false will also stop the roachtest in cockroachdb#86289 from consistently
failing due to cockroachdb#86806.

Fixes cockroachdb#86289

Release note: none

Release justification: prevents using an experimental feature by default
@cockroach-teamcity
Copy link
Member Author

roachtest.backupTPCC failed with artifacts on master @ 524fd14da3fefcd849f44a835cc5f88f5dbdadcc:

test artifacts and logs in: /artifacts/backupTPCC/run_1
	monitor.go:127,backup.go:975,test_runner.go:896: monitor failure: monitor task failed: pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerBackup.func4
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/backup.go:975
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\xf3\x88"/0,0#0,SET, "stats/\xf3\x88"/0,0#0,SET
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Aug 25, 2022
@aliher1911 @tbg
@maryliag @erikgrinaker @msbutler
Merge #86608 #86645 #86679 #86748 #86809
54bc65f 
86608: batcheval: add latch key protecting range key stats update r=erikgrinaker a=aliher1911

Previously GC needed to get a read latch with max timestamp to
ensure that range tombstones are not modified during GC. This
is causing all writers to get stuck in queue while GC is validating
request and removing range tombstone.
This commit adds a dedicated latch key
LocalRangeRangeTombstoneStatsUpdateLockSuffix to address the problem.
All range tombstone writers obtain this read latch on top of the write
latches for the ranges they are interested to update.
GC on the other hand will obtain write latch on that key. This
approach allows point writers to proceed during GC, but will block new
range tombstones from being written. Non conflicting writes of range
tombstones could still proceed since their write latch ranges don't
overlap.

Release justification: this is a safe change as range tombstone
behaviour is not enabled yet and the change is needed to address
potential performance regressions.

Release note: None

86645: kvserver: log when raft send/recv queue fills up r=pavelkalinnikov a=tbg

Inspired by https://github.com/cockroachlabs/support/issues/1770.

If either the raft send or receive queue fills up, wide-spread outages
can occur as replication progress stalls. We have metrics that can
indicate this, but straightforward logging is also appropriate to direct
attention to the fact, which this commit achieves.

Touches #79755

Release justification: important logging improvement
Release note: None


86679: server,ui: show internal stats with new cluster setting r=maryliag a=maryliag

Previously, we were not showing internal results on
fingerprint options on SQL Activity.
A new cluster setting created `sql.stats.response.show_internal`
can be set to `true` and internal statistics will be
displayed on SQL Activity page.

Fixes #79547

https://www.loom.com/share/1b89ba99a7c247edadb5c8b0d127755c

Release justification: low risk, high benefit change
Release note (sql change): New cluster setting
`sql.stats.response.show_internal` with default value of `false`
can be set to true, to display information about internal stats
on SQL Activity page, with fingerprint option.

86748: storage: rename `MVCCRangeKeyStack.FirstAbove/Below` r=tbg a=erikgrinaker

This patch renames `FirstAbove/Below` to `FirstAtOrAbove/Below`, for
clarity.

Release justification: bug fixes and low-risk updates to new functionality

Release note: None

86809: backupccl: set kv.bulkio.write_metadata_sst.enabled to default false r=stevendanna a=msbutler

This patch sets write_metadata_sst cluster setting to false in prep for the
22.2 branch cut, as there's additional worked required before this feature gets
used in production.

Setting this to false will also stop the roachtest in #86289 from consistently
failing due to #86806.

Fixes #86289

Release note: none

Release justification: prevents using an experimental feature by default

Co-authored-by: Oleg Afanasyev <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: Marylia Gutierrez <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: Michael Butler <[email protected]>
@craig craig bot closed this as completed in f74435a Aug 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
22.2
Development

Successfully merging a pull request may close this issue.

backupccl: set kv.bulkio.write_metadata_sst.enabled to default false msbutler/cockroach
2 participants
@cockroach-teamcity @msbutler