Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage/metamorphic: TestPebbleEquivalence failed #72261

Closed
cockroach-teamcity opened this issue Oct 30, 2021 · 2 comments · Fixed by #72288
Closed

storage/metamorphic: TestPebbleEquivalence failed #72261

cockroach-teamcity opened this issue Oct 30, 2021 · 2 comments · Fixed by #72288
Assignees
@itsbilal
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

storage/metamorphic.TestPebbleEquivalence failed with artifacts on master @ db229cad354c2776ea35c9dc4f78e6ce66b9437c:

Fatal error:

panic: min timestamp hint set without max timestamp hint [recovered]
	panic: min timestamp hint set without max timestamp hint

Stack:

goroutine 145 [running]:
testing.tRunner.func1.2(0x12d8900, 0x1690a30)
	/usr/local/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc000103b00)
	/usr/local/go/src/testing/testing.go:1146 +0x4b6
panic(0x12d8900, 0x1690a30)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).init(0xc00067ac60, 0x16c71e0, 0xc00075c480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0005b0790, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:157 +0x92f
github.com/cockroachdb/cockroach/pkg/storage.newPebbleIterator(0x16c71e0, 0xc00075c480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0005b0790, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:90 +0xd8
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).NewMVCCIterator(0xc0006e0000, 0x1, 0x0, 0x0, 0x0, 0x0, 0xc0005b0790, 0x8, 0x8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble.go:861 +0x1e5
github.com/cockroachdb/cockroach/pkg/storage.NewMVCCIncrementalIterator(0x16d6b90, 0xc0006e0000, 0x1, 0xc0005b0790, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc_incremental_iterator.go:177 +0x235
github.com/cockroachdb/cockroach/pkg/storage.MVCCClearTimeRange(0x16cbab0, 0xc00019e010, 0x7f4ad41c59d8, 0xc0006e0000, 0xc00275d860, 0xc0005b0760, 0xd, 0xd, 0xc0005b0790, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2269 +0x298
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.mvccClearTimeRangeOp.run(0xc000001c80, 0x14bdb1f, 0x6, 0xc0005b0760, 0xd, 0xd, 0xc0005b0790, 0x8, 0x8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/operations.go:308 +0x1c5
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.(*metaTestRunner).generateAndRun(0xc000001c80, 0x186a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/generator.go:303 +0x142
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.runMetaTestForEngines(0x16cbab0, 0xc00019e010, 0xc000103b00, 0x1, 0x69a6524fff68a54f, 0x0, 0x0, 0x0, 0x16a95a0, 0xc000010018, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/meta_test.go:89 +0x465
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.runMetaTest.func2(0xc000103b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/meta_test.go:155 +0x4b8
testing.tRunner(0xc000103b00, 0xc0000a2f30)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3
Log preceding fatal error 

=== RUN   TestPebbleEquivalence
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestPebbleEquivalence581982961
    test_log_scope.go:80: use -show-logs to present logs inline
first run output file: /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence623417244/output.meta
--- FAIL: TestPebbleEquivalence (1.36s)
=== RUN   TestPebbleEquivalence/engine/standard=0
check file = 
output file = /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence623417244/output.meta
store path = /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence_engine_standard=0478329646/store
    --- FAIL: TestPebbleEquivalence/engine/standard=0 (1.35s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=4

/cc @cockroachdb/storage itsbilal

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Oct 30, 2021
@blathers-crl blathers-crl bot added the T-storage Storage Team label Oct 30, 2021
@cockroach-teamcity
Copy link
Member Author

storage/metamorphic.TestPebbleEquivalence failed with artifacts on master @ db229cad354c2776ea35c9dc4f78e6ce66b9437c:

Fatal error:

panic: min timestamp hint set without max timestamp hint [recovered]
	panic: min timestamp hint set without max timestamp hint

Stack:

goroutine 38 [running]:
testing.tRunner.func1.2(0x12d4340, 0x168b1d0)
	/usr/local/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc00048c480)
	/usr/local/go/src/testing/testing.go:1146 +0x4b6
panic(0x12d4340, 0x168b1d0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).init(0xc0003dea20, 0x16c0e20, 0xc000213680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00063c538, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:157 +0x92f
github.com/cockroachdb/cockroach/pkg/storage.newPebbleIterator(0x16c0e20, 0xc000213680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00063c538, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:90 +0xd8
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).NewMVCCIterator(0xc00016c400, 0x1, 0x0, 0x0, 0x0, 0x0, 0xc00063c538, 0x8, 0x8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble.go:861 +0x1e5
github.com/cockroachdb/cockroach/pkg/storage.NewMVCCIncrementalIterator(0x16d0510, 0xc00016c400, 0x1, 0xc00063c538, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc_incremental_iterator.go:177 +0x235
github.com/cockroachdb/cockroach/pkg/storage.MVCCClearTimeRange(0x16c55b0, 0xc000124010, 0x7f85e06a5eb0, 0xc00016c400, 0xc003c15860, 0xc00063c530, 0x8, 0x8, 0xc00063c538, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2269 +0x298
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.mvccClearTimeRangeOp.run(0xc00048c780, 0x14b86d5, 0x6, 0xc00063c530, 0x8, 0x8, 0xc00063c538, 0x8, 0x8, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/operations.go:308 +0x1c5
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.(*metaTestRunner).generateAndRun(0xc00048c780, 0x186a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/generator.go:303 +0x142
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.runMetaTestForEngines(0x16c55b0, 0xc000124010, 0xc00048c480, 0x1, 0xf45fec00694ff42, 0x0, 0x0, 0x0, 0x16a3340, 0xc0001262a0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/meta_test.go:89 +0x465
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.runMetaTest.func2(0xc00048c480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/meta_test.go:155 +0x4b8
testing.tRunner(0xc00048c480, 0xc0004849f0)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3
Log preceding fatal error 

=== RUN   TestPebbleEquivalence
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestPebbleEquivalence126532895
    test_log_scope.go:80: use -show-logs to present logs inline
first run output file: /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence322760690/output.meta
--- FAIL: TestPebbleEquivalence (0.83s)
=== RUN   TestPebbleEquivalence/engine/standard=0
check file = 
output file = /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence322760690/output.meta
store path = /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence_engine_standard=0464881652/store
    --- FAIL: TestPebbleEquivalence/engine/standard=0 (0.83s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • GOFLAGS=-parallel=4

/cc @cockroachdb/storage itsbilal

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

storage/metamorphic.TestPebbleEquivalence failed with artifacts on master @ a15b840db34de504bf060b2599276e049cc64799:

Fatal error:

panic: min timestamp hint set without max timestamp hint [recovered]
	panic: min timestamp hint set without max timestamp hint

Stack:

goroutine 37 [running]:
testing.tRunner.func1.2(0x12d8900, 0x1690a30)
	/usr/local/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc0003fe480)
	/usr/local/go/src/testing/testing.go:1146 +0x4b6
panic(0x12d8900, 0x1690a30)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).init(0xc0000ef320, 0x16c71e0, 0xc000209200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00058c800, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:157 +0x92f
github.com/cockroachdb/cockroach/pkg/storage.newPebbleIterator(0x16c71e0, 0xc000209200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00058c800, 0xe, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:90 +0xd8
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).NewMVCCIterator(0xc0005d8000, 0x1, 0x0, 0x0, 0x0, 0x0, 0xc00058c800, 0xe, 0xe, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/pebble.go:861 +0x1e5
github.com/cockroachdb/cockroach/pkg/storage.NewMVCCIncrementalIterator(0x16d6b90, 0xc0005d8000, 0x1, 0xc00058c800, 0xe, 0xe, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc_incremental_iterator.go:177 +0x235
github.com/cockroachdb/cockroach/pkg/storage.MVCCClearTimeRange(0x16cbab0, 0xc000040108, 0x7f862c10e208, 0xc0005d8000, 0xc0026bd860, 0xc00058c880, 0xd, 0xd, 0xc00058c800, 0xe, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2269 +0x298
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.mvccClearTimeRangeOp.run(0xc0003fe600, 0x14bdb1f, 0x6, 0xc00058c880, 0xd, 0xd, 0xc00058c800, 0xe, 0xe, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/operations.go:308 +0x1c5
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.(*metaTestRunner).generateAndRun(0xc0003fe600, 0x186a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/generator.go:303 +0x142
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.runMetaTestForEngines(0x16cbab0, 0xc000040108, 0xc0003fe480, 0x1, 0x153a7e64c6f96106, 0x0, 0x0, 0x0, 0x16a95a0, 0xc00011c020, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/meta_test.go:89 +0x465
github.com/cockroachdb/cockroach/pkg/storage/metamorphic.runMetaTest.func2(0xc0003fe480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/metamorphic/meta_test.go:155 +0x4b8
testing.tRunner(0xc0003fe480, 0xc0004223f0)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3
Log preceding fatal error 

=== RUN   TestPebbleEquivalence
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestPebbleEquivalence384970526
    test_log_scope.go:80: use -show-logs to present logs inline
first run output file: /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence512034277/output.meta
--- FAIL: TestPebbleEquivalence (1.38s)
=== RUN   TestPebbleEquivalence/engine/standard=0
check file = 
output file = /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence512034277/output.meta
store path = /go/src/github.com/cockroachdb/cockroach/artifacts/TestPebbleEquivalence_engine_standard=0109308255/store
    --- FAIL: TestPebbleEquivalence/engine/standard=0 (1.37s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=4

/cc @cockroachdb/storage itsbilal

This test on roachdash | Improve this report!

@itsbilal itsbilal self-assigned this Nov 1, 2021
@itsbilal itsbilal mentioned this issue Nov 1, 2021
Merged
craig bot pushed a commit that referenced this issue Nov 1, 2021
@itsbilal
Merge #72288
039ed5f 
72288: storage/metamorphic: Increment endTime in ClearTimeRange r=nicktrav a=itsbilal

Eliminates a panic when endTime ends up being the zero
timestamp, and startTime is slightly ahead of endTime
as it gets next()'d in TBI creation but endTime doesn't.

Fixes #72269
Fixes #72261

Release note: None.

Co-authored-by: Bilal Akhtar <[email protected]>
@craig craig bot closed this as completed in d54673d Nov 1, 2021
@craig craig bot closed this as completed in #72288 Nov 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-storage Storage Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage/metamorphic: Increment endTime in ClearTimeRange itsbilal/cockroach
2 participants
@itsbilal @cockroach-teamcity