Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver/loqrecovery: TestRetrieveApplyStatus failed #111739

Closed
cockroach-teamcity opened this issue Oct 4, 2023 · 3 comments
Closed

kv/kvserver/loqrecovery: TestRetrieveApplyStatus failed #111739

cockroach-teamcity opened this issue Oct 4, 2023 · 3 comments
Assignees
@aliher1911
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. X-duplicate Closed as a duplicate of another issue.
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 4, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvserver/loqrecovery.TestRetrieveApplyStatus failed with artifacts on master @ 1ad178f80eff22c5cfe188c8b0c14fc5f834485d:

F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	GOROOT/src/sync/cond.go:70 +0x8c
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00356d2d0, {0x6451208, 0xc001180f00}, {0x6436bc0, 0xc00698ea80}, 0xc00698e000)
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x278
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6451208?, 0xc001180f00?})
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !goroutine 162418 [select]:
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00dc3cd20, 0x1)
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/controlbuf.go:416 +0x115
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00e4f8780)
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/controlbuf.go:534 +0x85
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go:448 +0x6d
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !created by google.golang.org/grpc/internal/transport.newHTTP2Client
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go:446 +0x23fc
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !goroutine 162499 [select]:
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/server.(*testServer).Activate.func2()
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/server/testserver.go:726 +0xc5
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/server.(*testServer).Activate
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/server/testserver.go:723 +0x15c
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/95fc0dff57a5bf3a6a9cc9666d301bb7/logTestRetrieveApplyStatus3190158304
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/95fc0dff57a5bf3a6a9cc9666d301bb7/logTestRetrieveApplyStatus3190158304
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !****************************************************************************
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !This node experienced a fatal error (printed above), and as a result the
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !process is terminating.
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !will try to determine the root cause, recommend next steps, and we can
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !improve CockroachDB based on your report.
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !Please submit a crash report by following the instructions here:
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !If you would rather not post publicly, please contact us directly at:
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !    [email protected]
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !
F231004 11:09:36.548266 168219 3@pebble/objstorage/objstorageprovider/provider.go:218  [n2,s2,pebble] 1 !The Cockroach Labs team appreciates your feedback.

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-32049
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv-replication labels Oct 4, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Oct 4, 2023
@aliher1911 aliher1911 self-assigned this Oct 9, 2023
@aliher1911
Copy link
Contributor

Fails because pebble panic: file 000449 (type 2) unknown to the objstorage provider: file does not exist

Did we corrupt the store on node shutdown somehow?

@erikgrinaker
Copy link
Contributor

I think this is a dupe of cockroachdb/pebble#2973

@aliher1911 aliher1911 added the X-duplicate Closed as a duplicate of another issue. label Oct 9, 2023
@aliher1911
Copy link
Contributor

Looks like it.
Fixed by pebble bump in #111776

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aliher1911 aliher1911

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. X-duplicate Closed as a duplicate of another issue.
Projects
None yet
Milestone
23.2
Development

No branches or pull requests
3 participants
@erikgrinaker @aliher1911 @cockroach-teamcity