Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: TestBackupRestoreTenant failed #121200

Closed
cockroach-teamcity opened this issue Mar 27, 2024 · 0 comments · Fixed by #121217
Closed

ccl/backupccl: TestBackupRestoreTenant failed #121200

cockroach-teamcity opened this issue Mar 27, 2024 · 0 comments · Fixed by #121217
Assignees
@aadityasondhi
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 27, 2024 •
edited by cockroach-jira-scripts
Loading

ccl/backupccl.TestBackupRestoreTenant failed on master @ 87b22ac094b519b3a7b807758aa5249c120a895d:

      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:177 +0x4f
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.Group.GoCtx.func2()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x46
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91

Previous write at 0x00c01277bcd8 by goroutine 61168:
  github.com/cockroachdb/cockroach/pkg/util/admission.(*Pacer).Pace()
      github.com/cockroachdb/cockroach/pkg/util/admission/pacer.go:47 +0x23e
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*fileSSTSink).copyPointKeys()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/file_sst_sink.go:312 +0x44a
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*fileSSTSink).write()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/file_sst_sink.go:255 +0x7fc
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runBackupProcessor.func2()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_processor.go:718 +0x25bb
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:177 +0x4f
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.Group.GoCtx.func2()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x46
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91

Goroutine 61170 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x124
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x64
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:177 +0x8f
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runBackupProcessor()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_processor.go:486 +0x184f
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*backupDataProcessor).Start.func2()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_processor.go:234 +0xe7
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x262

Goroutine 61168 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x124
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x64
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:177 +0x8f
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runBackupProcessor()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_processor.go:486 +0x184f
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*backupDataProcessor).Start.func2()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_processor.go:234 +0xe7
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x262
==================

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=44
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-37134
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Mar 27, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 27, 2024
@msbutler msbutler removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 27, 2024
@aadityasondhi aadityasondhi mentioned this issue Mar 27, 2024
Merged
craig bot pushed a commit that referenced this issue Mar 27, 2024
@arulajmani @rharding6373
@aadityasondhi @yuzefovich
Merge #119933 #120787 #121217 #121222
3e488c5 
119933: kv: add ability to verify pipelined replicated shared/exclusive locks  r=nvanbenschoten a=arulajmani

Previously, QueryIntent requests were only used to verify whether an
intent was successfully evaluated and replicated. This patch extends
QueryIntent request to also be able to verify whether a pipelined
shared or exclusive lock was successfully replicated or not.

Informs #117978

Release note: None

120787: ui: use max aggregator for commit latency on changefeed dashboard r=rharding6373 a=rharding6373

Previously, the commit latency in the changefeed dashboard in the db console would be aggregated by sum across all nodes. This was confusing for users who might see unexpectedly high commit latency.

In this change, we use max aggregation for the commit latency so that users see the max commit latency from all the nodes instead of the sum. This provides more useful observability into changefeed behavior.

Fixes: #119246
Fixes: #112947
Epic: None

Release note (ui change): The "Commit Latency" chart in the changefeed dashboard now aggregates by max instead of by sum for multi-node changefeeds. This more accurately reflects the amount of time for events to be acknowledged by the downstream sink.

121217: backupccl: fix data race with admission pacer r=msbutler a=aadityasondhi

We now use one pacer per fileSSTSink.

Fixes #121199.
Fixes #121202.
Fixes #121201.
Fixes #121200.
Fixes #121198.
Fixes #121197.
Fixes #121196.
Fixes #121195.
Fixes #121194.
Fixes #121193.
Fixes #121192.
Fixes #121191.
Fixes #121190.
Fixes #121189.
Fixes #121188.
Fixes #121187.

Release note: None

121222: optbuilder: fix recently introduced nil pointer in error case r=yuzefovich a=yuzefovich

This commit fixes a recently introduced nil pointer internal error when attempting to CALL not a procedure that is specified not by its name. `ResolvableFunctionReference` might not have `ReferenceByName`, so this commit switches to using `FunctionReference` that is always set.

Fixes: #121095.

Release note: None

Co-authored-by: Arul Ajmani <[email protected]>
Co-authored-by: rharding6373 <[email protected]>
Co-authored-by: Aaditya Sondhi <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in 1a26361 Mar 27, 2024
@craig craig bot closed this as completed in #121217 Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aadityasondhi aadityasondhi

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Status: Done
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

backupccl: fix data race with admission pacer aadityasondhi/cockroach
3 participants
@aadityasondhi @cockroach-teamcity @msbutler