Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/changefeedccl: TestChangefeedColumnDropsWithFamilyAndNonFamilyTargets failed #110235

Closed
cockroach-teamcity opened this issue Sep 8, 2023 · 0 comments · Fixed by #110250
Closed
Assignees
Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 8, 2023

ccl/changefeedccl.TestChangefeedColumnDropsWithFamilyAndNonFamilyTargets failed with artifacts on master @ 1184f961a15f57104709f72ed9ed62c731f03bca:

      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Previous write at 0x00c0081da578 by goroutine 434165:
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*sliMetrics).getLaggingRangesCallback.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/metrics.go:804 +0x8e
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.startLaggingRangesObserver.func1.1.1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:185 +0x32
  runtime.deferreturn()
      GOROOT/src/runtime/panic.go:476 +0x32
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Goroutine 435186 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12e
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x109
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.startLaggingRangesObserver.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:182 +0x145
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:227 +0x424
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans-fm()
      <autogenerated>:1 +0xe4
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.rangefeedFactory.Run.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/physical_kv_feed.go:93 +0x11d
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Goroutine 434165 (finished) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12e
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x109
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.startLaggingRangesObserver.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:182 +0x145
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:227 +0x424
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans-fm()
      <autogenerated>:1 +0xe4
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.rangefeedFactory.Run.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/physical_kv_feed.go:93 +0x11d
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86
==================

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-31337

Epic CRDB-11732

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Sep 8, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 8, 2023
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Sep 8, 2023
@miretskiy miretskiy self-assigned this Sep 8, 2023
miretskiy pushed a commit to miretskiy/cockroach that referenced this issue Sep 8, 2023
Fix a race bug in lagging spans metric.

Fixes: cockroachdb#110235

Release note: None
craig bot pushed a commit that referenced this issue Sep 8, 2023
108824: sql: implement datetime builtins r=rafiss,chrisseto a=annrpom

Previously, `make_date`, `make_timestamp`, and `make_timestamptz`
built-ins were not implemented. In addition, a new signature for
`date_trunc` was not implemented. This was inadequate because it caused
an incompatibility issue for tools that needed these datetime functions.
To address this, this patch adds said datetime built-ins. Note that behaviors
do not align with postgres when negative years are inpits, as we adhere
to ISO 8601 standard.

Epic: none
Fixes: #108448

Release note (sql change): Datetime built-ins (make_date,
make_timestamp, and make_timestamptz) are implemented - allowing for
the creation of timestamps, timestamps with time zones, and dates. In
addition, date_trunc now allows for a timestamp to be truncated in a
provided timezone (to a provided precision).

109346: kvcoord: Pace rangefeed client goroutine creation r=miretskiy a=miretskiy

Acquire catchup scan quota prior to goroutine creation in order to pace the goroutine creation rate.

This change results in nice and smooth growth in
goroutine count, thus reducing the pressure on goroutine scheduler, which in turn reduces the impact on SQL latency during changefeed startup.

This change also improves observability in rangefeed client by introducing new counters:
 * `distsender.rangefeed.retry.<reason>`: counter keeping track of the number of ranges that ecountered a retryable error of particular type (e.g. slow counsumer, range split, etc).

Observability also enhanced by adding a column to
 `crdb_internal.active_rangefeed` virtual table augment to indicate
if the range is currently in catchup scan mode.

Fixes #98842

Release note (enterprise change): Pace rangefeed goroutine creation
rate to improve scheduler latency.  Improve observability by adding
additional metrics indicating the reason for rangefeed restart
as well as additional column in the `crdb_internal.active_rangefeed`
table to indicate if the range is currently in catchup scan mode.

109694: ui: all xhr paths from db console are now relative r=santamaura,zachlite,j82w a=dhartunian

This commit removes all prefix `/` characters from request paths in the DB Console and Cluster UI codebases. This ensures that if the DB Console is proxied at a subpath the requests continue to work as expected and are relative to the correct base path.

Resolves: cockroachdb/helm-charts#228
Resolves: #91429

Epic CRDB-21265

Release note (ops change, ui change): The DB Console now constructs client-side requests using relative URLs instead of absolute ones. This enables proxying of the DB Console at arbitrary subpaths.

109774: sql,keys: disregard checks in *ZoneConfig*Batch r=rafiss a=annrpom

Previously, a bug occured where a transactional schema change,
 `ALTER RANGE default CONFIGURE ZONE...`, statement would
produce a CPut failure, even though both statements do take effect.
This was due to a guard in the code that blocked us from adding the
first zone config to the uncommitted cache, causing the expValues
being updated to the KV batch to be the same for both statements.
This patch addresses the issue by removing the check and allowing
for `default` and psuedotables (like system ranges) to be added to
the uncommitted cache.

Epic: none
Release note (bug fix): two `ALTER RANGE default CONFIGURE ZONE` 
statements on the same line no longer displays an error. 
Fixes: #108253

110250: changefeedccl: Fix data race in lagging spans metric r=miretskiy a=miretskiy

Fix a race bug in lagging spans metric.

Fixes: #110235

Release note: None

Co-authored-by: Annie Pompa <[email protected]>
Co-authored-by: Yevgeniy Miretskiy <[email protected]>
Co-authored-by: David Hartunian <[email protected]>
@craig craig bot closed this as completed in #110250 Sep 8, 2023
jayshrivastava pushed a commit to jayshrivastava/cockroach that referenced this issue Sep 20, 2023
Fix a race bug in lagging spans metric.

Fixes: cockroachdb#110235

Release note: None
jayshrivastava pushed a commit to jayshrivastava/cockroach that referenced this issue Sep 26, 2023
Fix a race bug in lagging spans metric.

Fixes: cockroachdb#110235

Release note: None
jayshrivastava pushed a commit to jayshrivastava/cockroach that referenced this issue Sep 26, 2023
Fix a race bug in lagging spans metric.

Fixes: cockroachdb#110235

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants