ccl/changefeedccl: TestChangefeedHandlesDrainingNodes failed #100903

cockroach-teamcity · 2023-04-07T06:48:15Z

ccl/changefeedccl.TestChangefeedHandlesDrainingNodes failed with artifacts on release-23.1 @ 571a1744c72415894dc0e742bd89e74938ea71f4:

=== RUN   TestChangefeedHandlesDrainingNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedHandlesDrainingNodes3712854043
    test_log_scope.go:79: use -show-logs to present logs inline
    helpers_test.go:895: making webhook feed factory
    changefeed_test.go:6055: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:195
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:270
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:6055
        	Error:      	Received unexpected error:
        	            	Details for job 854521624819171329: pq: system-jobs-scan: could not register flowID 4a24d537-00fc-4e4e-9115-0ac45b500bf2 because the registry is draining
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*jobFeed).Details
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/testfeed_test.go:468
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*webhookFeed).Next
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/testfeed_test.go:2126
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:101
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:206
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:198
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:263
        	            	  | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:259
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:270
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestChangefeedHandlesDrainingNodes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:6055
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	Wraps: (2) Details for job 854521624819171329
        	            	Wraps: (3) pq: system-jobs-scan: could not register flowID 4a24d537-00fc-4e4e-9115-0ac45b500bf2 because the registry is draining
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestChangefeedHandlesDrainingNodes
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/905465f697dc04c39f7d8a82cb057bfe/logTestChangefeedHandlesDrainingNodes3712854043
--- FAIL: TestChangefeedHandlesDrainingNodes (8.12s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc _{This test on roachdash | Improve this report!

Jira issue: CRDB-26682
Epic CRDB-11732}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-04-08T07:39:15Z

ccl/changefeedccl.TestChangefeedHandlesDrainingNodes failed with artifacts on release-23.1 @ de239a7438f44d382c9aefceb65d9c39911dabd2:

=== RUN   TestChangefeedHandlesDrainingNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1c3ce425109520152572148b124af7f1/logTestChangefeedHandlesDrainingNodes2127889315
    test_log_scope.go:79: use -show-logs to present logs inline
    helpers_test.go:895: making pubsub feed factory
    changefeed_test.go:6056: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:195
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:270
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:6056
        	Error:      	Received unexpected error:
        	            	Details for job 854811279810789377: pq: system-jobs-scan: could not register flowID c5a93cf8-208c-4492-b506-713bcd61b350 because the registry is draining
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*jobFeed).Details
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/testfeed_test.go:473
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*pubsubFeed).Next
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/testfeed_test.go:2489
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:101
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:206
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:198
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:263
        	            	  | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:259
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:270
        	            	  | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestChangefeedHandlesDrainingNodes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:6056
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	Wraps: (2) Details for job 854811279810789377
        	            	Wraps: (3) pq: system-jobs-scan: could not register flowID c5a93cf8-208c-4492-b506-713bcd61b350 because the registry is draining
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestChangefeedHandlesDrainingNodes
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1c3ce425109520152572148b124af7f1/logTestChangefeedHandlesDrainingNodes2127889315
--- FAIL: TestChangefeedHandlesDrainingNodes (6.23s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-09T14:22:08Z

ccl/changefeedccl.TestChangefeedHandlesDrainingNodes failed with artifacts on release-23.1 @ cfbd751ae4c82e2c4b5314376cd6e6c29cc30aea:

=== RUN   TestChangefeedHandlesDrainingNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8664cbf2d4e437e0e06ce871fe9210c9/logTestChangefeedHandlesDrainingNodes1936342772
    test_log_scope.go:79: use -show-logs to present logs inline
    helpers_test.go:895: making pubsub feed factory
    changefeed_test.go:6048: failed to start feed for job 0: pq: opening client: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/8664cbf2d4e437e0e06ce871fe9210c9/logTestChangefeedHandlesDrainingNodes1936342772
--- FAIL: TestChangefeedHandlesDrainingNodes (8.21s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-10T09:22:11Z

ccl/changefeedccl.TestChangefeedHandlesDrainingNodes failed with artifacts on release-23.1 @ 2f96695f75b07c872ec5f146acc1fa198135768f:

=== RUN   TestChangefeedHandlesDrainingNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8664cbf2d4e437e0e06ce871fe9210c9/logTestChangefeedHandlesDrainingNodes144194294
    test_log_scope.go:79: use -show-logs to present logs inline
    helpers_test.go:895: making pubsub feed factory
    changefeed_test.go:6048: failed to start feed for job 0: pq: opening client: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/8664cbf2d4e437e0e06ce871fe9210c9/logTestChangefeedHandlesDrainingNodes144194294
--- FAIL: TestChangefeedHandlesDrainingNodes (11.75s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

knz · 2023-04-10T09:43:57Z

cc @miretskiy for triage.

This bug was already there -- it was just recently revealed when we merged #100476.

miretskiy · 2023-04-10T13:28:04Z

I'm not sure it's fair to characterize this issue as a bug that was already there -- perhaps the test was relying on some shutdown ordering that changed. But clearly, the test wasn't failing before. I'll take a look.

cockroach-teamcity · 2023-04-10T14:48:15Z

ccl/changefeedccl.TestChangefeedHandlesDrainingNodes failed with artifacts on release-23.1 @ 416beb787ce5cf90695f13ebb7c95f749dc14061:

=== RUN   TestChangefeedHandlesDrainingNodes
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/8664cbf2d4e437e0e06ce871fe9210c9/logTestChangefeedHandlesDrainingNodes701142468
    test_log_scope.go:79: use -show-logs to present logs inline
    helpers_test.go:895: making pubsub feed factory
    changefeed_test.go:6048: failed to start feed for job 0: pq: opening client: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/8664cbf2d4e437e0e06ce871fe9210c9/logTestChangefeedHandlesDrainingNodes701142468
--- FAIL: TestChangefeedHandlesDrainingNodes (9.32s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

100893: ui: search criteria ux improvements r=maryliag a=maryliag Some of the names of sort on search criteria were not a match for the column name on the tables, which could cause confusion. This commit updates the values of "P99" to "P99 Latency" and "Service Latency" to "Statement time" and "Transaction time". Epic: None Release note (ui change): Update sort label on Search Criteria to match the name on the table columns. 101058: roachtest: bump tpccbench timeout r=srosenberg a=renatolabs Looking at the test history, we see that tpccbench may sometimes take longer than 5h, especially in multi-region setups. For that reason, we bump the timeout for this test to 7h, which should be sufficient and avoid failures due to timeouts. This commit also removes an unused `MinVersion` field in the `tpccBenchSpec` struct. Resolves #100975. Release note: None 101077: changefeedccl: Fix TestChangefeedHandlesDrainingNodes test r=miretskiy a=miretskiy The test becamse flaky after #100476 merged Fixes #100903 Release note: None 101097: server: fix a race condition during server initialization r=irfansharif a=knz Fixes #91414. Fixes #101010. Fixes #100902. The call to `registerEnginesForDiskStatsMap` needs to wait until the store IDs are known. Release note: None Epic: None Co-authored-by: maryliag <[email protected]> Co-authored-by: Renato Costa <[email protected]> Co-authored-by: Yevgeniy Miretskiy <[email protected]> Co-authored-by: Raphael 'kena' Poss <[email protected]>

The test becamse flaky after #100476 merged Fixes #100903 Release note: None

cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 7, 2023

cockroach-teamcity added this to the 23.1 milestone Apr 7, 2023

knz added GA-blocker A-cdc Change Data Capture branch-master Failures and bugs on the master branch. T-cdc labels Apr 10, 2023

miretskiy self-assigned this Apr 10, 2023

miretskiy mentioned this issue Apr 10, 2023

changefeedccl: Fix TestChangefeedHandlesDrainingNodes test #101077

Merged

craig bot closed this as completed in e0b9e1c Apr 10, 2023

blathers-crl bot mentioned this issue Apr 10, 2023

release-23.1: changefeedccl: Fix TestChangefeedHandlesDrainingNodes test #101124

Merged

blathers-crl bot pushed a commit that referenced this issue Apr 10, 2023

changefeedccl: Fix TestChangefeedHandlesDrainingNodes test

f6df923

The test becamse flaky after #100476 merged Fixes #100903 Release note: None

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/changefeedccl: TestChangefeedHandlesDrainingNodes failed #100903

ccl/changefeedccl: TestChangefeedHandlesDrainingNodes failed #100903

cockroach-teamcity commented Apr 7, 2023 •

edited by exalate-issue-sync bot

Loading

cockroach-teamcity commented Apr 8, 2023

cockroach-teamcity commented Apr 9, 2023

cockroach-teamcity commented Apr 10, 2023

knz commented Apr 10, 2023

miretskiy commented Apr 10, 2023

cockroach-teamcity commented Apr 10, 2023

ccl/changefeedccl: TestChangefeedHandlesDrainingNodes failed #100903

ccl/changefeedccl: TestChangefeedHandlesDrainingNodes failed #100903

Comments

cockroach-teamcity commented Apr 7, 2023 • edited by exalate-issue-sync bot Loading

cockroach-teamcity commented Apr 8, 2023

cockroach-teamcity commented Apr 9, 2023

cockroach-teamcity commented Apr 10, 2023

knz commented Apr 10, 2023

miretskiy commented Apr 10, 2023

cockroach-teamcity commented Apr 10, 2023

cockroach-teamcity commented Apr 7, 2023 •

edited by exalate-issue-sync bot

Loading