Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

changefeedccl: Test panic #110772

Closed
miretskiy opened this issue Sep 16, 2023 · 1 comment · Fixed by #110773
Closed

changefeedccl: Test panic #110772

miretskiy opened this issue Sep 16, 2023 · 1 comment · Fixed by #110773
Labels
A-cdc Change Data Capture C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). T-cdc

Comments

@miretskiy
Copy link
Contributor

miretskiy commented Sep 16, 2023 •
edited by cockroach-jira-scripts
Loading

Noticed in a failed CI run.
https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/11801001?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

No pass/skip/fail event found for test
=== RUN   TestChangefeedPropagatesTerminalError/aggregator
    helpers_test.go:930: making cloudstorage feed factory
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:884
*   | runtime.panicmem
*   |   GOROOT/src/runtime/panic.go:260
*   | runtime.sigpanic
*   |   GOROOT/src/runtime/signal_unix.go:841
*   | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).close
*   |   github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:546
*   | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).ConsumerClosed
*   |   github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:831
*   | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).ConsumerClosedOnHeadProc
*   |   github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:679
*   | github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Cleanup
*   |   github.com/cockroachdb/cockroach/pkg/sql/rowflow/row_based_flow.go:460
*   | github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupFlow.func1
*   |   github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:675
*   | github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupFlow
*   |   github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:689
*   | github.com/cockroachdb/cockroach/pkg/sql/execinfrapb._DistSQL_SetupFlow_Handler.func1
*   |   github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/bazel-out/k8-fastbuild/bin/pkg/sql/execinfrapb/execinfrapb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/api.pb.go:602
*   | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
*   | google.golang.org/grpc.getChainUnaryHandler.func1
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
*   | google.golang.org/grpc.getChainUnaryHandler.func1
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
*   | google.golang.org/grpc.getChainUnaryHandler.func1
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
*   | google.golang.org/grpc.chainUnaryInterceptors.func1
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154
*   | github.com/cockroachdb/cockroach/pkg/sql/execinfrapb._DistSQL_SetupFlow_Handler
*   |   github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/bazel-out/k8-fastbuild/bin/pkg/sql/execinfrapb/execinfrapb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/api.pb.go:604
*   | google.golang.org/grpc.(*Server).processUnaryRPC
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336
*   | google.golang.org/grpc.(*Server).handleStream
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704
*   | google.golang.org/grpc.(*Server).serveStreams.func1.2
*   |   google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
*   | runtime.goexit
*   |   src/runtime/asm_amd64.s:1598
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
  panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4caa1f2]
goroutine 86029 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc008626e50?, {0x7bcefe0, 0xc002d3eea0})
  github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5b9c680, 0xb169a70})
  GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).close(0xc0076d6f00)
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:546 +0x32
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).ConsumerClosed(0x0?)
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:831 +0x19
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).ConsumerClosedOnHeadProc(0xc008627070?)
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:679 +0x46
github.com/cockroachdb/cockroach/pkg/sql/rowflow.(*rowBasedFlow).Cleanup(0xc0041f2f30, {0x7bcef38, 0xc005fcb220})
  github.com/cockroachdb/cockroach/pkg/sql/rowflow/row_based_flow.go:460 +0x98
github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupFlow.func1(0xc0070e9000, 0xc008627290, 0xc00022a000?, 0xc00ac55040?, {0x7bcefe0, 0xc002d3eea0})
  github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:675 +0x1f9
github.com/cockroachdb/cockroach/pkg/sql/distsql.(*ServerImpl).SetupFlow(0xc0070e9000, {0x7bcef38, 0xc005fcb220}, 0xc00ac55040)
  github.com/cockroachdb/cockroach/pkg/sql/distsql/server.go:689 +0x20f
github.com/cockroachdb/cockroach/pkg/sql/execinfrapb._DistSQL_SetupFlow_Handler.func1({0x7bcefe0, 0xc002d3eea0}, {0x629b460?, 0xc00ac55040})
  github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/bazel-out/k8-fastbuild/bin/pkg/sql/execinfrapb/execinfrapb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/api.pb.go:602 +0x78
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1({0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040}, 0xc0098d29c0, 0xc0041f2ea0)
  github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x254
google.golang.org/grpc.getChainUnaryHandler.func1({0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040})
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040}, 0xc0098d29c0?, 0xc00c4ac100)
  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x83
google.golang.org/grpc.getChainUnaryHandler.func1({0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040})
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0045ba000?, {{0x45f647?}, {0x7c0d330?, 0xc006773950?}}}, {0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040}, 0xc0098d29c0, 0xc00c4ac040)
  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x288
google.golang.org/grpc.getChainUnaryHandler.func1({0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040})
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x7bcefe0?, 0xc002d3eea0?})
  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0079fd050, {0x7bcefe0, 0xc002d3eea0}, {0x0?, 0xc00c2f7980?}, 0xc00c2f7978)
  github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd1
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x7bcefe0?, 0xc002d3eea0?}, {0x629b460?, 0xc00ac55040?}, 0xc0098d29c0?, 0xc0041f2ea0?)
  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x95
google.golang.org/grpc.chainUnaryInterceptors.func1({0x7bcefe0, 0xc002d3eea0}, {0x629b460, 0xc00ac55040}, 0xc007feba50?, 0x5b9aee0?)
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x8f
github.com/cockroachdb/cockroach/pkg/sql/execinfrapb._DistSQL_SetupFlow_Handler({0x637be00?, 0xc0070e9000}, {0x7bcefe0, 0xc002d3eea0}, 0xc00861b810, 0xc008842d80)
  github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/bazel-out/k8-fastbuild/bin/pkg/sql/execinfrapb/execinfrapb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/execinfrapb/api.pb.go:604 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc005a8b4a0, {0x7c21500, 0xc007c15380}, 0xc00d7f45a0, 0xc003f3dce0, 0xb18bd00, 0x0)
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd33
google.golang.org/grpc.(*Server).handleStream(0xc005a8b4a0, {0x7c21500, 0xc007c15380}, 0xc00d7f45a0, 0x0)
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
  google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x28a

Jira issue: CRDB-31605
@miretskiy miretskiy added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). A-cdc Change Data Capture T-cdc labels Sep 16, 2023
@blathers-crl
Copy link

blathers-crl bot commented Sep 16, 2023

cc @cockroachdb/cdc

@miretskiy miretskiy mentioned this issue Sep 16, 2023
Merged
craig bot pushed a commit that referenced this issue Sep 18, 2023
@Xiang-Gu
Merge #110686 #110773
8fc109c 
110686: release: update predecessor map for 22.2.14 r=Xiang-Gu a=Xiang-Gu

Epic: None
Release justification: normal release process
Release note: None

110773: changefeedccl: Fix potential shutdown NPE r=miretskiy a=miretskiy

A flow may be cancelled immediately after it's created, before it has `Start`ed.  If this happens, change aggregator may panic because certain state may not be initialized.

Fixes #110772

Release note: None

Co-authored-by: Xiang Gu <[email protected]>
Co-authored-by: Yevgeniy Miretskiy <[email protected]>
@jayshrivastava jayshrivastava mentioned this issue Sep 18, 2023
Closed
@craig craig bot closed this as completed in 9039d6d Sep 18, 2023
blathers-crl bot pushed a commit that referenced this issue Sep 18, 2023
changefeedccl: Fix potential shutdown NPE
004cb4d 
A flow may be cancelled immediately after it's created,
before it has `Start`ed.  If this happens, change aggregator
may panic because certain state may not be initialized.

Fixes #110772

Release note: None
This was referenced Sep 18, 2023
Merged
Merged
blathers-crl bot pushed a commit that referenced this issue Sep 18, 2023
changefeedccl: Fix potential shutdown NPE
8b5382e 
A flow may be cancelled immediately after it's created,
before it has `Start`ed.  If this happens, change aggregator
may panic because certain state may not be initialized.

Fixes #110772

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-cdc Change Data Capture C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). T-cdc
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

changefeedccl: Fix potential shutdown NPE miretskiy/cockroach
1 participant
@miretskiy