Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

p2p: grpc: received message larger than max #8004

Closed
overvenus opened this issue Dec 30, 2022 · 1 comment · Fixed by #8005
Closed

p2p: grpc: received message larger than max #8004

overvenus opened this issue Dec 30, 2022 · 1 comment · Fixed by #8005
Labels
affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 affects-6.3 affects-6.4 affects-6.5 This bug affects the 6.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@overvenus
Copy link
Member

overvenus commented Dec 30, 2022 •
edited
Loading

Checkpoint can not advance when replicating 72k tables in v6.5.0.

[2022/12/30 17:01:00.000 +08:00] [INFO] [server.go:535] ["peer connection received"] [senderID=379a741e-154d-4863-a15c-da459813c5d1] [senderAdvertiseAddr=172.16.5.37:48112] [addr=172.16.5.37:57070] [epoch=372]
[2022/12/30 17:01:00.006 +08:00] [WARN] [server.go:630] ["peer-to-peer message handler error"] [error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4207687 vs. 4194304)"] [errorVerbose="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4207687 vs. 4194304)\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).receive\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:722\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).SendMessage.func1\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:629\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594"]
[2022/12/30 17:01:00.006 +08:00] [WARN] [server.go:633] ["error receiving from peer"] [error="context canceled"]
[2022/12/30 17:01:00.025 +08:00] [WARN] [server.go:334] ["sending response to peer failed"] [error="[CDC:ErrPeerMessageInternalSenderClosed]peer-to-peer message server tries to send to a closed stream. Internal only."] [errorVerbose="[CDC:ErrPeerMessageInternalSenderClosed]peer-to-peer message server tries to send to a closed stream. Internal only.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/[email protected]/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/p2p.(*streamHandle).Send\n\tgithub.com/pingcap/tiflow/pkg/p2p/server_stream_handle.go:66\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).tick\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:329\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).run\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:228\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).Run.func2\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:206\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594"]
[2022/12/30 17:01:00.025 +08:00] [INFO] [server.go:343] ["Deregistering peer"] [sender=379a741e-154d-4863-a15c-da459813c5d1] [epoch=372] []
[2022/12/30 17:01:00.391 +08:00] [INFO] [server.go:535] ["peer connection received"] [senderID=0dc8a568-2e7c-4f35-b6fa-a5d3991cf1c6] [senderAdvertiseAddr=172.16.5.37:48114] [addr=172.16.5.37:57074] [epoch=406]
@overvenus overvenus added type/bug The issue is confirmed as a bug. area/ticdc Issues or PRs related to TiCDC. labels Dec 30, 2022
@overvenus
Copy link
Member Author

This issue causes changefeed stuck, checkpoint can no longer advance.
image

@overvenus overvenus mentioned this issue Dec 30, 2022
Merged
@overvenus overvenus added affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 affects-6.3 affects-6.4 affects-6.5 This bug affects the 6.5.x(LTS) versions. labels Dec 30, 2022
ti-chi-bot pushed a commit that referenced this issue Jan 3, 2023
@overvenus
p2p(ticdc): use a larger max receive message size (#8005)
5da8a95 
close #8004
This was referenced Jan 3, 2023
Merged
Merged
overvenus added a commit to ti-chi-bot/tiflow that referenced this issue Jan 6, 2023
@overvenus
p2p(ticdc): use a larger max receive message size (pingcap#8005)
b33937c 
close pingcap#8004

Signed-off-by: Neil Shen <[email protected]>
overvenus added a commit to ti-chi-bot/tiflow that referenced this issue Jan 6, 2023
@overvenus
p2p(ticdc): use a larger max receive message size (pingcap#8005)
1bcd5bd 
close pingcap#8004

Signed-off-by: Neil Shen <[email protected]>
ti-chi-bot added a commit that referenced this issue Jan 6, 2023
@ti-chi-bot
p2p(ticdc): use a larger max receive message size (#8005) (#8014)
ca4a0bf 
close #8004
ti-chi-bot added a commit that referenced this issue Jan 10, 2023
@ti-chi-bot
p2p(ticdc): use a larger max receive message size (#8005) (#8013)
ee2a385 
close #8004
@nongfushanquan nongfushanquan mentioned this issue Feb 3, 2023
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 affects-6.3 affects-6.4 affects-6.5 This bug affects the 6.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

p2p(ticdc): use a larger max receive message size overvenus/ticdc
1 participant
@overvenus