Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiCDC gets the Kafka metadata too frequently and too aggressive #8959

Closed
2 tasks
Rustin170506 opened this issue May 15, 2023 · 1 comment · Fixed by #9060, #9107 or #9287
Closed
2 tasks

TiCDC gets the Kafka metadata too frequently and too aggressive #8959

Rustin170506 opened this issue May 15, 2023 · 1 comment · Fixed by #9060, #9107 or #9287
Assignees
@Rustin170506
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. component/sink Sink component. kafka type/enhancement The issue or PR belongs to an enhancement.

Comments

@Rustin170506
Copy link
Member

Rustin170506 commented May 15, 2023 •
edited
Loading

In TiCDC, we get all topic info with the admin interface:

tiflow/cdc/sink/dmlsink/mq/manager/kafka_manager.go

Line 138 in 1335f98

topicMetaList, err := m.admin.GetTopicsMeta(ctx, topicList, true)

This would be a problem when you have a lot of topics and changefeeds.

At the same time, the frequency is too high:

tiflow/cdc/sink/dmlsink/mq/manager/kafka_manager.go

Line 87 in 1335f98

if time.Since(time.Unix(m.lastMetadataRefresh.Load(), 0)) > time.Minute {

So we need to solve the problem to get better performance and avoid affecting the downstream Kafka.

  1. Change the frequency to 5-10min
  2. Only get used topic information instead of all topic information

See more at https://asktug.com/t/topic/1005884/2
@Rustin170506 Rustin170506 added component/sink Sink component. area/ticdc Issues or PRs related to TiCDC. kafka labels May 15, 2023
@Rustin170506 Rustin170506 self-assigned this May 15, 2023
@Rustin170506 Rustin170506 mentioned this issue May 25, 2023
Merged
@Rustin170506 Rustin170506 added the type/enhancement The issue or PR belongs to an enhancement. label May 25, 2023
@nongfushanquan
Copy link
Contributor

/label affects-7.1
/label affects-6.5

@ti-chi-bot ti-chi-bot bot added affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. labels May 26, 2023
ti-chi-bot bot pushed a commit that referenced this issue May 31, 2023
@Rustin170506
sink(ticdc): Optimize the performance when getting the metadata of Ka…
aefbf33 
…fka topics (#9060)

close #8957, close #8959
This was referenced May 31, 2023
Merged
Merged
@Rustin170506 Rustin170506 mentioned this issue May 31, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 1, 2023
@Rustin170506
pkg/sink(ticdc): add GetTopicConfig support (#9107)
493c0f6 
close #8959
This was referenced Jun 1, 2023
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 2, 2023
@ti-chi-bot
sink(ticdc): Optimize the performance when getting the metadata of Ka…
8a3bb9c 
…fka topics (#9060) (#9104)

close #8957, close #8959
@nongfushanquan nongfushanquan mentioned this issue Jun 5, 2023
Closed
@ti-chi-bot ti-chi-bot mentioned this issue Jun 7, 2023
Closed
@Rustin170506 Rustin170506 mentioned this issue Jun 7, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 8, 2023
@ti-chi-bot
sink(ticdc): Optimize the performance when getting the metadata of Ka…
8c3a94b 
…fka topics (#9060) (#9105)

close #8957, close #8959
@nongfushanquan nongfushanquan mentioned this issue Jun 9, 2023
Merged
17 tasks
Rustin170506 added a commit to ti-chi-bot/tiflow that referenced this issue Jun 12, 2023
@Rustin170506
pkg/sink(ticdc): add GetTopicConfig support (pingcap#9107)
669422f 
close pingcap#8959
ti-chi-bot bot pushed a commit that referenced this issue Jun 12, 2023
@ti-chi-bot
pkg/sink(ticdc): add GetTopicConfig support (#9107) (#9111)
ebabf06 
close #8959
ti-chi-bot bot pushed a commit that referenced this issue Jun 13, 2023
@Rustin170506
sink(ticdc): Optimize the performance when getting the metadata of Ka…
e94472f 
…fka topics (#9146)

close #8957, close #8959
@Rustin170506 Rustin170506 reopened this Jun 21, 2023
@Rustin170506 Rustin170506 mentioned this issue Jun 21, 2023
Merged
@nongfushanquan nongfushanquan mentioned this issue Jun 21, 2023
Merged
16 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jun 26, 2023
@Rustin170506
pkg/sink(ticdc): Update Kafka cluster every 30min (#9287)
006eda3 
close #8959
This was referenced Jun 26, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 27, 2023
@ti-chi-bot
pkg/sink(ticdc): Update Kafka cluster every 30min (#9287) (#9293)
631ef8c 
close #8959
ti-chi-bot bot pushed a commit that referenced this issue Jun 30, 2023
@ti-chi-bot
pkg/sink(ticdc): Update Kafka cluster every 30min (#9287) (#9292)
2b80a47 
close #8959
@asddongmen asddongmen mentioned this issue Aug 3, 2023
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Rustin170506 Rustin170506

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. component/sink Sink component. kafka type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
2 participants
@Rustin170506 @nongfushanquan