Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][connector] add reader config to pulsar-io-debezium and pulsar-io-kafka-connect-adaptor #16675

Merged

Conversation

freeznet
Copy link
Contributor

@freeznet freeznet commented Jul 19, 2022

Fixes #16674

Motivation

Allow user customize the reader configs in pulsar-io-debezium and pulsar-io-kafka-connect-adaptor.

Modifications

  • add offset.storage.reader.config to PulsarKafkaWorkerConfig
  • add pulsar.reader.config to PulsarDatabaseHistory
  • add utils to pulsar-io-common
  • add pulsar-io-common dependency to pulsar-io-kafka-connect-adaptor
  • add tests

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@freeznet freeznet added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages area/connector labels Jul 19, 2022
@freeznet freeznet self-assigned this Jul 19, 2022
@freeznet freeznet added this to the 2.11.0 milestone Jul 19, 2022
@github-actions github-actions bot added the doc-required Your PR changes impact docs and you will update later. label Jul 19, 2022
@dave2wave dave2wave requested review from dlg99 and nicoloboschi July 19, 2022 15:15
Copy link
Contributor

@codelipenghui codelipenghui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left a minor comment.

@freeznet freeznet force-pushed the freeznet/add-sub-name-for-debezium-reader branch from ed2e9d4 to 1470168 Compare July 21, 2022 01:30
@freeznet
Copy link
Contributor Author

/pulsarbot run-failure-checks

@freeznet freeznet force-pushed the freeznet/add-sub-name-for-debezium-reader branch from 1470168 to 84cee00 Compare July 22, 2022 03:09
@freeznet
Copy link
Contributor Author

/pulsarbot run-failure-checks

@codelipenghui codelipenghui merged commit 9c19976 into apache:master Jul 25, 2022
codelipenghui pushed a commit that referenced this pull request Jul 25, 2022
…ulsar-io-kafka-connect-adaptor` (#16675)

(cherry picked from commit 9c19976)
codelipenghui pushed a commit that referenced this pull request Jul 25, 2022
…ulsar-io-kafka-connect-adaptor` (#16675)

(cherry picked from commit 9c19976)
@momo-jun
Copy link
Contributor

Hi @freeznet, do you have any updates on the required doc?

@Anonymitaet Anonymitaet added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-required Your PR changes impact docs and you will update later. labels Aug 2, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Aug 16, 2022
…ulsar-io-kafka-connect-adaptor` (apache#16675)

(cherry picked from commit 9c19976)
(cherry picked from commit d923c84)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connector cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-complete Your PR changes impact docs and the related docs have been already added. release/2.9.4 release/2.10.2 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add ability to configure subscription name in debezium connector
5 participants