Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cdc): init default value for cdc table columns #19354

Merged
merged 14 commits into from
Nov 26, 2024

Conversation

StrikeW
Copy link
Contributor

@StrikeW StrikeW commented Nov 12, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Read external db to init default value for cdc table columns

related: #19319

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

let config = ExternalTableConfig::try_from_btreemap(options, secret_refs)
.context("failed to extract external table config")?;

let table = ExternalTableImpl::connect(config)
Copy link
Contributor Author

@StrikeW StrikeW Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read external catalog will cause the create_source_with_cdc_backfill planner test fail, because we didn't setup a upstream db for planner test and it cannot connect to it. @st1page any idea to work around the issue?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can add a new type of ExternalTableImpl (MockCDC) and CdcSourceType (mock-cdc) only for testing purpose. This may help for other tests as well.

Copy link
Contributor

@st1page st1page Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you take a try to test it in the e2e test with the explain statment?

CREATE TABLE test_default_value (
"id" int,
"name" varchar(64),
"city" varchar(200) default 'Shanghai',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why we only support inferring default value from pg not mysql?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added test for mysql too.

let config = ExternalTableConfig::try_from_btreemap(options, secret_refs)
.context("failed to extract external table config")?;

let table = ExternalTableImpl::connect(config)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can add a new type of ExternalTableImpl (MockCDC) and CdcSourceType (mock-cdc) only for testing purpose. This may help for other tests as well.

This reverts commit 0b82adc.
Copy link
Contributor

@st1page st1page left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in this PR we need to also ban the user specified default column in the create table statement

@StrikeW
Copy link
Contributor Author

StrikeW commented Nov 19, 2024

I think in this PR we need to also ban the user specified default column in the create table statement

+1, I'll do it.

@StrikeW StrikeW changed the title feat(pg-cdc): init default value for cdc table columns feat(cdc): init default value for cdc table columns Nov 19, 2024
@KeXiangWang KeXiangWang force-pushed the siyuan/fix-pg-creat-table branch from 659b8ea to f95a8ae Compare November 19, 2024 21:15
@graphite-app graphite-app bot requested a review from a team November 19, 2024 21:48
CREATE TABLE test_default_value (
"id" int,
"name" varchar(64),
"city" varchar(200) default 'Shanghai',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added test for mysql too.

None => {
for column_def in &column_defs {
for option_def in &column_def.options {
if let ColumnOption::DefaultColumns(_) = option_def.option {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ban default value for columns when user is creating a cdc table.

@StrikeW StrikeW requested a review from st1page November 25, 2024 03:55
@StrikeW StrikeW enabled auto-merge November 26, 2024 08:48
@StrikeW StrikeW added this pull request to the merge queue Nov 26, 2024
Merged via the queue into main with commit 759ea19 Nov 26, 2024
29 of 30 checks passed
@StrikeW StrikeW deleted the siyuan/fix-pg-creat-table branch November 26, 2024 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants