🐛Source-mssql: aligned regular and cdc syncs and its datatype tests #14379

etsybaev · 2022-07-03T13:17:37Z

What

Currently, we have 2 different classes for integration datatype tests CdcMssqlSourceDatatypeTest and MssqlSourceDatatypeTest. They have the same input args, but differ "expected result" for some types.
Some of Cdc types were not supported at all.

How

Aligned regular and cdc syncs and its datatype tests + added support for some missed new types in CDC migration.

🚨 User Impact 🚨

Shouldn't introduce any breaking changes.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…tests

…r-and-cdc-syncs

etsybaev · 2022-07-04T07:20:57Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2608386871
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2608386871
No Python unittests run

Build Passed

Test summary info:

All Passed

…r-and-cdc-syncs

etsybaev · 2022-07-05T14:12:31Z

Created a follow-up ticket
https://app.zenhub.com/workspaces/java-connectors-625f0102b11304001e0f6c6e/issues/airbytehq/airbyte/14411

etsybaev · 2022-07-05T14:13:06Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2616751722
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2616751722
No Python unittests run

Build Passed

Test summary info:

All Passed

grishick · 2022-07-06T01:19:00Z

@etsybaev could you please separate functional changes that add support for new types from the refactoring of tests and split this into two PRs:

add support for some missed new types in CDC migration
test refactoring

etsybaev · 2022-07-06T07:27:28Z

@etsybaev could you please separate functional changes that add support for new types from the refactoring of tests and split this into two PRs:

add support for some missed new types in CDC migration

test refactoring

hi @grishick .
A lot of tests will start failing if I remove the logic to support new datatypes (that never worked before for CDC). Because now we have the same tests for both normal and CDC migrations. I had to add it to make tests passed on both migrations types.
It would also take me lot's of additional time. As I firstly would need to revert all the changes, spend some time for fixing CDC tests. Wait for its merging, then again spend some time for merging both types into a single tests class and again wait for it to merge.
But if you insist I will do it.
Thanks

…r-and-cdc-syncs

etsybaev · 2022-07-14T19:07:13Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2672544357
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2672544357
No Python unittests run

Build Passed

Test summary info:

All Passed

etsybaev · 2022-07-14T20:33:08Z

/publish connector=connectors/source-mssql

🕑 Publishing the following connectors:
connectors/source-mssql
https://github.com/airbytehq/airbyte/actions/runs/2672979382

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grishick · 2022-07-14T22:59:59Z

@etsybaev don't forget to update and publish source-mssql-strict-encrypt

etsybaev · 2022-07-15T06:25:27Z

/test connector=source-mssql-strict-encrypt

🕑 source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2675045483
✅ source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2675045483
No Python unittests run

Build Passed

Test summary info:

All Passed

etsybaev · 2022-07-15T06:53:12Z

/test connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2675160863
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2675160863
No Python unittests run

Build Passed

Test summary info:

All Passed

etsybaev · 2022-07-15T07:07:30Z

/publish connector=connectors/source-mssql-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-mssql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2675226698

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

etsybaev · 2022-07-15T12:22:16Z

/publish connector=connectors/source-mssql-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-mssql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2676810126

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql-strict-encrypt	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

etsybaev · 2022-07-15T12:35:24Z

As per logs from the second attempt the strict-encrypt image had been published from the first attempt

…rbytehq-master * 'master' of https://github.com/airbytehq/airbyte: (1141 commits) pass USE_STREAM_CAPABLE_STATE env var to containers/deployments (airbytehq#14737) Bump mqtt connector (airbytehq#14648) Add error code to ManualOperationResult (airbytehq#14657) Bump elasticsearch version (airbytehq#14640) Ryan/sync oracle version number (airbytehq#14736) Fixed linter issue with add_fields.py comments (airbytehq#14742) 🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (airbytehq#14494) 🐛Source-mssql: aligned regular and cdc syncs and its datatype tests (airbytehq#14379) 🎉 Source Amazon Seller Partner: Add new streams (airbytehq#13604) bump source-file-secure (airbytehq#14704) 🎉 New source: Timely airbytehq#13292 (airbytehq#14335) 🪟🔧 Refactor feature service (airbytehq#14559) [low code cdk] add a transformation for adding fields into an outgoing record (airbytehq#14638) Bump destination-postgres to 0.3.21 (airbytehq#14479) Remove `additionalProperties: false` from JDBC destination connectors (airbytehq#14618) 🎉 Source Notion: add OAuth authorization for source-notion connector (airbytehq#14706) Use the configuration diff calculation in the update endpoint (airbytehq#14626) 🪟 🐛 Fix input validation on blur and cleanup signup error handling (airbytehq#14724) lower sleep after wait for successful job (airbytehq#14725) Add configuration diff (airbytehq#14603) ...

[13606] source-mssql: aligned regular and cdc syncs and its datatype …

ed6147f

…tests

octavia-squidington-iii added the connectors/source/mssql label Jul 3, 2022

github-actions bot added the area/connectors Connector related issues label Jul 3, 2022

etsybaev temporarily deployed to more-secrets July 3, 2022 13:19 Inactive

Merge branch 'master' into etsybaev/13606-source-mssql-aligned-regual…

910586c

…r-and-cdc-syncs

etsybaev linked an issue Jul 4, 2022 that may be closed by this pull request

Source-mssql: align regular and CDC integration tests and data mappers #13606

Closed

etsybaev requested a review from DoNotPanicUA July 5, 2022 07:23

DoNotPanicUA approved these changes Jul 5, 2022

View reviewed changes

etsybaev added 2 commits July 5, 2022 15:28

Merge branch 'master' into etsybaev/13606-source-mssql-aligned-regual…

ba21d99

…r-and-cdc-syncs

[13606] source-mssql: updated doc

f009968

github-actions bot added the area/documentation Improvements or additions to documentation label Jul 5, 2022

etsybaev temporarily deployed to more-secrets July 5, 2022 14:13 Inactive

etsybaev changed the title ~~[13606] source-mssql: aligned regular and cdc syncs and its datatype …~~ 🐛Source-mssql: aligned regular and cdc syncs and its datatype tests + added support for new types Jul 5, 2022

etsybaev marked this pull request as ready for review July 5, 2022 14:23

[13606] source-mssql: updated doc

49e7569

etsybaev temporarily deployed to more-secrets July 5, 2022 14:29 Inactive

etsybaev requested a review from grishick July 5, 2022 14:47

etsybaev temporarily deployed to more-secrets July 5, 2022 21:05 Inactive

etsybaev mentioned this pull request Jul 8, 2022

🐛Source-mssql: added support for some datatypes #14449

Closed

37 tasks

grishick approved these changes Jul 14, 2022

View reviewed changes

Merge branch 'master' into etsybaev/13606-source-mssql-aligned-regual…

67b669d

…r-and-cdc-syncs

etsybaev requested a review from a team as a code owner July 14, 2022 18:36

etsybaev temporarily deployed to more-secrets July 14, 2022 18:38 Inactive

etsybaev changed the title ~~🐛Source-mssql: aligned regular and cdc syncs and its datatype tests + added support for new types~~ 🐛Source-mssql: aligned regular and cdc syncs and its datatype tests Jul 14, 2022

bumped version

f87ffc8

etsybaev temporarily deployed to more-secrets July 14, 2022 20:34 Inactive

auto-bump connector version

fa2c664

octavia-squidington-iii temporarily deployed to more-secrets July 14, 2022 21:37 Inactive

bumped version for strict-encrypt

5db6a8a

octavia-squidington-iii added the connectors/source/mssql-strict-encrypt label Jul 15, 2022

etsybaev temporarily deployed to more-secrets July 15, 2022 06:53 Inactive

etsybaev merged commit 95b3424 into master Jul 15, 2022

etsybaev deleted the etsybaev/13606-source-mssql-aligned-regualr-and-cdc-syncs branch July 15, 2022 12:36

octavia-squidington-iii mentioned this pull request Jul 23, 2022

Bump Airbyte version from 0.39.37-alpha to 0.39.38-alpha #14976

Merged

xiaohansong mentioned this pull request Dec 21, 2023

Source MSSQL: Connector provides binary values instead of string for hierarchyid, geometry and geography datatypes #7739

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛Source-mssql: aligned regular and cdc syncs and its datatype tests #14379

🐛Source-mssql: aligned regular and cdc syncs and its datatype tests #14379

etsybaev commented Jul 3, 2022 •

edited

Loading

etsybaev commented Jul 4, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 5, 2022

etsybaev commented Jul 5, 2022 •

edited by github-actions bot

Loading

grishick commented Jul 6, 2022

etsybaev commented Jul 6, 2022

etsybaev commented Jul 14, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 14, 2022 •

edited by github-actions bot

Loading

grishick commented Jul 14, 2022

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022

🐛Source-mssql: aligned regular and cdc syncs and its datatype tests #14379

🐛Source-mssql: aligned regular and cdc syncs and its datatype tests #14379

Conversation

etsybaev commented Jul 3, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

etsybaev commented Jul 4, 2022 • edited by github-actions bot Loading

Build Passed

etsybaev commented Jul 5, 2022

etsybaev commented Jul 5, 2022 • edited by github-actions bot Loading

Build Passed

grishick commented Jul 6, 2022

etsybaev commented Jul 6, 2022

etsybaev commented Jul 14, 2022 • edited by github-actions bot Loading

Build Passed

etsybaev commented Jul 14, 2022 • edited by github-actions bot Loading

grishick commented Jul 14, 2022

etsybaev commented Jul 15, 2022 • edited by github-actions bot Loading

Build Passed

etsybaev commented Jul 15, 2022 • edited by github-actions bot Loading

Build Passed

etsybaev commented Jul 15, 2022 • edited by github-actions bot Loading

etsybaev commented Jul 15, 2022 • edited by github-actions bot Loading

etsybaev commented Jul 15, 2022

etsybaev commented Jul 3, 2022 •

edited

Loading

etsybaev commented Jul 4, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 5, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 14, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 14, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jul 15, 2022 •

edited by github-actions bot

Loading