🐛 Correct `unnest_column_name` conflict with `table_alias` #5467

marcosmarxm · 2021-08-17T16:16:10Z

What

Closes #5462

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

marcosmarxm · 2021-08-17T16:17:35Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1140015388
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1140015388

airbyte-integrations/bases/base-normalization/integration_tests/test_normalization.py

...ions/bases/base-normalization/dbt-project-template/macros/cross_db_utils/json_operations.sql

marcosmarxm · 2021-08-19T14:17:37Z

...te-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py

+        if "unnested_column_value" in json_column_name and unnest_dest_alias:
+            table_alias = ''


@ChristopheDuong wdyt?

marcosmarxm · 2021-08-19T18:36:10Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1148090967
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1148090967

marcosmarxm · 2021-08-19T18:58:13Z

...y_key_streams/models/generated/airbyte_ctes/test_normalization/unnest_alias_children_ab1.sql

+    {{ json_extract('', unnested_column_value('children'), ['owner'], ['owner']) }} as {{ adapter.quote('owner') }},
+    _airbyte_emitted_at
+from {{ ref('unnest_alias') }} as table_alias
+{{ cross_join_unnest('unnest_alias', 'children') }}


@ChristopheDuong the new stream to test the condition

marcosmarxm · 2021-08-19T19:43:07Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1148275116
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1148275116

marcosmarxm · 2021-08-19T20:38:21Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1148425507
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1148425507

* add unnest_column case conflict * add redshift files * format * change logic * change logic for unnest * bump normalization version * add files * add stream test unnest_alias

add unnest_column case conflict

0eaf195

github-actions bot added the normalization label Aug 17, 2021

jrhizor temporarily deployed to more-secrets August 17, 2021 16:20 Inactive

marcosmarxm added 2 commits August 17, 2021 15:15

add redshift files

3e66b37

format

cd22347

github-actions bot added the area/connectors Connector related issues label Aug 17, 2021

ChristopheDuong reviewed Aug 18, 2021

View reviewed changes

airbyte-integrations/bases/base-normalization/integration_tests/test_normalization.py Outdated Show resolved Hide resolved

ChristopheDuong reviewed Aug 18, 2021

View reviewed changes

...ions/bases/base-normalization/dbt-project-template/macros/cross_db_utils/json_operations.sql Outdated Show resolved Hide resolved

change logic

ec9e5d4

marcosmarxm mentioned this pull request Aug 19, 2021

Stripe Source can't sync Customers on 0.29.9-alpha - Normalization Failed #5515

Closed

change logic for unnest

50f2b05

marcosmarxm requested a review from ChristopheDuong August 19, 2021 14:17

marcosmarxm commented Aug 19, 2021

View reviewed changes

Merge branch 'master' into marcosmarxm/normalization-table-not-exist

50fa78e

github-actions bot removed the area/connectors Connector related issues label Aug 19, 2021

bump normalization version

8c1a5ba

marcosmarxm mentioned this pull request Aug 19, 2021

Instagram Connector: Airbyte Reset/Upgrade and Connector Failing #5435

Closed

add files

9d8cad4

github-actions bot added the area/worker Related to worker label Aug 19, 2021

add stream test unnest_alias

69652b3

jrhizor temporarily deployed to more-secrets August 19, 2021 18:38 Inactive

marcosmarxm commented Aug 19, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets August 19, 2021 19:46 Inactive

jrhizor temporarily deployed to more-secrets August 19, 2021 20:40 Inactive

ChristopheDuong approved these changes Aug 20, 2021

View reviewed changes

marcosmarxm merged commit a9b2c08 into master Aug 20, 2021

marcosmarxm deleted the marcosmarxm/normalization-table-not-exist branch August 20, 2021 14:09

marcosmarxm changed the title ~~Add condition for unnest_column_name for pg/redshift/mysql~~ 🐛 Add condition for unnest_column_name Aug 20, 2021

marcosmarxm changed the title ~~🐛 Add condition for unnest_column_name~~ 🐛 Correct unnest_column_name conflict with table_alias Aug 20, 2021

marcosmarxm mentioned this pull request Aug 23, 2021

Zendesk support new connector is not working properly #5566

Closed

davinchia mentioned this pull request Aug 25, 2021

Kube attempts not catching connector exception properly #5526

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Correct `unnest_column_name` conflict with `table_alias` #5467

🐛 Correct `unnest_column_name` conflict with `table_alias` #5467

marcosmarxm commented Aug 17, 2021

marcosmarxm commented Aug 17, 2021 •

edited by github-actions bot

Loading

marcosmarxm Aug 19, 2021

marcosmarxm commented Aug 19, 2021 •

edited by github-actions bot

Loading

marcosmarxm Aug 19, 2021

marcosmarxm commented Aug 19, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 19, 2021 •

edited by github-actions bot

Loading

		if "unnested_column_value" in json_column_name and unnest_dest_alias:
		table_alias = ''

🐛 Correct unnest_column_name conflict with table_alias #5467

🐛 Correct unnest_column_name conflict with table_alias #5467

Conversation

marcosmarxm commented Aug 17, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

marcosmarxm commented Aug 17, 2021 • edited by github-actions bot Loading

marcosmarxm Aug 19, 2021

Choose a reason for hiding this comment

marcosmarxm commented Aug 19, 2021 • edited by github-actions bot Loading

marcosmarxm Aug 19, 2021

Choose a reason for hiding this comment

marcosmarxm commented Aug 19, 2021 • edited by github-actions bot Loading

marcosmarxm commented Aug 19, 2021 • edited by github-actions bot Loading

🐛 Correct `unnest_column_name` conflict with `table_alias` #5467

🐛 Correct `unnest_column_name` conflict with `table_alias` #5467

marcosmarxm commented Aug 17, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 19, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 19, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 19, 2021 •

edited by github-actions bot

Loading