Normalization handles quote in column names #5027
Conversation
| @@ -8,6 +8,7 @@ select | |||
| _airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid, | |||
| json_extract_array(`partition`, "$['double_array_data']") as double_array_data, | |||
| json_extract_array(`partition`, "$['DATA']") as DATA, | |||
| json_extract_array(`partition`, "$['column___with__quotes']") as column___with__quotes, | |||
There was a problem hiding this comment.
Thanks to PR #5026, the raw json blob in BigQuery also contains some sanitized columns that we can extract from where quote characters have already been replaced by _
| '$."DATA"') as `DATA`, | ||
| '$."DATA"') as `DATA`, | ||
| json_extract(`partition`, | ||
| '$."column___with__quotes"') as `column__'with"_quotes`, |
There was a problem hiding this comment.
Thanks to PR #5026, the raw json blob in MySql also contains some sanitized columns that we can extract from where quote characters have already been replaced by _
Notice that the json_extract can't handle quotes in the json path but it is possible to create column names using ' or " in MySQL...
github-actions
bot
added
area/documentation
| @@ -6,6 +6,7 @@ select | |||
| _airbyte_nested_stre__nto_long_names_hashid, | |||
| jsonb_extract_path("partition", 'double_array_data') as double_array_data, | |||
| jsonb_extract_path("partition", 'DATA') as "DATA", | |||
| jsonb_extract_path("partition", 'column`_''with"_quotes') as "column`_'with""_quotes", | |||
There was a problem hiding this comment.
Postgres is able to parse the json blob just fine. It just has an edge case of doubling the ' characters to "escape" it...
| @@ -6,6 +6,7 @@ select | |||
| _AIRBYTE_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_HASHID, | |||
| get_path(parse_json(PARTITION), '"double_array_data"') as DOUBLE_ARRAY_DATA, | |||
| get_path(parse_json(PARTITION), '"DATA"') as DATA, | |||
| get_path(parse_json(PARTITION), '"column`_''with""_quotes"') as "column`_'with""_quotes", | |||
There was a problem hiding this comment.
Snowflake is able to parse the json blob just fine. It just has an edge case of doubling the ' or '"' characters to "escape" them...
What
This PR depends on some changes made in bigquery/mysql destinations: #5026
Closes #4729
To avoid noise, output test files are regenerated and part of another PR #5028
How
Parse JSON blob while managing quote characters depending on the destination
Recommended reading order
x.javay.pythonPre-merge Checklist
Expand the checklist which is relevant for this PR.
Connector checklist
airbyte_secretin the connector's spec./gradlew :airbyte-integrations:connectors:<name>:integrationTest./test connector=connectors/<name>command as documented here is passing.README.mddocs/SUMMARY.mdif it's a new connectordocs/integrations/<source or destination>/<name>.docs/integrations/.... See changelog exampledocs/integrations/README.mdcontains a reference to the new connector/publishcommand described hereConnector Generator checklist
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changes