Remove json avro schema converter hack #9232

tuliren · 2021-12-31T02:09:08Z

What

In BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files #8788, a hack is introduced to the Json Avro schema converter.
The BigQuery uploader converts the Json schema to both Avro and BigQuery schema.
- The Avro schema is used to convert Json objects to Avro objects.
- The BigQuery schema is used to create the BigQuery table.
The correct process should be:
1. Json schema -> Avro schema
2. Json schema -> BigQuery schema
The BigQuery uploader incorrectly uses the BigQuery schema as the input for 1.
- So it is Json schema -> BigQuery schema -> Avro schema
This is problematic because BigQuery schema is actually similar to the Avro schema.
- The date-time format in the Json schema is changed to timestamp-micros.
But the Json to Avro schema converter expects a pure Json schema.
Consequently, the Json to Avro schema converter needs to handle the Avro format timestamp-micros, which should not be necessarily.

How

This PR fixes this hack by using the correct Json schema as the input to the Json to Avro schema converter.

Review Order

GcsBigQueryDenormalizedRecordFormatter.java
JsonToAvroSchemaConverter.java.
All other changes are automatic changes that add finals by the IDE.

Checkline

This PR does not change any functionality. No connector publishing is needed.

tuliren · 2021-12-31T02:10:27Z

...byte/integrations/destination/bigquery/formatter/GcsBigQueryDenormalizedRecordFormatter.java

-     * https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#logical_types Replace
-     * date-time by timestamp
-     */
-    textJson = textJson.replace("\"format\":\"date-time\"", "\"format\":\"timestamp-micros\"");


This is moved to getBigQuerySchema because this is only needed in the BigQuery schema.

tuliren · 2021-12-31T02:10:55Z

...-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java

@@ -201,7 +201,6 @@ Schema getSingleFieldType(final String fieldName,
        if (fieldDefinition.has("format")) {
          final String format = fieldDefinition.get("format").asText();
          fieldSchema = switch (format) {
-            case "timestamp-micros" -> LogicalTypes.timestampMicros().addToSchema(Schema.create(Schema.Type.LONG));


This was the hack that should not be added in the first place. Json schema does not have such format. This is an Avro format.

tuliren · 2021-12-31T04:12:06Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1639692275
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1639692275
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     124      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 494    313    37%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         146     32    78%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1199    501    58%

tuliren · 2021-12-31T04:12:14Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1639692402
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1639692402
🐛

tuliren · 2021-12-31T07:06:37Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1640030146
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1640030146
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     124      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 494    313    37%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         146     32    78%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1199    501    58%

tuliren · 2021-12-31T07:06:44Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1640030503
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1640030503
No Python unittests run

Remove hack code

da4c6ae

tuliren linked an issue Dec 31, 2021 that may be closed by this pull request

S3 Destination : Json to Avro converter should change format of date-time to timestamp-micros #8886

Closed

github-actions bot added the area/connectors Connector related issues label Dec 31, 2021

tuliren commented Dec 31, 2021

View reviewed changes

tuliren mentioned this pull request Dec 31, 2021

S3 Destination : Json to Avro converter should change format of date-time to timestamp-micros #8886

Closed

Format code

8eca092

tuliren requested a review from sherifnada December 31, 2021 02:16

tuliren temporarily deployed to more-secrets December 31, 2021 02:18 Inactive

Update log message

fc95395

tuliren requested a review from DoNotPanicUA December 31, 2021 02:23

tuliren temporarily deployed to more-secrets December 31, 2021 02:24 Inactive

jrhizor temporarily deployed to more-secrets December 31, 2021 04:14 Inactive

Limit change to gcs formatter

66e8616

tuliren temporarily deployed to more-secrets December 31, 2021 07:07 Inactive

jrhizor temporarily deployed to more-secrets December 31, 2021 07:08 Inactive

tuliren merged commit e3b727b into master Dec 31, 2021

tuliren deleted the liren/separate-json-avro-converter branch December 31, 2021 07:31

jrhizor mentioned this pull request Jan 3, 2022

Bump Airbyte version from 0.35.2-alpha to 0.35.3-alpha #9262

Merged

karinakuz added connectors/destination/bigquery connectors/destinations-api connectors/destinations-warehouse and removed connectors/destinations-api labels Jan 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove json avro schema converter hack #9232

Remove json avro schema converter hack #9232

tuliren commented Dec 31, 2021 •

edited

Loading

tuliren Dec 31, 2021 •

edited

Loading

tuliren Dec 31, 2021 •

edited

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

Remove json avro schema converter hack #9232

Remove json avro schema converter hack #9232

Conversation

tuliren commented Dec 31, 2021 • edited Loading

What

How

Review Order

Checkline

tuliren Dec 31, 2021 • edited Loading

Choose a reason for hiding this comment

tuliren Dec 31, 2021 • edited Loading

Choose a reason for hiding this comment

tuliren commented Dec 31, 2021 • edited by github-actions bot Loading

tuliren commented Dec 31, 2021 • edited by github-actions bot Loading

tuliren commented Dec 31, 2021 • edited by github-actions bot Loading

tuliren commented Dec 31, 2021 • edited by github-actions bot Loading

tuliren commented Dec 31, 2021 •

edited

Loading

tuliren Dec 31, 2021 •

edited

Loading

tuliren Dec 31, 2021 •

edited

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 31, 2021 •

edited by github-actions bot

Loading