BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files

[DoNotPanicUA]

What

Decrease complexity of the BigQuery destinations and add possibility to load data using Avro GCS.

How

Reorganize current implementation to use different implementation instead of code duplication.
Make Avro file loader in line with BigQuery loader.
Reorganize JsonSchema flow in order to generate BigQuery schema and Avro file schema using already modified schema.

Recommended reading order

1. AbstractBigQueryUploader.java
2. BigQueryRecordFormatter.java
3. BigQueryDirectUploader.java
4. BigQueryTableWriter.java
5. CommonWriter.java
6. GcsAvroWriter.java
7. GcsAvroBigQueryUploader.java
8. DefaultBigQueryDenormalizedRecordFormatter.java
9. GcsBigQueryDenormalizedRecordFormatter.java
10. BigQueryDenormalizedDestination.java

[alexandr-shegeda]

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1602453679
✅ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1602453679
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     124      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 494    313    37%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         146     32    78%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1199    501    58%

[alexandr-shegeda]

/test connector=destination-bigquery-denormalized

🕑 destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1602454626
✅ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1602454626
No Python unittests run

[sherifnada]

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1602905932
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1602905932

[alexandr-shegeda]

/publish connector=connectors/destination-bigquery

Error: No ref found for: aleonets/oncall27-bigq

[alexandr-shegeda]

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1606794585
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1606794585