More robust data type & schema permutations testing in Destination Acceptance Tests

[sherifnada]

Tell us about the problem you're trying to solve

DAT provides some good guarantees that a destination works as intended. However, it does not do a thorough verification that any data type expressable via JsonSchema can be written to a destination. For example, we've had some issues recently related to data types and schema declarations in JsonSchema:

1. Json -> Avro / Parquet schema convert should support array field without items #7939 valid json schema declaration which failed in destination s3
2. 🎉 Destination S3 & GCS: support additional properties #7288 valid json schema declaration which failed in destination s3
3. NullPointerException while sync when source-s3 CSV -> destination-s3 PARQUET #6871 valid json schema declaration which failed in destination s3

Describe the solution you’d like

I would like to test all possible combinations of valid json schema declarations in DAT so that we can proactively verify that destinations can handle data coming from sources, rather than play whack-a-mole.

[sherifnada]

relevant: https://docs.airbyte.io/understanding-airbyte/json-avro-conversion#untyped-array
example of something not currently covered in the type system: #8280

potential steps:

1. defining the subset of JSONSchema that all destinations should be able to support
2. coding it into DAT

[edgao]

self notes:

• normalization supported types https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py
  • string, timestamp with timezone (string && format contains date-time), date (string && format contains date), number, integer, boolean, array, object
• json/avro converter - also supports null

[sherifnada]

goals for this week:

• wrap up design
• create project plan

[sherifnada]

tech spec was created, epic here: https://github.com/airbytehq/airbyte-internal-issues/issues/371