Destination S3 to Parquet/Avro: Handle empty objects with additionalProperties=true

[blotouta2]

Enviroment

• Airbyte version: example is 0.25.0-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: (if applicable example Salesforce 0.2.3) Slack 0.2.3
• Destination Connector and version: (if applicable example Postgres 0.3.3) Destination S3 0.1.7
• Severity: High
• Step where error happened: Sync job

Current Behavior

i was trying to send data from facebook marketing to destination s3 in parquet format but it give me error. it works well with S3 CSV format previously.

Expected Behavior

Data will be sync in parquet format.

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.
slack-s3-parquet.txt

LOG

Logs are attached as txt file.

[sherifnada]

Schema of the faulty object (reference):

"pinned_info": {
  "type": [
    "null",
    "object"
  ],
  "additionalProperties": true,
  "properties": {}
}

[tuliren]

The root cause is that pinned_info does not have predefined properties. Parquet does not like (is not compatible with) it, and currently we don't support additionalProperties. A related issue and potential solution is here: #4124.

[tuliren]

Ping Rytis Zolubas when this is done.

[sherifnada]

Desired behvavior:

if there is any data that is not present in the schema, we should put them in a hashmap field additionalProperties and whenever we see extra fields, we serialize them as string and put them in that field