Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Common: remove polymorphic fields from bad rows schemas #1075

Open
istreeter opened this issue Sep 4, 2020 · 1 comment
Open

Common: remove polymorphic fields from bad rows schemas #1075

istreeter opened this issue Sep 4, 2020 · 1 comment

Comments

@istreeter
Copy link
Contributor

Some of the bad rows schemas contain "polymorphic" fields, e.g. the field value can be an object or a string, depending on the context. This causes problems for querying the badrows in BigQuery, where fields must have a single well-defined type.

The schemas should be simplified so that each field has only one type.

Here is a list of the affected fields:

  • Enrichment failures: data.failure.message.error
  • Loader iglu error: data.failure.dataReports.targets
  • Loader recovery error: data.failure.error.location
  • Schema violations: data.failure.messages.error
  • Tracker protocol violations: data.failure.messages.error
@istreeter
Copy link
Contributor Author

See relevant ticket in badrows

@chuwy chuwy changed the title Remove polymorphic fields from bad rows schemas Common: remove polymorphic fields from bad rows schemas Sep 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant