Incomplete events schema

[colmsnowplow]

No description provided.

[colmsnowplow]

https://www.notion.so/Failure-Entity-Schema-Design-6361207b9801468ca8ac2bbf448c81f0?d=ad33201aefb0406b90dd6a916757ae21&pvs=4#29ce10ca04ad4648912dd079d7020768

cc @stanch

[colmsnowplow]

https://www.notion.so/Failure-Entity-Schema-Design-6361207b9801468ca8ac2bbf448c81f0?d=2405686971304cf2a32fa25e0e9e3e6b&pvs=4#831e46e694814714a68ec1ae5d7727dc

[colmsnowplow]

Initially the idea was to have defined fields, but we changed it to just an empty object because of how BQ will load data - any defined fields will create a static definition of the object, which is a pain to change, and additional properties would be ignored. An empty object, however, will just load what it gets.

This isn't necessarily the ideal scenario. We should consider separate fields for error message (array of strings), and error details (flexible object), but that design has downsides too.

[oguzhanunlu]

enum for safer usage, wdyt @colmsnowplow ?

[colmsnowplow]

No objection in principal, but need to validate that changing the enum in future won't result in any problems.

[miike]

Enum can still be a little awkward with Redshift as I don't believe we bump the maxLength as of yet.

[miike]

I would add something like an error_code. At the moment we put a lot of description elements into the failed event schemas but these aren't apparent as to what the failure relates to unless you actually add it into the schema. The way I've been thinking about this is to provide a high level classification e.g.,

D - the failure is related to data that has been sent with the event
S - the failure is related to an issue with a schema sent with the event
I - the failure is related to an internal processing error (e.g., pipeline issue)
E - the failure is related to an external processing issue (e.g., failed to get a response from an API enrichment).

[colmsnowplow]

Thanks Mike. Interesting suggestion. @stanch would love to hear what you think.

For incomplete events I'd be inclined to treat it as post-MVP, but depending on how difficult to implement it might be something we could do before rolling out the end product

[colmsnowplow]

@miike now that we have some firmer examples of what the actual data looks like, we tweaked some things. I think something like the requirement you describe will be satisfied by the failureType field.

This will essentially indicate the classification of the error: eg. validation error or schema violation, for example.

(additionally we're releasing 4.2.0 before this one, which fixes things so that schema violations at enrichment level don't get reported as enrichment errors.)

I'm not sure if this would give you everything you're asking for here, so we can still consider it post-MVP

[rlh1994]

I'm confused, are these trackers? App_ids? Or the part of the pipeline?

[colmsnowplow]

Component of the pipeline

[colmsnowplow]

I see the confusion though. Worth considering alternative names I guess. Any suggestions?

[miike]

component / component_version? I think we call this artifact at the moment but component is likely more clear.

[colmsnowplow]

Add failureType here

[colmsnowplow]

The most recent commits, following from a meeting to discuss design after seeing example failure data:

• Define failureType as the classification of the failure, rather than anything to do with where the failure came up (eg. validation error)
• removes source - source is to indicate the field which caused the issue, but this is actually specific to each error rather than general for the whole failure. For example one can have a failure where there is an error for page_url as well as page_referrer.
• Renames unclear fields
• Makes failureType required.