Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

common: update the validation of the enriched event with static field lengths #608

Closed
benjben opened this issue May 25, 2022 · 0 comments
Closed

Comments

@benjben
Copy link
Contributor

benjben commented May 25, 2022

In 93078e2 we've introduced the validation of the enriched event according to atomic schema before emitting.

The implementation is to convert the enriched event to a JSON and to use Iglu client to validate it against the schema. Being based on JSON manipulation, we've observed a significant bump in the use of CPU by the enriching step of the app; ~ x5 in case no enrichments are activated (other steps like reading from/writing to message queue being not impacted).

For this reason, for now we'd like to limit the validation to the validation of each field length (which was the original problem we were trying to solve), like this is done in analytics-sdk.

When this issue will be implemented, we'll see to add the full validation with Iglu client again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant