Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: log parser speed improvements #475

Closed
lcottercertinia opened this issue Jan 16, 2024 · 1 comment
Closed

feat: log parser speed improvements #475

lcottercertinia opened this issue Jan 16, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request epic
Milestone

Comments

@lcottercertinia
Copy link
Collaborator

lcottercertinia commented Jan 16, 2024

Is your feature request related to a problem?

We want the parser to be as fast as possible especially to give head room for adding new features.

Describe the solution you'd like

  1. Generator to reduce looping and heap usage
  2. Only substring / slice when needed (remove String.split) (Creating new objects , string, array is expensive)
  3. indexOf is our friend to avoid unnecessary string creation

Parser performance improvement from v1.12.1 for v1.14.0 pre release

Testing shows between 1.5 and 2.7 times faster
This is mainly down to

  • using a generator to create the tokens on demand and keep memory low as well and looping lines only once. This reduces GC overhead which is expensive.
  • cache namespaces to avoid having to re parse namespace strings which is expensive
  • aggregate up the tree using a map for depth rather than recursion. Recursion is very very expensive.
@lcottercertinia
Copy link
Collaborator Author

2 to 3 times faster parser performance between v1.12.0 and v1.14.0 pre release.
That is so far. We expect a few more minor improvements before the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request epic
Projects
None yet
Development

No branches or pull requests

1 participant