Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reuse transformer in stream indexing #9625

Merged
merged 3 commits into from
Apr 6, 2020
Merged

Reuse transformer in stream indexing #9625

merged 3 commits into from
Apr 6, 2020

Conversation

jihoonson
Copy link
Contributor

Description

Initializing a Transformer is not cheap; it includes parsing an expression. The Transformer should be reused in stream indexing rather than recreating it for every stream chunk. Batch ingestion doesn't have this issue since it decorates an InputSourceReader with a Transformer which is a reader for the entire input data.

Also fixed JSON serde of RegexInputFormat.

This PR has:

  • been self-reviewed.
  • [] added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.
  • been tested in a test Druid cluster.

@gianm gianm added this to the 0.18.0 milestone Apr 6, 2020
@gianm gianm merged commit 82ce60b into apache:master Apr 6, 2020
jihoonson added a commit to jihoonson/druid that referenced this pull request Apr 7, 2020
@jihoonson
Reuse transformer in stream indexing (apache#9625)
9a8a862 
* Reuse transformer in stream indexing

* remove unused method

* memoize complied pattern
@jihoonson jihoonson mentioned this pull request Apr 7, 2020
Merged
fjy pushed a commit that referenced this pull request Apr 7, 2020
@jihoonson
Reuse transformer in stream indexing (#9625) (#9639)
4df4eac 
* Reuse transformer in stream indexing

* remove unused method

* memoize complied pattern
This was referenced Apr 19, 2020
Closed
Closed
JulianJaffePinterest pushed a commit to JulianJaffePinterest/druid that referenced this pull request Jun 12, 2020
@jihoonson @JulianJaffePinterest
Reuse transformer in stream indexing (apache#9625)
46ade9c 
* Reuse transformer in stream indexing

* remove unused method

* memoize complied pattern
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gianm gianm gianm approved these changes

Assignees
No one assigned
Labels
Area - Streaming Ingestion Bug Performance
Projects
None yet
Milestone
0.18.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jihoonson @gianm