Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the Kafka buffer (and others that do not require the heap) to bypass the heap circuit breaker #3616

Closed
dlvenable opened this issue Nov 9, 2023 · 0 comments · Fixed by #3619
Assignees
Labels
enhancement New feature or request plugin - buffer A plugin for buffering incoming data
Milestone

Comments

@dlvenable
Copy link
Member

Is your feature request related to a problem? Please describe.

The kafka buffer is currently subject to the heap circuit breaker on receipt of data. Since this buffer writes directly to a Kafka topic, and thus leaves the JVM heap, it shouldn't have the heap circuit-breaker impact it.

Describe the solution you'd like

Provide a new function on Buffer to indicate if the buffer uses off-heap data.

boolean isWrittenOffHeapOnly()

When this is true, then do not apply the CircuitBreaker. This initial logic may not be ideal long-term as we could possibly support other types of circuit-breakers (say based on local disk-space, a CPU-based threshold, or the current pipeline latency). However, this will work for an initial implementation.

Additional context

The kafka buffer already uses the circuit-breaker when reading from the topic to place into the processor chains - #3578.

@dlvenable dlvenable added enhancement New feature or request plugin - buffer A plugin for buffering incoming data labels Nov 9, 2023
@dlvenable dlvenable added this to the v2.6 milestone Nov 9, 2023
@dlvenable dlvenable self-assigned this Nov 9, 2023
@dlvenable dlvenable moved this from Unplanned to In progress in Data Prepper Tracking Board Nov 9, 2023
dlvenable added a commit to dlvenable/data-prepper that referenced this issue Nov 9, 2023
…. This is currently only the Kafka buffer. Resolves opensearch-project#3616

Signed-off-by: David Venable <[email protected]>
dlvenable added a commit that referenced this issue Nov 10, 2023
#3619)

Disable the circuit breaker for buffers that write data off-heap only. This is currently only the Kafka buffer. Resolves #3616

Signed-off-by: David Venable <[email protected]>
@github-project-automation github-project-automation bot moved this from In progress to Done in Data Prepper Tracking Board Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin - buffer A plugin for buffering incoming data
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant