Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe die before failing engine #28973

Merged
merged 4 commits into from
Mar 10, 2018

Conversation

jasontedor
Copy link
Member

@jasontedor jasontedor commented Mar 9, 2018

Today we check for a few cases where we should maybe die before failing the engine (e.g., when a merge fails). However, there are still other cases where a fatal error can be hidden from us (for example, a failed index writer commit). This commit modifies the mechanism for failing the engine to always check for a fatal error before failing the engine.

Relates #27265, closes #28967

Today we check for a few cases where we should maybe die before failing
the engine (e.g., when a merge fails). However, there are still other
cases where a fatal error can be hidden from us (for example, a failed
index writer commit). This commit modifies the mechanism for failing the
engine to always check for a fatal error before failing the engine.
@jasontedor jasontedor added >bug review v7.0.0 v6.3.0 v6.1.4 v6.2.3 :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. v5.6.9 labels Mar 9, 2018
@jasontedor jasontedor requested a review from s1monw March 9, 2018 22:18
@jasontedor
Copy link
Member Author

jasontedor commented Mar 9, 2018

Note there is an existing test for this in EvilInternalEngineTests, this change does however cover more cases that are not covered by this test. I'll leave it to the reviewer to decide if more coverage is needed here.

Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I left one comment you may change or not change, up to you.

*/
@SuppressWarnings("finally")
private void maybeDie(final String maybeMessage, final Throwable maybeFatal) {
final Optional<Error> maybeError = ExceptionsHelper.maybeError(maybeFatal, logger);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as you're using Optional here, I think this could be

ExceptionsHelper.maybeError(maybeFatal, logger).ifPresent(e -> {
    try {
         logger.error(maybeMessage, e);
    } finally {
        throw e;
    }});

Up to you if you want though

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

/**
* If the specified throwable contains a fatal error in the throwable graph, such a fatal error will be thrown. Callers should ensure
* that there are no catch statements that would catch an error in the stack as the fatal error here should go uncaught and be handled
* by the uncaught exception handler that we install during bootstrap. If the specified throwable dies indeed contain a fatal error, the
Copy link

@richardun richardun Mar 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extremely insignificant, but just for clarity: I think you mean "does" (indeed contain a fatal error) on this line.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @richardun, I appreciate the attention to detail. I will push a correction in the morning.

@jasontedor jasontedor merged commit 4ba80a7 into elastic:master Mar 10, 2018
jasontedor added a commit that referenced this pull request Mar 10, 2018
Today we check for a few cases where we should maybe die before failing
the engine (e.g., when a merge fails). However, there are still other
cases where a fatal error can be hidden from us (for example, a failed
index writer commit). This commit modifies the mechanism for failing the
engine to always check for a fatal error before failing the engine.
jasontedor added a commit that referenced this pull request Mar 10, 2018
Today we check for a few cases where we should maybe die before failing
the engine (e.g., when a merge fails). However, there are still other
cases where a fatal error can be hidden from us (for example, a failed
index writer commit). This commit modifies the mechanism for failing the
engine to always check for a fatal error before failing the engine.
jasontedor added a commit that referenced this pull request Mar 10, 2018
Today we check for a few cases where we should maybe die before failing
the engine (e.g., when a merge fails). However, there are still other
cases where a fatal error can be hidden from us (for example, a failed
index writer commit). This commit modifies the mechanism for failing the
engine to always check for a fatal error before failing the engine.
jasontedor added a commit that referenced this pull request Mar 10, 2018
Today we check for a few cases where we should maybe die before failing
the engine (e.g., when a merge fails). However, there are still other
cases where a fatal error can be hidden from us (for example, a failed
index writer commit). This commit modifies the mechanism for failing the
engine to always check for a fatal error before failing the engine.
@jasontedor jasontedor deleted the maybe-die-when-failing-engine branch March 10, 2018 13:17
sebasjm pushed a commit to sebasjm/elasticsearch that referenced this pull request Mar 10, 2018
Today we check for a few cases where we should maybe die before failing
the engine (e.g., when a merge fails). However, there are still other
cases where a fatal error can be hidden from us (for example, a failed
index writer commit). This commit modifies the mechanism for failing the
engine to always check for a fatal error before failing the engine.
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Mar 12, 2018
* master: (28 commits)
  Maybe die before failing engine (elastic#28973)
  Remove special handling for _all in nodes info
  Remove Booleans use from XContent and ToXContent (elastic#28768)
  Update Gradle Testing Docs (elastic#28970)
  Make primary-replica resync failures less lenient (elastic#28534)
  Remove temporary file 10_basic.yml~
  Use different pipeline id in test. (pipelines do not get removed between tests extending from ESIntegTestCase)
  Use fixture to test the repository-gcs plugin (elastic#28788)
  Use String.join() to describe a list of tasks (elastic#28941)
  Fixed incorrect test try-catch statement
  Plugins: Consolidate plugin and module loading code (elastic#28815)
  percolator: Take `matchAllDocs` and `verified` of the sub result into account when analyzing a function_score query.
  Build: Remove rest tests on archive distribution projects (elastic#28952)
  Remove FastStringReader in favor of vanilla StringReader (elastic#28944)
  Remove FastCharArrayReader and FastCharArrayWriter (elastic#28951)
  Continue registering pipelines after one pipeline parse failure. (elastic#28752)
  Build: Fix ability to ignore when no tests are run (elastic#28930)
  [rest-api-spec] update doc link for /_rank_eval
  Switch XContentBuilder from BytesStreamOutput to ByteArrayOutputStream (elastic#28945)
  Factor UnknownNamedObjectException into its own class (elastic#28931)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. v5.6.9 v6.1.4 v6.2.3 v6.3.0 v7.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OutOfMemoryError not killing the node
5 participants