Skip to content

Commit

Permalink
[ML] Fix hang if renormalization is retrying writes during feature re…
Browse files Browse the repository at this point in the history
…set (elastic#92524)

This is basically a test issue, as feature reset isn't usable in production
due to reseting security.

Fixes elastic#92521
  • Loading branch information
droberts195 authored Dec 22, 2022
1 parent 8515535 commit 08515ea
Showing 1 changed file with 9 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.elasticsearch.common.util.CancellableThreads;
import org.elasticsearch.xpack.core.ml.job.process.autodetect.state.Quantiles;

import java.util.Date;
Expand Down Expand Up @@ -94,8 +95,15 @@ public void waitUntilIdle() throws InterruptedException {
try {
taskToWaitFor.get();
} catch (ExecutionException e) {
// This shouldn't happen, because we catch normalization errors inside the normalization loop
if (e.getCause() instanceof CancellableThreads.ExecutionCancelledException) {
// This happens if reset mode is enabled while result writes are being retried.
// Reset mode means the job whose results were being normalized will shortly
// cease to exist, so it's fine to consider the wait for renormalization complete.
break;
}
logger.error("[" + jobId + "] Error propagated from normalization", e);
// Don't loop again for the same task that caused the error
taskToWaitFor = null;
} catch (CancellationException e) {
// Convert cancellations to interruptions to simplify the interface
throw new InterruptedException("Normalization cancelled");
Expand Down

0 comments on commit 08515ea

Please sign in to comment.