[ML] Close results stream before data frame analytics job stops #67854
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Investigating the failures in elastic#67581 it looked like after restarting the regression job the process was started but no data was loaded. So the process was getting stuck waiting for data. Looking into the code it looks like this can be explained by the fact that AnalyticsResultProcessor counts down its completion latch before it closes the results stream. This means the job may go to stopped state while the out stream is still alive, which on windows results to the directory with the named pipes staying around. Then when the job is started again, which the test does immediately, the old pipes are used and thus the data is not sent to the the new process. This commit fixes this by ensuring the process output stream is consumed and closed when the anylytics process is closed. Fixes elastic#67581
dimitris-athanasiou
added
>test
|
Pinging @elastic/ml-core (:ml) |
dimitris-athanasiou
deleted the
close-results-stream-before-dfa-job-stops
branch
dimitris-athanasiou
added a commit
that referenced
this pull request
Jan 23, 2021
…67854) (#67864) Investigating the failures in #67581 it looked like after restarting the regression job the process was started but no data was loaded. So the process was getting stuck waiting for data. Looking into the code it looks like this can be explained by the fact that AnalyticsResultProcessor counts down its completion latch before it closes the results stream. This means the job may go to stopped state while the out stream is still alive, which on windows results to the directory with the named pipes staying around. Then when the job is started again, which the test does immediately, the old pipes are used and thus the data is not sent to the the new process. This commit fixes this by ensuring the process output stream is consumed and closed when the anylytics process is closed. Fixes #67581 Backport of #67854
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Investigating the failures in #67581 it looked like after restarting
the regression job the process was started but no data was loaded.
So the process was getting stuck waiting for data.
Looking into the code it looks like this can be explained by the
fact that AnalyticsResultProcessor counts down its completion
latch before it closes the results stream. This means the job
may go to stopped state while the out stream is still alive,
which on windows results to the directory with the named pipes
staying around. Then when the job is started again, which the
test does immediately, the old pipes are used and thus the
data is not sent to the the new process.
This commit fixes this by ensuring the process output stream
is consumed and closed when the anylytics process is closed.
Fixes #67581