This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Add coordinating node logic #267
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds coordinating node logic. To get feature data, the coordinating node aggregates log entries into multiple entity keys and their corresponding value vectors. Then, the coordinating node sends the key-value pair to the model hosting nodes using consistent hashing. Finally, it collects acknowledgements or exceptions and returns them to the job scheduler. Testing done: 1. Will add unit tests. 2. Manual testing passes.
Codecov Report
@@ Coverage Diff @@
## master #267 +/- ##
=========================================
Coverage 73.01% 73.01%
Complexity 1461 1461
=========================================
Files 164 164
Lines 6834 6834
Branches 527 527
=========================================
Hits 4990 4990
Misses 1594 1594
Partials 250 250
Flags with carried forward coverage won't be shown. Click here to find out more.
|
ylwu-amzn
reviewed
Oct 16, 2020
| }); | ||
| } | ||
|
|
||
| listener.onResponse(new AnomalyResultResponse(0, 0, 0, new ArrayList<FeatureData>())); |
There was a problem hiding this comment.
Always return empty anomaly result to job runner to persist ?
There was a problem hiding this comment.
We need to return sth to job runner. Is it necessary to persist it? Currently I do as you told me to have a heartbeat.
There was a problem hiding this comment.
Heartbeat can be persisted in anomaly detection state.
I'm ok with this currently if no impact to frontend.
There was a problem hiding this comment.
I did this for you. If you think it is not necessary, I will remove it.
There was a problem hiding this comment.
Sure, you can remove it.
saratvemulapalli
approved these changes
Oct 16, 2020
ylwu-amzn
approved these changes
Oct 16, 2020
kaituo
added a commit
to kaituo/anomaly-detection
that referenced
this pull request
Oct 16, 2020
This PR is a conglomerate of the following PRs. opendistro-for-elasticsearch#247 opendistro-for-elasticsearch#249 opendistro-for-elasticsearch#250 opendistro-for-elasticsearch#252 opendistro-for-elasticsearch#253 opendistro-for-elasticsearch#256 opendistro-for-elasticsearch#257 opendistro-for-elasticsearch#258 opendistro-for-elasticsearch#259 opendistro-for-elasticsearch#260 opendistro-for-elasticsearch#261 opendistro-for-elasticsearch#262 opendistro-for-elasticsearch#263 opendistro-for-elasticsearch#264 opendistro-for-elasticsearch#265 opendistro-for-elasticsearch#266 opendistro-for-elasticsearch#267 opendistro-for-elasticsearch#268 opendistro-for-elasticsearch#269 This spreadsheet contains the mappings from files to PR number: https://quip-amazon.com/DiHkAmz9oSLu/HC-PR Testing done: 1. Add unit tests except four classes (excluded in build.gradle). Will add them in the later PR. 2. Manual testing passes.
kaituo
added a commit
that referenced
this pull request
Oct 16, 2020
* Add support filtering the data by one categorical variable This PR is a conglomerate of the following PRs. #247 #249 #250 #252 #253 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 This spreadsheet contains the mappings from files to PR number: https://quip-amazon.com/DiHkAmz9oSLu/HC-PR Testing done: 1. Add unit tests except four classes (excluded in build.gradle). Will add them in the later PR. 2. Manual testing passes.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: since there are a lot of dependencies, I only list the main class and test code to save reviewers' time. The build will fail due to missing dependencies. I will use that PR just for review. will not merge it. Will have a big one in the end and merge once after all review PRs get approved.
Issue #, if available:
Description of changes:
This PR adds coordinating node logic. To get feature data, the coordinating node aggregates log entries into multiple entity keys and their corresponding value vectors. Then, the coordinating node sends the key-value pair to the model hosting nodes using consistent hashing. Finally, it collects acknowledgements or exceptions and returns them to the job scheduler.
Testing done:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.