This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Get feature data #250
Closed
Get feature data #250
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To get feature data, the coordinating node regularly aggregates log entries into multiple entity keys and their corresponding value vectors. An entity key can be an IP address, while the value vector can contain the total bytes sent to the IP address over an interval (say, 1 minute). In this PR, We use terms aggregation (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) to calculate features. There can be a significant number (e.g., millions) of entities in one query. By defaut, we return 1000 entities per query at most. Customers have the option to increase the limit. Testing done: 1. added unit tests. 2. done end-to-end testing
Codecov Report
@@ Coverage Diff @@
## master #250 +/- ##
=========================================
Coverage 73.01% 73.01%
Complexity 1461 1461
=========================================
Files 164 164
Lines 6834 6834
Branches 527 527
=========================================
Hits 4990 4990
Misses 1594 1594
Partials 250 250
Flags with carried forward coverage won't be shown. Click here to find out more.
|
ohltyler
reviewed
Oct 14, 2020
Comment on lines
660
to
663
| logger.debug(() -> "getColdStartSamplesForPeriods: " + request.toString()); | ||
|
|
||
| client.search(request, ActionListener.wrap(response -> { | ||
| logger.debug(() -> "getColdStartSamplesForPeriods: " + response.toString()); |
There was a problem hiding this comment.
small: remove these debugging logs?
ohltyler
approved these changes
Oct 14, 2020
yizheliu-amazon
approved these changes
Oct 16, 2020
kaituo
added a commit
to kaituo/anomaly-detection
that referenced
this pull request
Oct 16, 2020
This PR is a conglomerate of the following PRs. opendistro-for-elasticsearch#247 opendistro-for-elasticsearch#249 opendistro-for-elasticsearch#250 opendistro-for-elasticsearch#252 opendistro-for-elasticsearch#253 opendistro-for-elasticsearch#256 opendistro-for-elasticsearch#257 opendistro-for-elasticsearch#258 opendistro-for-elasticsearch#259 opendistro-for-elasticsearch#260 opendistro-for-elasticsearch#261 opendistro-for-elasticsearch#262 opendistro-for-elasticsearch#263 opendistro-for-elasticsearch#264 opendistro-for-elasticsearch#265 opendistro-for-elasticsearch#266 opendistro-for-elasticsearch#267 opendistro-for-elasticsearch#268 opendistro-for-elasticsearch#269 This spreadsheet contains the mappings from files to PR number: https://quip-amazon.com/DiHkAmz9oSLu/HC-PR Testing done: 1. Add unit tests except four classes (excluded in build.gradle). Will add them in the later PR. 2. Manual testing passes.
kaituo
added a commit
that referenced
this pull request
Oct 16, 2020
* Add support filtering the data by one categorical variable This PR is a conglomerate of the following PRs. #247 #249 #250 #252 #253 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 This spreadsheet contains the mappings from files to PR number: https://quip-amazon.com/DiHkAmz9oSLu/HC-PR Testing done: 1. Add unit tests except four classes (excluded in build.gradle). Will add them in the later PR. 2. Manual testing passes.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: since there are a lot of dependencies, I only list the main class and test code to save reviewers' time. The build will fail due to missing dependencies. I will use that PR just for review. will not merge it. Will have a big one in the end and merge once after all review PRs get approved.
Issue #, if available:
Description of changes:
To get feature data, the coordinating node regularly aggregates log entries into multiple entity keys and their corresponding value vectors. An entity key can be an IP address, while the value vector can contain the total bytes sent to the IP address over an interval (say, 1 minute).
In this PR, We use terms aggregation (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) to calculate features. There can be a significant number (e.g., millions) of entities in one query. By defaut, we return 1000 entities per query at most. Customers have the option to increase the limit.
This PR also adds functions to get cold start samples within a range.
Testing done:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.