Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi-category support, rate limiting, and pagination #121

Merged
merged 13 commits into from
Jul 12, 2021
+10,651 −3,688
Merged

multi-category support, rate limiting, and pagination #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
4a8a882
Multi-category field
kaituo May 28, 2021
f5de35c
Bug fix in EntityColdStartWorker
kaituo Jun 24, 2021
112da7f
Bug fix in ColdEntityWorker
kaituo Jun 26, 2021
a208014
Stats API bug fix
kaituo Jun 28, 2021
6393e71
Remove unnecessary license and add comments
kaituo Jun 29, 2021
897a8b1
Calculate requests to prune in RateLimitedQueue.prune and do the prun…
kaituo Jun 29, 2021
c13314e
Add tests for QueuedRequest
kaituo Jun 30, 2021
3ed02af
cleaning expired requests before calling the drain method
kaituo Jul 1, 2021
c02cca1
Add unit tests for CheckpointDao
kaituo Jul 1, 2021
43d46dc
Fix bug in PriorityTracker
kaituo Jul 6, 2021
5823dec
Fix bug on EndRunException
kaituo Jul 6, 2021
0dee272
ignore jacoco coverage for a couple of classes for now
kaituo Jul 6, 2021
69490ff
address Tyler's comments; add more tests
kaituo Jul 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ evaluationDependsOnChildren()
task release(type: Copy, group: 'build') {
dependsOn allprojects*.tasks.build
from(zipTree(project.tasks.bundlePlugin.outputs.files.getSingleFile()))
into "build/plugins/opendistro-anomaly-detection"
into "build/plugins/opensearch-anomaly-detection"
includeEmptyDirs = false
// ES versions < 6.3 have a top-level opensearch directory inside the plugin zip which we need to remove
eachFile { it.path = it.path - "opensearch/" }
Expand Down Expand Up @@ -327,6 +327,20 @@ List<String> jacocoExclusions = [
'org.opensearch.ad.indices.AnomalyDetectionIndices',
'org.opensearch.ad.transport.handler.MultiEntityResultHandler',
'org.opensearch.ad.util.ThrowingSupplierWrapper',
'org.opensearch.ad.transport.EntityResultTransportAction',
'org.opensearch.ad.transport.EntityResultTransportAction.*',
'org.opensearch.ad.transport.AnomalyResultTransportAction.*',
'org.opensearch.ad.transport.ProfileNodeResponse',
'org.opensearch.ad.transport.ADResultBulkResponse',
'org.opensearch.ad.transport.AggregationType',
'org.opensearch.ad.EntityProfileRunner',
'org.opensearch.ad.NodeStateManager',
'org.opensearch.ad.util.BulkUtil',
'org.opensearch.ad.util.ExceptionUtil',
'org.opensearch.ad.feature.SearchFeatureDao',
'org.opensearch.ad.feature.CompositeRetriever.*',
'org.opensearch.ad.feature.ScriptMaker',
'org.opensearch.ad.ml.EntityModel',
]

jacocoTestCoverageVerification {
Expand Down Expand Up @@ -431,11 +445,11 @@ afterEvaluate {
prefix '/usr'

license 'ASL-2.0'
maintainer 'OpenDistro for Elasticsearch Team <opendistro@amazon.com>'
url 'https://opendistro.github.io/for-elasticsearch/downloads.html'
maintainer 'OpenSearch <opensearch@amazon.com>'
url 'https://opensearch.org/downloads.html'
summary '''
Anomaly Detection plugin for OpenDistro for Elasticsearch.
Reference documentation can be found at https://opendistro.github.io/for-elasticsearch-docs/.
Anomaly Detection plugin for OpenSearch.
Reference documentation can be found at https://docs-beta.opensearch.org/docs/ad/.
'''.stripIndent().replace('\n', ' ').trim()
}

Expand Down
9 changes: 5 additions & 4 deletions src/main/java/org/opensearch/ad/AnomalyDetectorJobRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,12 +468,14 @@ private void indexAnomalyResult(
String detectorId = jobParameter.getName();
detectorEndRunExceptionCount.remove(detectorId);
try {
// reset error if different from previously recorded one
detectionStateHandler.saveError(response.getError(), detectorId);
// skipping writing to the result index if not necessary
// For a single-entity detector, the result is not useful if error is null
// and rcf score (thus anomaly grade/confidence) is null.
// For a multi-entity detector, we don't need to save on the detector level.
// We always return 0 rcf score if there is no error.
if (response.getAnomalyScore() <= 0 && response.getError() == null) {
// For a HCAD detector, we don't need to save on the detector level.
// We return 0 or Double.NaN rcf score if there is no error.
if ((response.getAnomalyScore() <= 0 || Double.isNaN(response.getAnomalyScore())) && response.getError() == null) {
return;
}
IntervalTimeConfiguration windowDelay = (IntervalTimeConfiguration) ((AnomalyDetectorJob) jobParameter).getWindowDelay();
Expand All @@ -499,7 +501,6 @@ private void indexAnomalyResult(
indexUtil.getSchemaVersion(ADIndex.RESULT)
);
anomalyResultHandler.index(anomalyResult, detectorId);
detectionStateHandler.saveError(response.getError(), detectorId);
} catch (Exception e) {
log.error("Failed to index anomaly result for " + detectorId, e);
} finally {
Expand Down
Loading