Merge from master to 1.10 release #181

sruti1312 · 2020-08-24T02:48:58Z

Fixes #, if available:
Merge from master to 1.10 release

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Merging OpenDistro-1.2 into master

…ter node is fully up

…ction.

Merge fixes from OD-1.1 branch

Add RFC for RCA

This commit has 3 major changes - 1) Performance Analyzer code that is Elasticsearch version independent. 3) Performance optimization in the Elasticsearch plugin to emit events to a single event log file. This brings down CPU utilization by an order of magnitude on large clusters.

Update Performance Analyzer to support ElasticSearch version 7.3.2

…en in the split version of PA. Features and fixes introduced in this PR: Allow performance analyzer to be en(dis)abled through a cluster setting across the cluster. Allow logging to be controllable through the cluster setting Capture a node's role along with the host address and the node name Checkstyle compliance Some issues that are still not addressed in this PR: Update build scripts to start the agent from the reader location instead of the plugin location Remove pa_config, pa_bin and other folders that are already present in the reader

… performanceanalyzer

Remove NewFormatProcessorTest class which is not used.

…-unused-test Delete Unused test class

…on in node stats collector (#83)

…etailsStatus

Make the unit test backward compatible with the _cat/master fix

* Create gradle.yml * Update gradle.yml * Update gradle.yml

* add cd.yml * upgrade ospackage version to 8.2.0 * change s3 * update cd.yml

* Making sure that we don't throw exceptions while intercepting ES requests PerformanceAnalyzer intercepts various ES request paths toget detailed metrics. But today if we throw an exception, then it will bubble all the way upto ES and fail the request. * Addressing the PR comments * Updating the .gitignore * style changes

* Adding Shard Size Metric as a part of Node Stats * removing the -i flag * fix a bug in StatsTests.java (#97) * Update CONTRIBUTORS.md * Update CONTRIBUTORS.md * Adressing Typos Co-authored-by: Aditya Jindal <[email protected]> Co-authored-by: Joydeep Sinha <[email protected]> Co-authored-by: Ruizhen Guo <[email protected]> Co-authored-by: Balaji <[email protected]>

Node Collector split is created based on the metrics which are required for all the shards on the node and other which can be collected on a few number of shards per iteration. Built the Jar from this Patch and applied on the AES cluster. The Cache related metrics which should be collected for all the shards irrespective of the value of shardsPerCollection value are getting collected. Tested with a zero value of this parameter (shardsPerCollection).

* Use the correct ctor for NodeDetailsCollector * Check for null ConfigOverrides wrapper while appending timestamps

* Use the correct ctor for NodeDetailsCollector * Check for null ConfigOverrides wrapper while appending timestamps * Add unit test for null cluster setting for config overrides

…#168)

* Fix invalid cluster state * Address PR comments

# Conflicts: # .github/workflows/cd.yml # .github/workflows/gradle.yml # CONTRIBUTORS.md # build.gradle # gradle.properties # gradle/wrapper/gradle-wrapper.properties # licenses/commons-lang3-LICENSE.txt # licenses/commons-lang3-NOTICE.txt # licenses/sqlite-jdbc-LICENSE.txt # release-notes # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/PerformanceAnalyzerPlugin.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/collectors/NodeDetailsCollector.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/collectors/NodeStatsMetricsCollector.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/collectors/ThreadPoolMetricsCollector.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/config/setting/ClusterSettingsManager.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/config/setting/PerformanceAnalyzerClusterSettings.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/config/setting/handler/PerformanceAnalyzerClusterSettingHandler.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/http_action/config/PerformanceAnalyzerClusterConfigAction.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/http_action/config/PerformanceAnalyzerConfigAction.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/http_action/config/PerformanceAnalyzerResourceProvider.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/http_action/whoami/WhoAmIAction.java # src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/util/Utils.java # src/test/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/PerformanceAnalyzerIT.java # src/test/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/collectors/JsonKeyTests.java # src/test/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/collectors/NodeStatsMetricsCollectorTests.java # src/test/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/config/PerformanceAnalyzerClusterSettingHandlerTest.java # src/test/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/hwnet/CollectMetricsTests.java # src/test/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/reader/AbstractReaderTests.java

khushbr

LGTM! Thank you for making these changes.

yojs

Is it possible to have a test, that checks for the lib dependencies that we share with ES, we are using the same version ? Just to make sure, we don't upgrade libs and run into this in future ?

sruti1312 · 2020-08-24T21:30:17Z

Is it possible to have a test, that checks for the lib dependencies that we share with ES, we are using the same version ? Just to make sure, we don't upgrade libs and run into this in future ?

I ran onto this issue when testing with docker. Definitely adding a test around this is going to be helpful in the future. I will add them on subsequent PR.

yojs

We added some new files. Can we add a javadoc comment as why they were required ?

ktkrg and others added 30 commits July 24, 2019 16:44

Merge pull request #65 from ktkrg/opendistro-1.2

f161118

Merging OpenDistro-1.2 into master

Fix NullPointerException when PA starts collecting metrics before mas…

92e53b8

…ter node is fully up

Fix unit tests on Mac. Fix NPE during MasterServiceEventMetrics colle…

f29d127

…ction.

Reorder imports, refactor unit tests

19a4d2b

Merge pull request #67 from ktkrg/master_merge

f68487b

Merge fixes from OD-1.1 branch

add RFC for RCA

392e8a0

Merge pull request #72 from opendistro-for-elasticsearch/rcarfc

12232b0

Add RFC for RCA

Create README.md

7b1c023

Update Performance Analyzer to support ElasticSearch version 7.3.2

76cce41

Merge pull request #74 from dvjyothsna/master

275abd1

Update Performance Analyzer to support ElasticSearch version 7.3.2

Adding the dependency on the renamed jar performanceanalyzer-rca from…

9570ab0

… performanceanalyzer

Delete Unused test class

9411a6e

Remove NewFormatProcessorTest class which is not used.

Merge pull request #82 from opendistro-for-elasticsearch/ktkrg-delete…

e89b287

…-unused-test Delete Unused test class

Adding shardsPerCollection REST API to update the shards Per Collecti…

64698d6

…on in node stats collector (#83)

Update gradle wrapper

5287ec6

Add isMasterNode to NodeDetailsStatus (#84)

cc7ce7a

make the unit test backward compatible with the isMasterNode in NodeD…

9152076

…etailsStatus

Merge pull request #85 from opendistro-for-elasticsearch/rguo-merge

9e2cc2b

Make the unit test backward compatible with the _cat/master fix

Create gradle.yml (#87)

bb337b8

* Create gradle.yml * Update gradle.yml * Update gradle.yml

Added the bouncy castle jars

703a9e4

Added the licenses file

e6a5106

Add cd.yml and enable CD pipeline to upload artifact to S3 (#90)

506b0a3

* add cd.yml * upgrade ospackage version to 8.2.0 * change s3 * update cd.yml

removing the -i flag

07250e8

fix a bug in StatsTests.java (#97)

2d00113

Update CONTRIBUTORS.md

c3aa8d5

Update CONTRIBUTORS.md

5db7fc8

adityaj1107 and others added 16 commits August 3, 2020 16:10

Use the correct ctor for NodeDetailsCollector (#166)

5810f42

* Use the correct ctor for NodeDetailsCollector * Check for null ConfigOverrides wrapper while appending timestamps

Add unit test for null cluster setting (#167)

7bc70ff

* Use the correct ctor for NodeDetailsCollector * Check for null ConfigOverrides wrapper while appending timestamps * Add unit test for null cluster setting for config overrides

Split capacity/latency collecting logic into separate try/catch block (…

6830838

…#168)

Update PULL_REQUEST_TEMPLATE.md

7dc225a

Fix invalid cluster state (#172)

bcf65ef

* Fix invalid cluster state * Address PR comments

Skip RCA tests when building PA in Github workflows (#177)

8568b8e

Build against elasticsearch 7.9 and resolve dependency conflicts

3a38289

Add licenses for dependencies

fbc54fb

Add licenses for dependencies

3361a3c

Change minor version

9324d62

Add release notes and contributors

1de8a4a

Changed licenses

ded6d00

Modify github workflows

5d8abf1

Fix merge conflicts

ee82a71

sruti1312 requested review from ktkrg and sidheart August 24, 2020 02:48

Base automatically changed from partsrut-opendistro-1.10 to opendistro-1.10 August 24, 2020 18:07

sruti1312 and others added 2 commits August 24, 2020 11:21

Merge branch 'opendistro-1.10' into partsrut-1.10-merge

72d466a

Fix jarHell around log4j

115f1d7

sruti1312 requested review from khushbr and yojs August 24, 2020 18:40

khushbr approved these changes Aug 24, 2020

View reviewed changes

yojs reviewed Aug 24, 2020

View reviewed changes

sidheart approved these changes Aug 24, 2020

View reviewed changes

sruti1312 merged commit cb5ed15 into opendistro-1.10 Aug 24, 2020

sruti1312 deleted the partsrut-1.10-merge branch August 24, 2020 23:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge from master to 1.10 release #181

Merge from master to 1.10 release #181

sruti1312 commented Aug 24, 2020

khushbr left a comment

yojs left a comment

sruti1312 commented Aug 24, 2020

yojs left a comment

Merge from master to 1.10 release #181

Merge from master to 1.10 release #181

Conversation

sruti1312 commented Aug 24, 2020

khushbr left a comment

Choose a reason for hiding this comment

yojs left a comment

Choose a reason for hiding this comment

sruti1312 commented Aug 24, 2020

yojs left a comment

Choose a reason for hiding this comment