Report test coverage

[saratvemulapalli]

Report test coverage for all the tests done in OpenSearch.

One of the options include using https://github.com/codecov

[tlfeng]

Currently code coverage for specific tests can be obtained through Intellij IDEA when running the test manually.
As far as I know, the biggest obstacle to get code coverage report of Elasticsearch using JaCoCo Gradle plugin that mentioned in [ https://github.com/elastic/elasticsearch/issues/ 28867#issuecomment-372714244 ]
has been resolved in elastic/elasticsearch@2b2a3f5.
So using JaCoCo Gradle plugin to collect code coverage for unit tests of OpenSearch is possible, however there are still some known issues in using JaCoCo for integration tests.

JaCoCo Gradle plugin will make all Gradle tasks of type "org.gradle.api.tasks.testing.Test" to provide coverage information.

In OpenSearch, there are several Gradle tasks to run tests:
test, integTest, javaRestTest, yamlRestTest, internalClusterTest, forbiddenApisTest, largeBlobYamlRestTest, mixedClusterTest and singleNodeIntegTest

All the above tasks, except "forbiddenApisTest", are extended by "org.gradle.api.tasks.testing.Test" class, so they can be enbanced by JaCoCo plugin to provide coverage information.

Note:

1. Gradle "test" task in OpenSearch will execute unit tests (the class filename ends with "Tests"), which is defined by OpenSearchTestBasePlugin.
2. Gradle "integTest" task is created by OpenSearch in RestTestPlugin, and it includes all class files ends with "IT".

[tlfeng]

There are several sub-projects in the whole repository that don't have tests, which can be found by searching for test.enabled = false in the code base.

[saratvemulapalli]

This is nice. Couple of questions:

1. How many tests do we have for forbiddenApisTest
2. How many and which subprojects do not have test.enabled = false

[tlfeng]

How many tests do we have for forbiddenApisTest

I realized that forbiddenApisTest is not a Gradle task defined by OpenSearch, and not used for running test either. So we can ignore it from the test code coverage.

forbiddenApisTest "Runs forbiddenApis checks on 'test' classes.", which is written in the comment in this link ( https://github.com/policeman-tools/forbidden-apis/issues/ 68#issuecomment-138122778 ),

It's defined by https://github.com/policeman-tools/forbidden-apis, for its Gradle plugin, it will register a separate task for each defined sourceSet. "For default Java projects, two tasks are created: forbiddenApisMain and forbiddenApisTest."
(For detail: https://github.com/policeman-tools/forbidden-apis/wiki/GradleUsage
https://github.com/policeman-tools/forbidden-apis/blob/3.1/src/main/resources/de/thetaphi/forbiddenapis/gradle/plugin-init.groovy#L36)

How many and which subprojects do not have test.enabled = false

Sorry my above statement was not correct, the sub-projects that disabled test task (test.enabled = false ) means they don't have unit test, but they may have integration tests.
The sub-projects that don't have unit tests are:

:benchmarks
:rest-api-spec (but has yamlRestTest task)
:client:benchmark
:client:client-benchmark-noop-api-plugin
:client:test
:example-plugins:custom-suggester (but has yamlRestTest task)
:example-plugins:painless-whitelist (but has yamlRestTest task)
:example-plugins:rest-handler (but has javaRestTest and yamlRestTest task)
:example-plugins:script-expert-scoring (but has yamlRestTest task)
:libs:opensearch-cli
:libs:opensearch-plugin-classloader
:modules:geo (but has yamlRestTest task)
:plugins:discovery-azure-classic (but has internalClusterTest and yamlRestTest task)
:plugins:mapper-size (but has internalClusterTest and yamlRestTest task)
:qa:die-with-dignity (but has javaRestTest task)
:qa:os
:qa:wildfly (but has integTest task)
:distribution:tools:java-version-checker
:modules:lang-painless:spi
:test:fixtures:azure-fixture
:test:fixtures:gcs-fixture
:test:fixtures:hdfs-fixture
:test:fixtures:old-elasticsearch
:test:fixtures:s3-fixture

[tlfeng]

Created 3 issues to complete the code coverage reporting:

• Upload code coverage report through CI #1320
• Add status badge for showing code coverage in README #1323
• Add status check of code coverage for pull requests #1324

[dreamer-89]

@tlfeng : Thanks for the deep dive and summary of actions to be done. With jenkins still choice of infra; I think we are unblocked on #1320

[dreamer-89]

Generated codecov report on local machine and uploaded on server against forked repository. The results shows ~70% code coverage. Waiting on infra side changes for enabling code cov on OpenSearch repo ETA June 24.

https://app.codecov.io/gh/dreamer-89/OpenSearch/

[peterzhuamazon]

Hi @dreamer-89 this is complete for now:

• Setup code coverage for OpenSearch repo opensearch-build#2193