[META] After tests are complete there should be a test-run manifest that allows for finding all related test artifacts

[peternied]

Problem

After integration tests are run there is alot of information on the standard out and a final summary message that looks like:

2021-11-22 22:39:29 INFO     | alerting             | with-security        | PASS  |    0 |
2021-11-22 22:39:29 INFO     | alerting             | without-security     | PASS  |    0 |
2021-11-22 22:39:29 INFO     | anomaly-detection    | with-security        | PASS  |    0 |
2021-11-22 22:39:29 INFO     | anomaly-detection    | without-security     | PASS  |    0 |
2021-11-22 22:39:29 INFO     | asynchronous-search  | with-security        | PASS  |    0 |
2021-11-22 22:39:29 INFO     | asynchronous-search  | without-security     | PASS  |    0 |
2021-11-22 22:39:29 INFO     | dashboards-reports   | without-security     | PASS  |    0 |
2021-11-22 22:39:29 INFO     | index-management     | with-security        | PASS  |    0 |
2021-11-22 22:39:29 INFO     | index-management     | without-security     | PASS  |    0 |
2021-11-22 22:39:29 INFO     | k-NN                 | with-security        | PASS  |    0 |
2021-11-22 22:39:29 INFO     | k-NN                 | without-security     | PASS  |    0 |
2021-11-22 22:39:29 INFO     | sql                  | with-security        | PASS  |    0 |
2021-11-22 22:39:29 INFO     | sql                  | without-security     | PASS  |    0 |

This is useful for humans, but harder for our automation systems to integrate with. If we are going to automatically file bugs or analyze test results there is critical information missing such as how to look up service logs, or test result logs.

Proposal 1: Create a test-run manifest / update component manifests

Rather than modify on this table based approach which is very useful for the happy path, I suggest we create a new test-run yml manifest. The existing component manifest files would be augmented to include the hierarchy of all files under them for deep diving.

test-run manifest contents:

• Exact command used to execute, for reproducabillity
• Extracted details about the test circumstances
• List of components (Same list as from test-manifest)
  • Name of the component
  • List configurations ran against that component
  • Name of the configuration
  • Arguments to the configuration
  • Test component manifest with additional and all associated artifacts
  • (Optional) the central report for investigation easy for a human to parse through

component manifest content Existing implementation

• Existing contents: Name, exist code, test run, etc..
• List of all files under the directory so they can be looked up.

test-run manifest located at https://{BASE_URL}/test-results/test-run-b30b804a1a3e494089b720544627dd2e.yml

schema-version: '1.0'
test-run:
  Command: ./test.sh integ-test manifests/1.3.0/opensearch-1.3.0-test.yml https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1032/linux/x64
  TestType: integ-test
  TestManifest: manifests/1.3.0/opensearch-1.3.0-test.yml
  DistributionManifest: https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1032/linux/x64/dist/manifest.yml
  Product: OpenSearch
  TestId: b30b804a1a3e494089b720544627dd2e
components:
- name: asynchronous-search
  configs:
  - name: with-security
    pass: true
    report: https://{BASE_URL}/b30b804a1a3e494089b720544627dd2e/integ-test/asynchronous-search/with-security/test-results/opensearch-integ-test/index.html
    manifest: https://{BASE_URL}/b30b804a1a3e494089b720544627dd2e/integ-test/asynchronous-search/with-security/test-results/asynchronous-search.yml
  - name: without-security
    pass: true
    report: https://{BASE_URL}/b30b804a1a3e494089b720544627dd2e/integ-test/asynchronous-search/without-security/test-results/opensearch-integ-test/index.html
    manifest: https://{BASE_URL}/b30b804a1a3e494089b720544627dd2e/integ-test/asynchronous-search/without-security/test-results/asynchronous-search.yml
- name: alerting
  configs:
  - name: with-security
    pass: true
    manifest: https://{BASE_URL}/b30b804a1a3e494089b720544627dd2e/integ-test/alerting/with-security/test-results/alerting.yml
  - name: without-security
    pass: true
    manifest: https://{BASE_URL}/b30b804a1a3e494089b720544627dd2e/integ-test/alerting/with-security/test-results/alerting.yml
  ...

component outcome manifest located at https://{BASE_URL}/test-results/b30b804a1a3e494089b720544627dd2e/integ-test/alerting/with-security/alerting.yml

component_name: alerting
exit_code: 0
test_config: with-security
test_run_id: b30b804a1a3e494089b720544627dd2e
test_type: integ-test
files:
- https://{BASE_URL}/{COMPONT_PATH}/stderr.txt
- https://{BASE_URL}/{COMPONT_PATH}/stdout.txt
- https://{BASE_URL}/{COMPONT_PATH}/local-cluster-logs/stderr.txt
- https://{BASE_URL}/{COMPONT_PATH}/local-cluster-logs/stdout.txt
- https://{BASE_URL}/{COMPONT_PATH}/local-cluster-logs/opensearch-service-logs/opensearch.log
...

Additional Considerations

Re-path test results

It would be useful to reorganize the existing output to be slightly more collapsed as well, removing the test-results folder from the component/config-name/test-results path and placing the results within the component/config-name folder path. We can look for other ways to make sure the data is very clear. There might be other opportunities to improve the 'browsing experience'

Add correlating timestamps in stderr/stdout results

Not sure if this needs to be added as a wrapper of the console capturing code or if it would be better to change test.sh output from gradle to include these timestamps. This is critical for aligning service failures against test execution when there isn't a clear output from a test failure.

Action Items

• [Bug]: Fix missing local test cluster logs for integration tests #3382
• Update component.yml to get required information #3381
• Adopt base-path feature to jenkins workflow to generate enhanced component yml from CI #3499

[dblock]

I like this. I would only call out that most of the runs should be against artifacts on ci.opensearch.org and thus any test results should contain complete HTTPs links that work.

[peternied]

@ohltyler After the automation is running we will need something like this to make test results usable by everyone

[zelinh]

We would follow this approach for this issue.
Create a test-run manifest / update component manifests on Python workflow base

The idea of this approach is to augment our current python test workflow, so the manifest generated could be also retrieved if we are running the code base locally.
We want to generate a new manifest temporarily called test-run.yml generated after the integration tests run which would contains below contents:

• High level overview of the tests run and extracted details about the test circumstances.
  • Test Type (integ-test)
  • Test manifest used (manifests/2.6.0/opensearch-2.6.0-test.yml)
  • Type of the product tests run on
  • Version of the distribution.
  • General command to run the tests
  • Distribution manifest we run tests on. (URL or local URL path)
• List of components (Same list of components as from test-manifest)
  • Name of the component
  • Name of the configuration (with/without-security)
  • URL or local path to the test component manifest with additional information
  • Exact command used to execute, for reproducibility
  • List of all files under the directory so they can be looked up. e.g. standard output, standard error logs...

A sample test-run.yml could look like this

---
schema-version: '1.0'
name: OpenSearch
test-run:
  Command: ./test.sh integ-test manifests/2.6.0/opensearch-2.6.0-test.yml --paths
    opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.6.0/7083/linux/x64/tar
  TestType: integ-test
  TestManifest: manifests/2.6.0/opensearch-2.6.0-test.yml
  DistributionManifest: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.6.0/7083/linux/x64/tar/dist/opensearch/manifest.yml
  TestID: '2345'
components:
  - name: sql
    command: ./test.sh integ-test manifests/2.6.0/opensearch-2.6.0-test.yml --component
      sql --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.6.0/7083/linux/x64/tar
    configs:
      - name: with-security
        status: pass
        logs:
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/with-security/test-results/stderr.txt
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/with-security/test-results/stdout.txt
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/with-security/local-cluster-logs/stderr.txt
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/with-security/local-cluster-logs/stdout.txt
      - name: without-security
        status: fail
        logs:
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/without-security/test-results/stderr.txt
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/without-security/test-results/stdout.txt
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/without-security/local-cluster-logs/stderr.txt
          - https://ci.opensearch.org/ci/dbc/integ-test/2.6.0/7083/linux/x64/tar/test-results/2345/integ-test/sql/without-security/local-cluster-logs/stdout.txt

[zelinh]

Currently I have the groovy library merged and the changes in our integ tests workflow PR pending. Once the PR is merged, we would be able to have the one-in-all manifest including all details about each test-workflow. We could also update the auto cut issue to refer to the new test-report manifest.

[zelinh]

Found a issue and I'm working on the fix.#3925

[zelinh]

The test-report manifest is now onboarded and generated for each test run. The link to access this test-report manifest is attached in the autocut GitHub test issue.

I will also add an instruction for how to use the test-report manifest to our build repo so everyone can be self-served on this.

[zelinh]

Closing this issue as this enhancement request is fulfilled.