[Bug]: Fix missing local test cluster logs for integration tests

[gaiksaya]

Describe the bug

Currently, the local test cluster only records stdout and stderr for local cluster that is deployed by the test workflow.
Missing logs files:

- gc.log.00
- opensearchcluster1.log
- opensearchcluster1_deprecation.json
- opensearchcluster1_deprecation.log
- opensearchcluster1_index_indexing_slowlog.json
- opensearchcluster1_index_indexing_slowlog.log
- opensearchcluster1_index_search_slowlog.json
- opensearchcluster1_index_search_slowlog.log
- opensearchcluster1_server.json
- opensearchcluster1_task_detailslog.json
- opensearchcluster1_task_detailslog.log

To reproduce

Run below command locally and see that test-results/../local-test-cluster folder only includes stdout and stderr files

./test.sh integ-test manifests/2.7.0/opensearch-2.7.0-test.yml --component geospatial -p opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.7.0/7325/linux/x64/tar --keep

Expected behavior

Should record all log files

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

Estimate: 3 points

Acceptance Criteria

• Local cluster logs should be recorded in test-results folder
• The temp directory (where local opensearch and dashboards is downloaded) should be cleaned up after each test
• Add tests cases verifying the behavior

[zelinh]

This issue is reproduced with command

./test.sh integ-test manifests/2.6.0/opensearch-2.6.0-test.yml --component sql -p opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.6.0/7160/linux/x64/tar

I could only see

[opensearch@bcd027be88ff local-cluster-logs]$ ls
stderr.txt  stdout.txt

and

[opensearch@bcd027be88ff test-results]$ ls
sql.yml  stderr.txt  stdout.txt

[gaiksaya]

That's the missing piece @zelinh. See the tmp directory after using --keep option. The cluster logs are present where opensearch is downloaded. so for 2.7.0 /some/tmp/directory/opensearch-2.7.0/logs/*

[zelinh]

Basically the reason of these missing logs is that the workflow is forced to deleted everything when terminating the local cluster for tarball & zip.
e.g.

opensearch-build/src/test_workflow/integ_test/distribution_tar.py

Lines 41 to 43 in c871a82

	def uninstall(self) -> None:
	logging.info(f"Cleanup {self.work_dir}/* content after the test")
	subprocess.check_call(f"rm -rf {self.work_dir}/*", shell=True)

I would raise a PR to remove this part so the local-test-cluster folder within the TemporaryDirectory would subject to the keep argument.

[rishabh6788]

Do we have any anecdotes from plugin teams requiring any of the above mentioned logs apart from the OS log which is present in stdout file?

[zelinh]

Closing this issue as this has been fixed.