Run packaging tests on CI infrastructure

[reta]

Is your feature request related to a problem? Please describe

It seems like the CI does not run any packaging tests, /gradlew packagingTest or ./gradlew distroTest, right now (see please [1]). As of today, this is the state of packaging tests:

1. Running them manually on main branch ends up with tons of failing tasks
2. What is more concerning, trying to install produced RPM or DEB distributions manually on supporting platform is not successful (the install scripts are failing)

@peterzhuamazon @dblock it seems like it would help with #27, at least for Opensearch part

[1] https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-packaging

Describe the solution you'd like

Running packaging tests on CI infrastructure regularly (not sure about the need to run on each pull request or merge) to keep the supported distributions verifyable and installable.

Describe alternatives you've considered

Support only archives

Additional context

N/A

[gaiksaya]

@bbarani Can you please add comments here?
Thanks!

[bbarani]

@reta can you please help us understand the scope of packaging / distro test? We are planning to create automated validation pipelines for RPM builds as part of #1244 but want to understand the current functionality of packing test.

[reta]

@bbarani absolutely, so right now there are 4 types of packaging (at least) produced by build scripts:

• Docker
• deb
• rpm
• archive (*.tar.gz)

The distribution tests [1] use Vagrant tol spin off a large set of *nx / Windows distributions, deploy all these package types on each of those (one by one), and verify that:
a) they are deployed properly
b) the Opensearch server starts successfully

Is it helpful? Or you have been asking for something else?
Thank you.

[1] https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-packaging

[reta]

@jcgraybill fyi

[gaiksaya]

[Triage] Hi @owaiskazi19, AFAIK you were looking into running gradle check via different means, does this overlaps with your efforts?

We can add the testing for distribution to our backlog. For OpenSearch specific tests, that should be run in OpenSearch repository.

[bbarani]

As mentioned by @gaiksaya, we would focus more on executing tests at distribution level and not at repo level. Having said that, we dont have plans to add additional distribution types to -min artifact but it would be ideal to run these tests as part of your gradle check to surface any gaps that we might encounter when we create the OpenSearch distributions.

[bbarani]

@reta @owaiskazi19 Can you track this issue as part of your gradle check improvements plan?

[bbarani]

@peterzhuamazon and @rishabh6788 are working on moving the gradle check to public Jenkins.
@reta @owaiskazi19 Please make the required changes to support your requirements once the migration is complete

[bbarani]

@reta @saratvemulapalli Before we include this as part of Gradle check, we need to fix the existing flaky tests. Do you have any plans to improve the current gradle check process? We might need to modularize and start splitting the different sections of gradle checks to make it run at granular level based on the need. The current gradle check are not scalable and its going to become a bottleneck very soon.

CC: @CEHENKLE @dblock @peterzhuamazon

[reta]

@bbarani I think we should not be blocked here by flakyness of existing tests, the packaging tests are well isolated and could be run as separate step. But flaky tests and monolithic checks are serious issues, we [1], [2], [3] to tackle that.

[1] opensearch-project/OpenSearch#2496
[2] opensearch-project/OpenSearch#1715
[3] opensearch-project/OpenSearch#1975

[saratvemulapalli]

@bbarani, +1 to @reta flaky tests should not really prevent us from adding packaging tests.
But a single check for all tests is a bad recipe.

[prudhvigodithi]

Hey @reta is this issue still required? today we run . /gradlew check in jenkins, but when I listed the tasks for check I dont see packagingTest or distroTest, however since this RPM or DEB is handled from the build scripts, do we still need this issue?
@bbarani @peterzhuamazon @gaiksaya @rishabh6788

[reta]

Hey @reta is this issue still required?

Hey @prudhvigodithi , this issue still needs resolution - we do have large chunk of scaffolding to run these distribution tests, if build scripts take care of that, we should remove this dead code from OpenSearch. However, if there are gaps, we could brings them to build scripts .

[prudhvigodithi]

Hey @prudhvigodithi , this issue still needs resolution - we do have large chunk of scaffolding to run these distribution tests, if build scripts take care of that, we should remove this dead code from OpenSearch. However, if there are gaps, we could brings them to build scripts .

Got it, AFAIK the build code after build, the assemble logic will take of bundling into RPM and DEB using rpmbuild and debmake. If this is the case then ya we should remove the dead code from OpenSearch.