[Feature] Publish ARCHIVE test distribution to Maven Local

[navneet1v]

Describe the bug
mavenLocalPublish Zip of OpenSearch Core repo doesn't contain the core plugins(geo) present in the repo. This is leading to the failure of integration test on all the platforms(macOs, AmazonLinux2 etc).

To Reproduce
Steps to reproduce the behavior:

1. Checkout the OpenSearch repo.
2. Run ./gradlew publishToMavenLocal -Dbuild.version_qualifier=beta1 -Dbuild.snapshot=false
3. Validate a zipfile is present in the path: ~/.m2/repository/org/opensearch/distribution/integ-test-zip/opensearch/3.0.0-beta1/
4. Checkout the out the geo-spatial repo from here: https://github.com/navneet1v/geospatial/tree/mavenLocalTesting
5. Then run ./gradlew integTest -PcustomDistributionUrl="" example: ./gradlew integTest -PcustomDistributionUrl="/Users/navneev/.m2/repository/org/opensearch/distribution/integ-test-zip/opensearch/3.0.0-beta1/opensearch-3.0.0-beta1.zip"

Expected behavior
The expectation is the integ Test should succeed.

Plugins
None

Host/Environment (please complete the following information):

• OS: macOs, AmazonLinux2
• Version 11.6.5,

Impact
This is impacting the ability to run the Integration test with the custom version of OpenSearch when as a developer you need to make changes in both the repos at same time.

[saratvemulapalli]

@navneet1v INTEG_TEST distribution does not have modules installed.
Use ARCHIVE distributions if the plugin depends on a module.

[navneet1v]

Hi @saratvemulapalli
Please let me know then how I can do this:

1. How to publish the ARCHIVE distribution in the maven Local?

[saratvemulapalli]

@navneet1v are you specifically looking to export this to Maven local ?
Why aren't you relying on DistributionDownloadPlugin which is part of gradle plugin opensearchplugin ?

[navneet1v]

@saratvemulapalli
The reason I cannot use it because I have changes in main OpenSearch Repo and Geo-Spatial Plugin which I need to test together. For testing, as of now I am publishing the OpenSearch code to Maven Local and then picking it up in the Integration tests via PcustomDistributionUrl flag.
The DistributionDownloadPlugin will not provide me the build against which I can test as per my understanding.

Is there a way I can publish my build to some remote artifact and pick it up from there by using DistributionDownloadPlugin. If yes can you please provide the steps. If no, please suggest the alternative.

[saratvemulapalli]

@navneet1v Thanks a lot for that information.
That makes more sense. I think what you are looking for is a publish ARCHIVE distribution to maven which includes all modules.

Couple of ideas:

• Can you consume opensearch tarball, setup the cluster manually (i.e install your plugin) and run remote IntegTests overriding the regular URLs[1].
• Longer term, I see value in publishing ARCHIVE distribution to Maven. We can use this issue to take it as a feature.

What do you think?

[1] opensearch-project/anomaly-detection#298

[navneet1v]

Hi @saratvemulapalli
No, that will not work. Because there are changes in the OpenSearch code also and those changes will not be in the Tarbell. Hence the option 1 is not an option.

[saratvemulapalli]

@navneet1v you should be able to build your own tarball using ./gradlew assemble.
Take a look at: https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#build

[navneet1v]

After discussing with @saratvemulapalli I was able to get an alternative way in which I can create the local TAR for the opensearch code. The idea was to use the commands provided here: https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#build to generate the tar for the specific platform. Once this is generated It can be easily accessed on this path:
<OpenSearch-Repo-Path>/distribution/archives/darwin-tar/build/distributions/
and then use this path with the flag which I was already using -PcustomDistributionUrl with my integration tests. Volla, it worked.

Thanks @saratvemulapalli for providing me details around how I can access the tar.gz

Please feel free to resolve this issue. With the above solution we don't need mavenlocal, but it will be good to have as first thing a developer look is the maven local for the artifacts created locally.

[saratvemulapalli]

Feel free to reopen this.
credits to: @Rishikesh1159 who built customDistributionUrl