[Enhancement] Adding TYPE Support With archives/packages/dirs Options

[peterzhuamazon]

This issue is a expansion of #2517.

We want to add a TYPE parameter (-Dbuild.type ?) during core build, TYPE=archives/packages/dirs.
User can do distribution=rpm;type=dirs and they will be provided with directory of files that is built for rpm, instead of rpm pkg as a final result.

This is specifically useful when we want to install additional plugins, more changes, extra tweaks before packaging them into final release artifacts, without the need to untar/unzip/extract rpm before repackaging them back.

This would also avoid many issues during the extraction phase and we can completely remove the extraction out of the code base of opensearch-build repo.

Thanks.

[peterzhuamazon]

cc: @saratvemulapalli @anasalkouz @bbarani

[peterzhuamazon]

Initially we think running distribution:packages:rpm:build would give me extracted files.
Which runs pretty well in 1.2.0.

In 1.3.0 it failed as well and turns out it is using the same cpio utility.
See #2517.

./gradlew distribution:packages:rpm:build

> Task :distribution:packages:rpm:checkExtraction FAILED
Output for rpm:
warning: Unable to get systemd shutdown inhibition lock
warning: Unable to get systemd shutdown inhibition lock
error: unpacking of archive failed on file /local/home/zhujiaxi/OpenSearch/distribution/packages/rpm/build/extracted/rpm-extracted/usr/share/opensearch/jdk/man/ja: cpio: rename failed - Is a directory
error: opensearch-0:1.3.1~SNAPSHOT-1.x86_64: install failed

This is really a blocker now if we cannot fix it in the core.
But seems like it needs to delete ja as well.
Core needs to find a way to retain the files before the rpm generation even happens.

[xuezhou25]

It's not easy to retain the files before the rpm package, because packaging process is controlled by Gradle nebula ospackage plugin.
As far as I know, ospackage plugin doesn't give the opportunity to preserve source files before packaging.
Currently in OpenSearch build script, the files used for packaging is gathered in separate steps (for example:
https://github.com/opensearch-project/OpenSearch/blob/1.3.0/distribution/packages/build.gradle#L157-L175).
There is no existing directory that contains all the files to generate rpm/deb packages.
My thought is to copy all the required files into a directory before using the ospackage plugin, but it will lead to much code modification.
@saratvemulapalli @dblock

[dblock]

I agree with @xuezhou25. We're building RPM in core, but then we want to add things to RPM and rebuild RPM again for the bundle build. I think if we want directory output, or some other output we need to add a different distribution target.

[anasalkouz]

@peterzhuamazon what is your thought on the suggested approach?

[peterzhuamazon]

@dblock @saratvemulapalli This PR #2602 is raised to resolve #2517 temporarily.
@xuezhou25 is working on resolving this issue on a separate PR.

Thanks.

[andrross]

Closing this in favor of #2665 to track a longer term solution here