[Bug]: Distribution builds can have path collisions

[peternied]

Distribution builds are not namespace'd based on the type of artifact produced so an OpenSearch and OpenSearch Dashboard build could collide.

Actual

The current path generated is https://ci.opensearch.org/ci/dbc/builds/1.1.0/328/x64/manifest.yml.

Expected

It should have the distribution type in the path like https://ci.opensearch.org/ci/dbc/opensearch/builds/1.1.0/328/x64/manifest.yml and https://ci.opensearch.org/ci/dbc/opensearch-dashboards/builds/1.1.0/328/x64/manifest.yml

Required changes

• Update the build jobs to include the build type in the url for the generated manifests, see bundle_recorder.py.
• Update the jenkinsfile to upload to a destination based on the build type, see Jenkinsfile.

Other changes

• Migrate/ update existing artifacts to the new destination
• Update the roles to be separate between jobs to prevent any possible conflicts, then update the jenkins jobs to build to the correct destination.
• [Bug]: The location specified in bundle-manifest (manifest.yml) are invalid #661
• [Bug]: Build manifest has broken URLs missing architecture from path #714

[dblock]

What is dbc, btw?

[dblock]

Let's make sure to address #661 when we change these.

[peternied]

Looks like #714 is related to this change too

[dblock]

Should we consider merging the output of build and assemble and renaming bundle as well? #692 and #479?

We could end up with a single folder that contains build output and bundle:

https://ci.opensearch.org/ci/dbc/opensearch/builds/1.1.0/328/x64/min/openseach-min-1.1.0.tar.gz (currently bundle/...)
https://ci.opensearch.org/ci/dbc/opensearch/builds/1.1.0/328/x64/dist/openseach-1.1.0.tar.gz (now separate under bundles)
https://ci.opensearch.org/ci/dbc/opensearch/builds/1.1.0/328/x64/maven/...

[zelinh]

Regarding of "Migrate/ update existing artifacts to the new destination" sub-task, I think we should not change the location of existing artifacts because we didn't update the old paths for these artifacts.

[peternied]

@zelinh You can close out this task if you don't think we should do the migration. We don't have a strict policy around CI artifact storage so I'm fine with them haveing a different schema and eventually disappearing.

#788 is a much more targeted migration which would make sense to honor.

[zelinh]

I am closing out this issue since the problem on path collisions is resolved.