[BUG] Snapshot build: Artifact opensearch-job-scheduler-1.1.0.0.jar is invalid

[peternied]

Describe the bug
Artifact opensearch-job-scheduler-1.1.0.0.jar is invalid, expected to have Implementation-Version=any of ['1.1.0.0-SNAPSHOT', '1.1.0-SNAPSHOT', None], but was 1.1.0.0.

Error building job-scheduler, retry with

	/var/jenkins/workspace/bundle-build@2/bundle-workflow/build.sh manifests/opensearch-1.1.0.yml --component job-scheduler --snapshot

Traceback (most recent call last):
  File "/var/jenkins/workspace/bundle-build@2/bundle-workflow/src/build.py", line 61, in <module>
    builder.export_artifacts()
  File "/var/jenkins/workspace/bundle-build@2/bundle-workflow/src/build_workflow/builder.py", line 50, in export_artifacts
    self.component_name, artifact_type, relative_path, absolute_path
  File "/var/jenkins/workspace/bundle-build@2/bundle-workflow/src/build_workflow/build_recorder.py", line 101, in record_artifact
    self.__check_artifact(artifact_type, artifact_file)
  File "/var/jenkins/workspace/bundle-build@2/bundle-workflow/src/build_workflow/build_recorder.py", line 122, in __check_artifact
    self.__check_maven_artifact(artifact_file)
  File "/var/jenkins/workspace/bundle-build@2/bundle-workflow/src/build_workflow/build_recorder.py", line 165, in __check_maven_artifact
    [self.component_version, self.opensearch_version, None],
  File "/var/jenkins/workspace/bundle-build@2/bundle-workflow/src/build_workflow/build_recorder.py", line 58, in check_value_in
    f"expected to have {key}=any of {expected_values}, but was {value}",
build_workflow.build_recorder.ArtifactInvalidError: Artifact opensearch-job-scheduler-1.1.0.0.jar is invalid: expected to have Implementation-Version=any of ['1.1.0.0-SNAPSHOT', '1.1.0-SNAPSHOT', None], but was 1.1.0.0.

To Reproduce
Run a fresh snapshot build on the build server

Expected behavior
No failures

[peternied]

@dblock I thought this was addressed with opensearch-project/job-scheduler#49 Did we need additional follow up?

[dblock]

It was at the time. I'll take a look.

[dblock]

Looks like we're running the snapshot and non-snapshot builds on the same host in parallel which is stepping all over itself. Making the build independent would be "hard" since we build into subdirectories, pickup dependencies from maven local, put files into maven local and publish locally as well.

[peternied]

Offline: Neither of us could reproduce locally, it looks like the build job is running on the same node in parallel, I'll need to force allocation onto more than one node at a time instead of the current reuse pattern

SNAPSHOT Node Allocation

Running on EC2 (Amazon_ec2_cloud) - Jenkins-Agent-al2-x64-m5xlarge (i-0414b194e1bea86ac) in /var/jenkins/workspace/bundle-build

x86 Node Allocation

Running on EC2 (Amazon_ec2_cloud) - Jenkins-Agent-al2-x64-m5xlarge (i-0414b194e1bea86ac) in /var/jenkins/workspace/bundle-build@2