Support subfolders for distribution types for build and assemble loca…

…lly and in Jenkins (#1807) * Support subfolders for distribution types for build and assemble Signed-off-by: Tianle Huang <[email protected]> * fix syntax Signed-off-by: Tianle Huang <[email protected]> * update upload logic Signed-off-by: Tianle Huang <[email protected]> * add distribution Signed-off-by: Tianle Huang <[email protected]> * fix uploadMinSnapshotsToS3 Signed-off-by: Tianle Huang <[email protected]> * temp comment out most plugins to speed up test Signed-off-by: Tianle Huang <[email protected]> * modify Signed-off-by: Tianle Huang <[email protected]> * fix min paths Signed-off-by: Tianle Huang <[email protected]> * test against tar for now Signed-off-by: Tianle Huang <[email protected]> * address comments Signed-off-by: Tianle Huang <[email protected]> * add security Signed-off-by: Tianle Huang <[email protected]> * add pa Signed-off-by: Tianle Huang <[email protected]> * bring back all Signed-off-by: Tianle Huang <[email protected]> * fix file extension Signed-off-by: Tianle Huang <[email protected]> * add dot Signed-off-by: Tianle Huang <[email protected]> * use my repo for snapshot Signed-off-by: Tianle Huang <[email protected]> * set multiple stages Signed-off-by: Tianle Huang <[email protected]> * fix names Signed-off-by: Tianle Huang <[email protected]> * fix var Signed-off-by: Tianle Huang <[email protected]> * double quotes Signed-off-by: Tianle Huang <[email protected]> * fix min extensions Signed-off-by: Tianle Huang <[email protected]> * fix env name Signed-off-by: Tianle Huang <[email protected]> * add rpm file Signed-off-by: Tianle Huang <[email protected]> * fix name Signed-off-by: Tianle Huang <[email protected]> * undo osd Signed-off-by: Tianle Huang <[email protected]> * move distribution to beginning Signed-off-by: Tianle Huang <[email protected]> * fix url Signed-off-by: Tianle Huang <[email protected]> * remove extra whitespaces Signed-off-by: Tianle Huang <[email protected]> * add text Signed-off-by: Tianle Huang <[email protected]> * use 1.3 branch Signed-off-by: Tianle Huang <[email protected]> * fix syntax Signed-off-by: Tianle Huang <[email protected]> * fix syntax Signed-off-by: Tianle Huang <[email protected]> * use quotes Signed-off-by: Tianle Huang <[email protected]> * remove rpm snapshot Signed-off-by: Tianle Huang <[email protected]> * add tests and use peter's own account Signed-off-by: Tianle Huang <[email protected]> * fix more python tests Signed-off-by: Tianle Huang <[email protected]> * fix style Signed-off-by: Tianle Huang <[email protected]> * fix missing types Signed-off-by: Tianle Huang <[email protected]> * remove wrong import Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestArchiveAssembleUpload Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestAssembleUpload Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestBuildAssembleUpload Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestBuildManifest Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestPromoteArtifacts Signed-off-by: Tianle Huang <[email protected]> * fix TestUploadMinSnapshotsToS3 Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestArchiveAssembleUpload Signed-off-by: Tianle Huang <[email protected]> * fix ut of TestBuildAssembleUpload Signed-off-by: Tianle Huang <[email protected]> * combine updates after tests are done Signed-off-by: Tianle Huang <[email protected]> * fix syntax Signed-off-by: Tianle Huang <[email protected]> * reset to 1.3.0 main Signed-off-by: Tianle Huang <[email protected]> * create a new stage for upload index.json file Signed-off-by: Tianle Huang <[email protected]> * fix syntax Signed-off-by: Tianle Huang <[email protected]> * add agent Signed-off-by: Tianle Huang <[email protected]> * pass path directly Signed-off-by: Tianle Huang <[email protected]> * fix groovy tests Signed-off-by: Tianle Huang <[email protected]> * fix groovy tests Signed-off-by: Tianle Huang <[email protected]> * add comment Signed-off-by: Tianle Huang <[email protected]> * update how tests use path Signed-off-by: Tianle Huang <[email protected]> * run osd rpm Signed-off-by: Tianle Huang <[email protected]> * fix Signed-off-by: Tianle Huang <[email protected]> * fix tests Signed-off-by: Tianle Huang <[email protected]> * fix style Signed-off-by: Tianle Huang <[email protected]> * fix archive logic Signed-off-by: Tianle Huang <[email protected]> * fix unzipped Signed-off-by: Tianle Huang <[email protected]> * use new image Signed-off-by: Tianle Huang <[email protected]> * fix syntax Signed-off-by: Tianle Huang <[email protected]> * fix docker Signed-off-by: Tianle Huang <[email protected]> * revert Signed-off-by: Tianle Huang <[email protected]> * remove commented code Signed-off-by: Tianle Huang <[email protected]> * add distribution to archived-builds.zip Signed-off-by: Tianle Huang <[email protected]> * fix copyArtifacts Signed-off-by: Tianle Huang <[email protected]> * undo manifest Signed-off-by: Tianle Huang <[email protected]> * restore official repo name Signed-off-by: Tianle Huang <[email protected]> * fix failed ut Signed-off-by: Tianle Huang <[email protected]> * fix ut Signed-off-by: Tianle Huang <[email protected]> * update docker build and notification Signed-off-by: Tianle Huang <[email protected]> * fix echo Signed-off-by: Tianle Huang <[email protected]> * hardcode to test again Signed-off-by: Tianle Huang <[email protected]> * fix Signed-off-by: Tianle Huang <[email protected]> * restore manifest Signed-off-by: Tianle Huang <[email protected]> * fix test Signed-off-by: Tianle Huang <[email protected]> * fix ut Signed-off-by: Tianle Huang <[email protected]> * add ut for UploadIndexFile Signed-off-by: Tianle Huang <[email protected]> * fix tst Signed-off-by: Tianle Huang <[email protected]>
opensearch-project · Apr 8, 2022 · 7c4cb71 · 7c4cb71
1 parent 91d0a57
commit 7c4cb71
Show file tree

Hide file tree

Showing 74 changed files with 3,384 additions and 351 deletions.
diff --git a/.gitignore b/.gitignore
@@ -21,6 +21,10 @@ out.txt
 /dist/
 /test-results/
 
+/rpm/
+/tar/
+/zip/
+
 /.vscode/
 
 .project

diff --git a/jenkins/opensearch-dashboards/distribution-build.jenkinsfile b/jenkins/opensearch-dashboards/distribution-build.jenkinsfile
diff --git a/jenkins/opensearch/distribution-build.jenkinsfile b/jenkins/opensearch/distribution-build.jenkinsfile
diff --git a/src/assemble_workflow/bundle_file_location.py b/src/assemble_workflow/bundle_file_location.py
@@ -10,8 +10,8 @@
 
 
 class BundleFileLocation(BundleLocation):
-    def __init__(self, path: str, filename: str) -> None:
-        super().__init__(path, filename)
+    def __init__(self, path: str, filename: str, distribution: str) -> None:
+        super().__init__(path, filename, distribution)
 
     def join(self, *args: str) -> str:
         return os.path.join(self.path, *args)
diff --git a/src/assemble_workflow/bundle_location.py b/src/assemble_workflow/bundle_location.py
@@ -8,16 +8,17 @@
 
 
 class BundleLocation(abc.ABC):
-    def __init__(self, path: str, filename: str) -> None:
+    def __init__(self, path: str, filename: str, distribution: str) -> None:
         self.path = path
         self.filename = filename
+        self.distribution = distribution
 
     @abc.abstractmethod
     def join(self, *args: str) -> str:
         pass
 
     def get_build_location(self, target_name: str) -> str:
-        return self.join("builds", self.filename, target_name)
+        return self.join(self.distribution, "builds", self.filename, target_name)
 
     def get_bundle_location(self, target_name: str) -> str:
-        return self.join("dist", self.filename, target_name)
+        return self.join(self.distribution, "dist", self.filename, target_name)
diff --git a/src/assemble_workflow/bundle_locations.py b/src/assemble_workflow/bundle_locations.py
@@ -13,5 +13,5 @@
 class BundleLocations:
 
     @classmethod
-    def from_path(cls, url_path: str, file_path: str, filename: str) -> BundleLocation:
-        return BundleUrlLocation(url_path, filename) if url_path else BundleFileLocation(file_path, filename)
+    def from_path(cls, url_path: str, file_path: str, filename: str, distribution: str) -> BundleLocation:
+        return BundleUrlLocation(url_path, filename, distribution) if url_path else BundleFileLocation(file_path, filename, distribution)
diff --git a/src/assemble_workflow/bundle_url_location.py b/src/assemble_workflow/bundle_url_location.py
@@ -10,8 +10,8 @@
 
 
 class BundleUrlLocation(BundleLocation):
-    def __init__(self, path: str, filename: str) -> None:
-        super().__init__(path, filename)
+    def __init__(self, path: str, filename: str, distribution: str) -> None:
+        super().__init__(path, filename, distribution)
 
     def join(self, *args: str) -> str:
         sub_path = "/".join(args)

diff --git a/src/build_workflow/build_args.py b/src/build_workflow/build_args.py
@@ -68,7 +68,7 @@ def __init__(self) -> None:
             type=str,
             choices=self.SUPPORTED_DISTRIBUTIONS,
             help="Distribution to build.",
-            default=None,
+            default="tar",
             dest="distribution"
         )
 

diff --git a/src/build_workflow/builder_from_source.py b/src/build_workflow/builder_from_source.py
@@ -45,7 +45,7 @@ def build(self, build_recorder: BuildRecorder) -> None:
                     f"-q {self.target.qualifier}" if self.target.qualifier else None,
                     f"-p {self.target.platform}",
                     f"-a {self.target.architecture}",
-                    f"-d {self.target.distribution}" if self.target.distribution and (self.component.name in DISTRIBUTION_SUPPORTED_COMPONENTS) else None,
+                    f"-d {self.target.distribution}" if self.component.name in DISTRIBUTION_SUPPORTED_COMPONENTS else None,
                     f"-s {str(self.target.snapshot).lower()}",
                     f"-o {self.output_path}",
                 ]

diff --git a/src/jenkins/BuildManifest.groovy b/src/jenkins/BuildManifest.groovy
@@ -15,13 +15,15 @@ class BuildManifest implements Serializable {
         String version
         String platform
         String architecture
+        String distribution
 
         Build(Map data) {
             this.id = data.id
             this.name = data.name
             this.version = data.version
             this.platform = data.platform
             this.architecture = data.architecture
+            this.distribution = data.distribution
         }
 
         String getFilename() {
@@ -31,7 +33,17 @@ class BuildManifest implements Serializable {
         String getFilenameWithExtension(String platform = null, String architecture = null) {
             String resolvedPlatform = platform ?: this.platform
             String resolvedArchitecture = architecture ?: this.architecture
-            return "${this.getFilename()}-${this.version}-${resolvedPlatform}-${resolvedArchitecture}.${resolvedPlatform == 'windows' ? 'zip' : 'tar.gz'}"
+            return "${this.getFilename()}-${this.version}-${resolvedPlatform}-${resolvedArchitecture}.${getExtension()}"
+        }
+
+        String getExtension() {
+            if (this.distribution == 'zip') {
+                return 'zip'
+            } else if (this.distribution == 'rpm') {
+                return 'rpm'
+            } else {
+                return 'tar.gz'
+            }
         }
 
         String getPackageName() {
@@ -40,7 +52,7 @@ class BuildManifest implements Serializable {
                     this.version,
                     this.platform,
                     this.architecture,
-            ].join('-') + '.tar.gz'
+            ].join('-') + '.' + getExtension()
         }
     }
 
@@ -87,7 +99,8 @@ class BuildManifest implements Serializable {
                 this.build.version,
                 buildNumber,
                 this.build.platform,
-                this.build.architecture
+                this.build.architecture,
+                this.build.distribution
         ].join("/")
     }
 

diff --git a/src/paths/assemble_output_dir.py b/src/paths/assemble_output_dir.py
@@ -8,5 +8,5 @@
 
 
 class AssembleOutputDir(OutputDir):
-    def __init__(cls, filename: str, cwd: str = None, makedirs: bool = True) -> None:
-        super().__init__("dist", filename, cwd, makedirs)
+    def __init__(cls, filename: str, distribution: str, cwd: str = None, makedirs: bool = True) -> None:
+        super().__init__("dist", filename, distribution, cwd, makedirs)
diff --git a/src/paths/build_output_dir.py b/src/paths/build_output_dir.py
@@ -9,5 +9,5 @@
 
 
 class BuildOutputDir(OutputDir):
-    def __init__(self, filename: str, cwd: Optional[str] = None, makedirs: bool = True) -> None:
-        super().__init__("builds", filename, cwd, makedirs)
+    def __init__(self, filename: str, distribution: str, cwd: Optional[str] = None, makedirs: bool = True) -> None:
+        super().__init__("builds", filename, distribution, cwd, makedirs)
diff --git a/src/paths/output_dir.py b/src/paths/output_dir.py
@@ -10,9 +10,10 @@
 
 
 class OutputDir(ABC):
-    def __init__(cls, parent_dir: str, filename: str, cwd: Optional[str] = None, makedirs: bool = True) -> None:
+    def __init__(cls, parent_dir: str, filename: str, distribution: str, cwd: Optional[str] = None, makedirs: bool = True) -> None:
         cls.dir = os.path.join(
             cwd or os.getcwd(),
+            distribution,
             parent_dir,
             filename
         )

diff --git a/src/run_assemble.py b/src/run_assemble.py
@@ -28,15 +28,15 @@ def main() -> int:
     build = build_manifest.build
     artifacts_dir = os.path.dirname(os.path.realpath(args.manifest.name))
 
-    output_dir = AssembleOutputDir(build.filename).dir
+    output_dir = AssembleOutputDir(build.filename, build.distribution).dir
 
     logging.info(f"Bundling {build.name} ({build.architecture}) on {build.platform} into {output_dir} ...")
 
     bundle_recorder = BundleRecorder(
         build,
         output_dir,
         artifacts_dir,
-        BundleLocations.from_path(args.base_url, os.getcwd(), build.filename)
+        BundleLocations.from_path(args.base_url, os.getcwd(), build.filename, build.distribution)
     )
 
     with Bundles.create(build_manifest, artifacts_dir, bundle_recorder, args.keep) as bundle:

diff --git a/src/run_build.py b/src/run_build.py
@@ -38,7 +38,7 @@ def main():
             manifest.to_file(args.ref_manifest)
         exit(0)
 
-    output_dir = BuildOutputDir(manifest.build.filename).dir
+    output_dir = BuildOutputDir(manifest.build.filename, args.distribution).dir
 
     with TemporaryDirectory(keep=args.keep, chdir=True) as work_dir:
         logging.info(f"Building in {work_dir.name}")

diff --git a/tests/data/opensearch-build-1.1.0.yml b/tests/data/opensearch-build-1.1.0.yml
@@ -5,6 +5,7 @@ build:
   id: c3ff7a232d25403fa8cc14c97799c323
   name: OpenSearch
   version: 1.1.0
+  distribution: tar
 components:
   - artifacts:
       dist:
-Original file line number
+Diff line change
@@ Expand Up / @@ -21,6 +21,10 @@ out.txt @@
     /dist/
     /test-results/
+    /rpm/
+    /tar/
+    /zip/
     /.vscode/
     .project
@@ Expand Down @@