-
Notifications
You must be signed in to change notification settings - Fork 41
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
166: Customize builds for different platforms r=jgallag88 a=jgallag88
This change provides the ability to build versions of the appliance
customized for different platforms (hypervisors and cloud providers).
This is done by installing a different versions of the delphix-platform
and delphix-kernel packages depending on which platform we are
building for. Since we only want to have a single upgrade image per
variant, this change also adds a second stage to the build which
combines the live-build output for multiple platform versions of the
same variant into a single upgrade tarball.
The live-build stage of the build is now run by invoking 'gradle' with a
a target which is a combination of variant and platform, e.g.
`gradle buildInternalDevEsx`.
The second stage of the build is run by invoking 'gradle' with a variant
as a target, e.g. `gradle buildUpgradeImageInternalDev`. When the second
stage is run, an environment variable 'AWS_S3_URI_LIVEBUILD_ARTIFACTS'
can be passed. If it is used, previously built live-build artifacts will
be downloaded from the provided S3 URIs, and placed in
`live-build/build/artifacts` as if they had been built locally. If it is
not used, live-build will be invoked for each of the hypervisors
specified in the 'DELPHIX_PLATFORMS' environment variable.
A couple notes about the implementation:
1. This change replaces the Make build with a Gradle one. The build logic
needed for this change was difficult to express using Make and
resulted in a Makefile which was very difficult to understand. The
use of Gradle make handling this build logic more straightforward
and also made it possible to add better support for incremental
builds.
2. This change removes the idea of the 'base' live-build variant. The
base variant contains the kernel, and because the kernel differs
between hypervisors, it cannot be shared between different hypervisor
builds. It would be possible to have a different version of the base variant
per hypervisor, and share that between different variants built for
the same hypervisor. However, this likely isn't worth the effort
because it doesn't help in either of the two most common use cases:
- Building via a jenkins job: when building via Jenkins, each
variant will now be built via a sub-job running on its own build
VM, so the base would be rebuilt for each sub-job anyway.
- Developers iterating on changes on personal build VMs: in this
case developers are most likely to be building a single variant,
in which case the 'base' variant would be less likely to be
re-used.
3. We no longer do the live-build in place (that is, directly in
`live-build/variant/<variant>/`). Now that we have multiple builds per
variant, we need to make sure that intermediate live-build
artifacts from one build are not incorrectly re-used in the next
build of the same variant, which might be for a different
hypervisor. The simplest way to accomplish this is just to do the
live-build in a throw-away directory.
In light of these two changes, some of current layout of the
repository no longer makes sense, so this change re-arranges a number
of files in the repo, particularly in the `live-build/` directory.
Co-authored-by: John Gallagher <[email protected]>- Loading branch information
Showing
76 changed files
with
822 additions
and
427 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,19 @@ | ||
| ancillary-repository | ||
| # | ||
| # Copyright 2018 Delphix | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| .gradle/ | ||
| .gradleUserHome/ | ||
| build/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| /* | ||
| * Copyright 2019 Delphix | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| apply plugin: 'base' | ||
|
|
||
| apply from: "${rootProject.projectDir}/gradle-lib/util.gradle" | ||
|
|
||
| // Build upgrade images for KVM if no platforms are specified via an environment variable | ||
| def DEFAULT_PLATFORMS = 'kvm' | ||
|
|
||
| createArtifactsDirTask(this) | ||
|
|
||
| for (variant in allVariants) { | ||
| def taskName = "buildUpgradeImage${toCamelCase(variant).capitalize()}" | ||
| tasks.create(taskName, Exec) { task -> | ||
| group = 'Build' | ||
| description = "Builds an upgrade image for the ${variant} variant of the appliance" | ||
| dependsOn mkArtifactsDir | ||
|
|
||
| /* | ||
| * When building an upgrade image, there are two ways to get the *.debs.tar.gz artifacts | ||
| * that are produced by live build and consumed by the build-upgrade-image.sh script. We | ||
| * can directly run live build for the appropriate platforms for this variant, or we can | ||
| * fetch from S3 the artifacts from previous runs of live-build. Which strategy we use is | ||
| * controlled by the AWS_S3_URI_LIVEBUILD_ARTIFACTS and DELPHIX_PLATFORMS env variables, | ||
| * so check them and set the appropriate task dependencies. | ||
| */ | ||
| if (System.getenv("AWS_S3_URI_LIVEBUILD_ARTIFACTS") != null) { | ||
| dependsOn ":live-build:fetchLiveBuildArtifacts" | ||
| } else { | ||
| def platforms = System.getenv("DELPHIX_PLATFORMS") ?: DEFAULT_PLATFORMS | ||
| for (platform in platforms.trim().split()) { | ||
| def dependentTask = "build" + | ||
| toCamelCase(variant).capitalize() + | ||
| platform.capitalize() | ||
| dependsOn ":live-build:${dependentTask}" | ||
| } | ||
| } | ||
|
|
||
| for (envVar in ["DELPHIX_PLATFORMS", "AWS_S3_URI_LIVEBUILD_ARTIFACTS"]) { | ||
| inputs.property(envVar, System.getenv(envVar)).optional(true) | ||
| } | ||
|
|
||
| doFirst { | ||
| if (System.getenv("AWS_S3_URI_LIVEBUILD_ARTIFACTS") == null && | ||
| System.getenv("DELPHIX_PLATFORMS") == null) { | ||
|
|
||
| logger.quiet(""" | ||
| Neither 'AWS_S3_URI_LIVEBUILD_ARTIFACTS' nor 'DELPHIX_PLATFORMS' is defined as an | ||
| environment variable, so this upgrade image will be built for the default platform | ||
| ('${DEFAULT_PLATFORMS}'). To change which platforms are included in the image, | ||
| re-run with DELPHIX_PLATFORMS set to a space-delimited list of platforms for which | ||
| to build (e.g 'DELPHIX_PLATFORMS="esx aws kvm" gradle ...') or with | ||
| AWS_S3_URI_LIVEBUILD_ARTIFACTS set to a space-delimited set of S3 URIs from which | ||
| to fetch previously built live-build artifacts. | ||
| """.stripIndent()) | ||
| } | ||
| } | ||
|
|
||
| commandLine "${rootProject.projectDir}/scripts/build-upgrade-image.sh", "${variant}" | ||
| } | ||
| } | ||
|
|
||
| def shellScripts = fileTree("scripts") + | ||
| fileTree("live-build/config/hooks").include({ details -> | ||
| details.file.canExecute() | ||
| }) + | ||
| fileTree("live-build/misc/migration-scripts") + | ||
| fileTree("upgrade/upgrade-scripts", { | ||
| exclude "README.md" | ||
| }) | ||
|
|
||
| task shfmt(type: Exec) { | ||
| commandLine(["shfmt", "-w"] + shellScripts.getFiles()) | ||
| } | ||
|
|
||
| task shfmtCheck(type: Exec) { | ||
| commandLine(["shfmt", "-d"] + shellScripts.getFiles()) | ||
| } | ||
|
|
||
| task shellCheck(type: Exec) { | ||
| commandLine(["shellcheck", "--exclude=SC1090,SC1091"] + shellScripts.getFiles()) | ||
| } | ||
|
|
||
| task ansibleCheck(type: Exec) { | ||
| def ansibleFiles = fileTree("bootstrap").include("**/playbook.yml") + | ||
| fileTree("live-build/variants").include("**/playbook.yml") | ||
| commandLine(["ansible-lint", "--exclude=SC1090,SC1091"] + ansibleFiles.getFiles()) | ||
| } | ||
|
|
||
| tasks.check.dependsOn shellCheck, shfmtCheck, ansibleCheck | ||
|
|
||
| task format() { | ||
| dependsOn shfmt | ||
| group = "Formatting" | ||
| description "Runs all auto-formatting tasks" | ||
| } |
Oops, something went wrong.