#2 Add Github Action build workflow #40

aaron-gary · 2024-05-02T21:40:47Z

There will be a follow on to this PR to handle certain build issues such as the limited amount of disk space on the Action Runner.

# This is the 1st commit message: # This is a combination of 4 commits. # This is the 1st commit message: #2 Add Maven build workflow # This is the commit message #2: #2 update build workflow # This is the commit message #3: #2 add branch checkout to build workflow # This is the commit message #4: #2 Add on event to build workflow # This is the commit message #2: #2 Remove unused executions # This is the commit message #3: #2 Focus on build # This is the commit message #4: #2 Remove build execution where not needed # This is the commit message #5: #2 debug failing module # This is the commit message #6: #2 Remove unused target folder copy # This is the commit message #7: #2 Build spark and jenkins docker images # This is the commit message #8: #2 Retry full build # This is the commit message #9: #2 Omitting module # This is the commit message #10: #2 Fix for out of disk space # This is the commit message #11: #2 tagging docker images # This is the commit message #12: #2 Remove Temporarily remove Docker module # This is the commit message #13: #2 Build update # This is the commit message #14: #2 Build update # This is the commit message #15: #2 move chart dry-runs to IT profile # This is the commit message #16: #2 curl delta-hive assembly in docker build # This is the commit message #17: #2 cache m2 repo # This is the commit message #18: #2 prune docker build cache between images to save space # This is the commit message #19: #2 add maven build-cache to GH cache # This is the commit message #20: #2 run clean goal in build to clear docker cache # This is the commit message #21: #2 set maven caches to always save even if the build failed # This is the commit message #22: #2 adjust number of docker modules built # This is the commit message #23: #2 use the same cache for .m2 even if poms change # This is the commit message #24: #2 change from `save-always` flag to `if: always()` see actions/cache#1315 # This is the commit message #25: #2 further reduce docker images being built # This is the commit message #26: #2 disable modules that depend on helm charts # This is the commit message #27: #2 use maven wrapper # This is the commit message #28: #2 restore modules to test build-cache # This is the commit message #29: #2 fix build of modules with intra-project chart dependencies # This is the commit message #30: #2 use explict .m2 repo cache so we can fall-back to older caches # This is the commit message #31: #2 save maven caches on build failure

ewilkins-csi · 2024-05-02T21:45:31Z

.github/workflows/build.yml

+        type: string
+        default: '2-build-action'
+  push:
+    branches: [ "2-build-action" ]


S: Should update these to dev. It might also make sense to have this run when a PR to dev is open and use the PR branch as the input build branch. But that can come in later iterations once we get the dev branch building locally.

Good catch

Switch to dev branch

ewilkins-csi · 2024-05-02T21:48:34Z

...nsions/extensions-docker/aissemble-spark-infrastructure/src/main/resources/docker/Dockerfile

@@ -4,8 +4,9 @@ FROM ${DOCKER_BASELINE_REPO_ID}boozallen/aissemble-spark:${VERSION_AISSEMBLE}

 LABEL org.opencontainers.image.source = "https://github.com/boozallen/aissemble"

+RUN curl -L https://github.com/delta-io/connectors/releases/download/v0.6.0/delta-hive-assembly_2.12-0.6.0.jar \
+          -o "${SPARK_HOME}"/jars/delta-hive-assembly_2.12-0.6.0.jar


A: Should probably pass the connector version as a docker build arg from the POM so we can drive version numbers from Maven.

Agree

Move version to build arg

ewilkins-csi · 2024-05-02T21:50:49Z

extensions/extensions-docker/pom.xml

-        <module>aissemble-model-training-api-containers</module>
+<!--        <module>aissemble-versioning</module>-->
+<!--        <module>aissemble-mlflow</module>-->
+<!--        <module>aissemble-model-training-api-containers</module>-->


S: Revert this as local developers have the appropriate access keys setup to be able to build all of the images.

Agree

uncomment modules

ewilkins-csi · 2024-05-02T21:52:10Z

Don't forget to rebase. I reviewed but didn't approve as I probably helped author too many of the changes to be a valid reviewer.

carter-cundiff · 2024-05-02T22:41:44Z

extensions/extensions-docker/aissemble-airflow/src/main/resources/docker/Dockerfile

@@ -5,9 +5,7 @@ LABEL org.opencontainers.image.source = "https://github.com/boozallen/aissemble"

 USER root

-COPY ./target/cacerts/* /usr/local/share/ca-certificates/
-
-RUN update-ca-certificates && apt-get update && apt-get install -y --no-install-recommends apt-utils


Isn't this gonna make the images unbuildable from a BAH device since they needs the certs?

If you log out of your VPN to do the build you should be able to build just fine. But we can't really install certs specific to one organization. I think long-term it'd be nice to find some sort of just-in-time solution that allows VPN CA certs to be made available without actually modifying the final image.

…sion to pom file.

aaron-gary · 2024-05-03T04:53:22Z

Don't forget to rebase. I reviewed but didn't approve as I probably helped author too many of the changes to be a valid reviewer.

rebase

d-ryan-ashcraft · 2024-05-03T12:13:03Z

extensions/extensions-docker/aissemble-spark/src/main/resources/docker/Dockerfile


 # git and build-essential are used during pip install of some dependencies (e.g. python-deequ)
-# software-properties-common and *-dev dependencies are needed to build an updated version of Python
-# as Python 3.11.4 is packaged by default with the base spark-py image
 RUN update-ca-certificates && apt-get update -y && apt-get install --assume-yes \


S: looks like we might have missed the update-ca-certificates here

Good catch

Remove update-ca-certificates

aaron-gary added 2 commits May 2, 2024 14:36

remove unused code

2b85eff

ewilkins-csi reviewed May 2, 2024

View reviewed changes

carter-cundiff reviewed May 2, 2024

View reviewed changes

aaron-gary added 2 commits May 2, 2024 21:49

#2 Change build workflow to target dev branch. Extract delta hive ver…

1c5eff1

…sion to pom file.

#2 Add modules back in

abe7551

d-ryan-ashcraft approved these changes May 3, 2024

View reviewed changes

#2 Remove ca-certs update

fa4c79e

aaron-gary merged commit 23dde27 into dev May 3, 2024

aaron-gary deleted the 2-build-action branch May 3, 2024 15:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2 Add Github Action build workflow #40

#2 Add Github Action build workflow #40

aaron-gary commented May 2, 2024

ewilkins-csi May 2, 2024

aaron-gary May 3, 2024 •

edited

Loading

ewilkins-csi May 2, 2024

aaron-gary May 3, 2024 •

edited

Loading

ewilkins-csi May 2, 2024

aaron-gary May 3, 2024 •

edited

Loading

ewilkins-csi commented May 2, 2024 •

edited

Loading

carter-cundiff May 2, 2024

ewilkins-csi May 2, 2024

aaron-gary commented May 3, 2024 •

edited

Loading

d-ryan-ashcraft May 3, 2024

aaron-gary May 3, 2024 •

edited

Loading

#2 Add Github Action build workflow #40

#2 Add Github Action build workflow #40

Conversation

aaron-gary commented May 2, 2024

ewilkins-csi May 2, 2024

Choose a reason for hiding this comment

aaron-gary May 3, 2024 • edited Loading

Choose a reason for hiding this comment

ewilkins-csi May 2, 2024

Choose a reason for hiding this comment

aaron-gary May 3, 2024 • edited Loading

Choose a reason for hiding this comment

ewilkins-csi May 2, 2024

Choose a reason for hiding this comment

aaron-gary May 3, 2024 • edited Loading

Choose a reason for hiding this comment

ewilkins-csi commented May 2, 2024 • edited Loading

carter-cundiff May 2, 2024

Choose a reason for hiding this comment

ewilkins-csi May 2, 2024

Choose a reason for hiding this comment

aaron-gary commented May 3, 2024 • edited Loading

d-ryan-ashcraft May 3, 2024

Choose a reason for hiding this comment

aaron-gary May 3, 2024 • edited Loading

Choose a reason for hiding this comment

aaron-gary May 3, 2024 •

edited

Loading

aaron-gary May 3, 2024 •

edited

Loading

aaron-gary May 3, 2024 •

edited

Loading

ewilkins-csi commented May 2, 2024 •

edited

Loading

aaron-gary commented May 3, 2024 •

edited

Loading

aaron-gary May 3, 2024 •

edited

Loading