Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#2 Add Github Action build workflow #40

Merged
merged 5 commits into from
May 3, 2024
Merged

#2 Add Github Action build workflow #40

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bc94735
# This is a combination of 31 commits.
aaron-gary Apr 30, 2024
2b85eff
remove unused code
aaron-gary May 2, 2024
1c5eff1
#2 Change build workflow to target dev branch. Extract delta hive ve…
aaron-gary May 3, 2024
abe7551
#2 Add modules back in
aaron-gary May 3, 2024
fa4c79e
#2 Remove ca-certs update
aaron-gary May 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven

# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

name: Build aissemble

on:
workflow_dispatch:
inputs:
buildBranch:
description: "Branch you want to build"
required: true
type: string
default: '2-build-action'
push:
branches: [ "2-build-action" ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: Should update these to dev. It might also make sense to have this run when a PR to dev is open and use the PR branch as the input build branch. But that can come in later iterations once we get the dev branch building locally.

Copy link
Contributor Author

@aaron-gary aaron-gary May 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch

  • Switch to dev branch


jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.buildBranch }}
- name: Install Python # use direct install rather than pyenv for CI for large speed improvement
uses: actions/setup-python@v5
with:
python-version: '3.11.4'
- name: Load cached Poetry installation
id: cached-poetry
uses: actions/cache@v4
with:
path: ~/.local
key: poetry-0 # increment to reset cache
- name: Load m2 repository cache # Manually caching .m2 repo as the setup-java caching isn't falling back to older caches
id: cached-m2-repo
uses: actions/cache@v4
with:
path: ~/.m2/repository
key: maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
maven-
- name: Load m2 build cache
id: cached-m2-build
uses: actions/cache@v4
with:
path: ~/.m2/build-cache
key: maven-build-cache-${{ hashFiles('**/pom.xml') }}
restore-keys: |
maven-build-cache-
- name: Install Poetry
if: steps.cached-poetry.outputs.cache-hit != 'true'
uses: snok/install-poetry@v1
- name: Set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Install Docker
run: |
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
- name: Install Helm
run: |
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
- name: Build aiSSEMBLE
run: ./mvnw -B clean install --file pom.xml -Dhabushu.usePyenv=false -Pci
#NB: The following two explicit cache saves are necessary to ensure caches are saved on build failure, until
# https://github.com/actions/cache/issues/1315 is resolved
- name: Save m2 repository cache
id: save-m2-repo
uses: actions/cache/save@v4
if: always()
with:
path: ~/.m2/repository
key: maven-${{ hashFiles('**/pom.xml') }}
- name: Save m2 build cache
id: save-m2-build
uses: actions/cache/save@v4
if: always()
with:
path: ~/.m2/build-cache
key: maven-build-cache-${{ hashFiles('**/pom.xml') }}
45 changes: 0 additions & 45 deletions build-parent/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -963,51 +963,6 @@ To suppress enforce-helm-version rule, you must add following plugin to the root
</build>
</profile>

<!-- The profile aiops-release must be active when a project is released.
The configuration in this POM will automatically call this profile if using
the maven-release-plugin. If the maven-release-plugin is not used during
the release, this profile must be manually activated. -->
<profile>
<id>aiops-release</id>
<build>
<plugins>
<!-- Create a source-release artifact that contains the fully buildable
project directory source structure. This should be released to the Maven
repository for each AIOps project release. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<updateReleaseInfo>true</updateReleaseInfo>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>doclint-java8-disable</id>
<activation>
Expand Down
29 changes: 0 additions & 29 deletions extensions/extensions-docker/aissemble-airflow/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,33 +14,4 @@

<packaging>orphedomos</packaging>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack</id>
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>extensions-docker-cacerts</artifactId>
<version>${project.version}</version>
<overWrite>true</overWrite>
<outputDirectory>${project.build.directory}/cacerts</outputDirectory>
<type>zip</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
4 changes: 1 addition & 3 deletions extensions/extensions-docker/aissemble-airflow/src/main/resources/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@ LABEL org.opencontainers.image.source = "https://github.com/boozallen/aissemble"

USER root

COPY ./target/cacerts/* /usr/local/share/ca-certificates/

RUN update-ca-certificates && apt-get update && apt-get install -y --no-install-recommends apt-utils
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this gonna make the images unbuildable from a BAH device since they needs the certs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you log out of your VPN to do the build you should be able to build just fine. But we can't really install certs specific to one organization. I think long-term it'd be nice to find some sort of just-in-time solution that allows VPN CA certs to be made available without actually modifying the final image.

RUN apt-get update && apt-get install -y --no-install-recommends apt-utils
RUN apt-get update && apt-get install -y \
ant \
fontconfig \
Expand Down
30 changes: 0 additions & 30 deletions extensions/extensions-docker/aissemble-jenkins/aissemble-jenkins-controller/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,34 +13,4 @@
<description>A base image for a Jenkins Controller</description>

<packaging>orphedomos</packaging>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack</id>
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>extensions-docker-cacerts</artifactId>
<version>${project.version}</version>
<overWrite>true</overWrite>
<outputDirectory>${project.build.directory}/cacerts</outputDirectory>
<type>zip</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
6 changes: 0 additions & 6 deletions ...ocker/aissemble-jenkins/aissemble-jenkins-controller/src/main/resources/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,6 @@ FROM jenkins/jenkins:lts-jdk11
LABEL org.opencontainers.image.source = "https://github.com/boozallen/aissemble"

USER root
COPY ./target/cacerts/* /usr/local/share/ca-certificates/
# Update CA certificates and import each certificate in the directory
RUN update-ca-certificates && \
for cert in /usr/local/share/ca-certificates/*.crt; do \
keytool -import -trustcacerts -keystore "$JAVA_HOME/lib/security/cacerts" -storepass changeit -noprompt -file "$cert"; \
done
USER jenkins

RUN jenkins-plugin-cli --plugins git:4.11.2
Expand Down
30 changes: 0 additions & 30 deletions extensions/extensions-docker/aissemble-nvidia/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,34 +13,4 @@
<description>A base image for a Nvidia Cuda implementation</description>

<packaging>orphedomos</packaging>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack</id>
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>extensions-docker-cacerts</artifactId>
<version>${project.version}</version>
<overWrite>true</overWrite>
<outputDirectory>${project.build.directory}/cacerts</outputDirectory>
<type>zip</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
4 changes: 1 addition & 3 deletions extensions/extensions-docker/aissemble-nvidia/src/main/resources/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,7 @@ RUN printf '#!/bin/bash\n\nif [[ "$1" == "-r" ]] ;then\n echo '4.9.250'\n exit\n
RUN chmod 755 /bin/uname
# Workaround for ubuntu fix end

COPY ./target/cacerts/* /usr/local/share/ca-certificates/

RUN update-ca-certificates && apt-get update \
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y openjdk-11-jdk \
&& update-ca-certificates \
Expand Down
5 changes: 0 additions & 5 deletions extensions/extensions-docker/aissemble-spark-infrastructure/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,6 @@
<artifactId>delta-storage</artifactId>
<version>${version.delta}</version>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-hive-assembly_2.12</artifactId>
<version>${version.delta.hive.connector}</version>
</dependency>
</dependencies>

<build>
Expand Down
3 changes: 2 additions & 1 deletion ...ons/extensions-docker/aissemble-spark-infrastructure/src/main/resources/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ FROM ${DOCKER_BASELINE_REPO_ID}boozallen/aissemble-spark:${VERSION_AISSEMBLE}

LABEL org.opencontainers.image.source = "https://github.com/boozallen/aissemble"

RUN curl -L https://github.com/delta-io/connectors/releases/download/v0.6.0/delta-hive-assembly_2.12-0.6.0.jar \
-o "${SPARK_HOME}"/jars/delta-hive-assembly_2.12-0.6.0.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A: Should probably pass the connector version as a docker build arg from the POM so we can drive version numbers from Maven.

Copy link
Contributor Author

@aaron-gary aaron-gary May 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree

  • Move version to build arg

ARG JARS_DIR

ADD ${JARS_DIR}/* ${SPARK_HOME}/jars/

ENV SPARK_NO_DAEMONIZE=true
Expand Down
25 changes: 0 additions & 25 deletions extensions/extensions-docker/aissemble-spark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,31 +61,6 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack</id>
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>extensions-docker-cacerts</artifactId>
<version>${project.version}</version>
<overWrite>true</overWrite>
<outputDirectory>${project.build.directory}/cacerts</outputDirectory>
<type>zip</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Loading
Loading