Skip to content

Commit

Permalink
Merge pull request #478 from boozallen/aissemble-release-1.10
Browse files Browse the repository at this point in the history
Update to 1.11.0-SNAPSHOT
  • Loading branch information
ewilkins-csi authored Nov 20, 2024
2 parents a059d57 + be5511d commit b48bad6
Show file tree
Hide file tree
Showing 356 changed files with 186 additions and 5,688 deletions.
4 changes: 0 additions & 4 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,6 @@ jobs:
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.GPG_PRIVATE_KEY_PASSPHRASE }}
- name: Install git
run: |
sudo apt-get update
sudo apt install -y git
- uses: actions/checkout@v4
- name: Split version
id: split_version
Expand Down
76 changes: 13 additions & 63 deletions DRAFT_RELEASE_NOTES.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,22 @@
# Major Additions

## Java 17 Upgrade
The aiSSEMBLE project is now built with Java 17. All dependencies inherited from aiSSEMBLE have been updated to a Java 17 compatible version and automatic migrations (detailed below) have been created to streamline this transition for downstream projects. It is recommended to review non-inherited dependencies and custom logic within your project, as this upgrade can break existing functionality due to deprecated classes and incompatible dependencies.

# Breaking Changes
Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

- All upgrading projects must be built with Java 17 to ensure compatibility with this release.
- The new minimum required Maven version is now `3.9.6` to ensure compatibility with Java 17.
- We are dropping the support for the data and model lineage event legacy namespace option (`data.lineage.namespace`). This means you will need to define your own namespace for each event in the `data-lineage.properties` file, such that Jobs are tied to pipelines and Datasets are tied to data sources. For more guidance, please refer to the [Lineage Metadata GitHub documentation](https://boozallen.github.io/aissemble/aissemble/current-dev/lineage-medatada-capture-overview.html#_configuration).
_Note: instructions for adapting to these changes are outlined in the upgrade instructions below._

- _There are no breaking changes in teh 1.11 release._

# Known Issues
_There are no known issues with the 1.10 release._
_There are no known issues with the 1.11 release._

# Known Vulnerabilities
| Date<br/>identified | Vulnerability | Severity | Package | Affected <br/>versions | CVE | Fixed <br/>in |
|---------------------|---------------------------------------------------------------------------|----------|---------------------------|------------------------|-------------------------------------------------------------------|---------------|

| Date<br/>identified | Vulnerability | Severity | Package | Affected <br/>versions | CVE | Fixed <br/>in |
|---------------------|---------------|----------|---------|------------------------|-----|---------------|


# How to Upgrade
The following steps will upgrade your project to `1.10`. These instructions consist of multiple phases:

The following steps will upgrade your project to `1.11`. These instructions consist of multiple phases:
- Automatic Upgrades - no manual action required
- Precondition Steps - needed in all situations
- Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
Expand All @@ -34,15 +30,6 @@ To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate t
| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile |
| upgrade-v2-chart-files-aissemble-version-migration | Updates the Helm chart dependencies within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE |
| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE |
| spark-version-upgrade-migration | Updates the Spark Application executor failure parameters to their new key name to ensure compatibility with Apache Spark `3.5` |
| spark-pipeline-messaging-pom-migration | Updates a Spark pipeline module `pom.xml` with the new CDI classes dependency to ensure messaging compatibility with Java 17 |
| spark-pipeline-messaging-cdi-factory-migration | Updates a Spark pipeline module `CdiContainerFactory.java` with the new CDI classes to ensure messaging compatibility with Java 17 |
| spark-pipeline-servlet-api-migration | Updates a Spark pipeline module `pom.xml` with the `javax.servlet-api` dependency to ensure compatibility with Apache Spark `3.5` which has not migrated to Jakarta packages yet |
| it-infrastructure-java-upgrade-migration | Updates the Java docker image version in the integration test docker module to JDK 17 |
| log4j-maven-shade-plugin-migration | Updates the Maven Shade Plugin with the new Log4j dependency information |
| quarkus-bom-migration | Updates all references to the `quarkus-bom` and `quarkus-universe-bom` to use the new `aissemble-quarkus-bom` for managing Quarkus dependencies |
| pom-dependency-version-migration | Updates the pom dependencies previously managed by the aiSSEMBLE `bom-component` to include their necessary versions |
| java-package-migration | Updates the affected java classes from their old package name to their new package name to ensure compatibility with the updated Java 17 dependencies. This migration primarily updates `javax.*` packages to their new `jakarta.*` packages |

To deactivate any of these migrations, add the following configuration to the `baton-maven-plugin` within your root `pom.xml`:

Expand All @@ -69,60 +56,23 @@ To deactivate any of these migrations, add the following configuration to the `b
## Precondition Steps - Required for All Projects

### Beginning the Upgrade
To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.10.0 version of the build-parent:
To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.11.0 version of the build-parent:
```xml
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>build-parent</artifactId>
<version>1.10.0</version>
<version>1.11.0</version>
</parent>
```

### BOM Component Replacement
All references to the `com.boozallen.aissemble:bom-component` should be replaced with the new `com.boozallen.aissemble:aissemble-quarkus-bom`:
```diff
<dependency>
<groupId>com.boozallen.aissemble</groupId>
- <artifactId>bom-component</artifactId>
+ <artifactId>aissemble-quarkus-bom</artifactId>
<version>${version.aissemble}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
```

## Conditional Steps

### Split Data Records for the Spark Pipeline
If your spark pipeline is using `aissemble-data-records-separate-module` profile for your data records, you must add the `<version>` tag for
the `jackson-mapper-asl` dependency artifact in the root pom.xml file to enable the build.
```xml
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
+ <version>${version.jackson.mapper.asl}</version>
</dependency>
```

### Kafka Clients Dependency Update
Data Access and the Java testing module (`<YOUR_PROJECT>-tests/<YOUR_PROJECT>-tests-java/pom.xml`) no longer need to explicitly set the `kafka-clients` dependency. It can be removed along with its exclusion from the `quarkus-smallrye-reactive-messaging-kafka` dependency.

### For projects that have customized the Spark Operator Service Account permissions
The service account for the pipeline invocation service is now separated from spark operator and configured solely for the service.
If you added any custom configurations to the `sparkoperator` service account pertaining to the pipeline invocation service, you will need to migrate the related changes to the new `pipeline-invocation-service-sa`. Refer to Pipeline Invocation Helm Chart [README](https://github.com/boozallen/aissemble/blob/dev/extensions/extensions-helm/extensions-helm-pipeline-invocation/aissemble-pipeline-invocation-app-chart/README.md) for detail.

## Final Steps - Required for All Projects
### Finalizing the Upgrade
1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations
2. Remove the `com.boozallen.aissemble:aissemble-quarkus-bom:${version.aissemble}` dependency from the root `pom.xml` and `<YOUR_PROJECT>-tests/<YOUR_PROJECT>-tests-java/pom.xml` of your project
- **NOTE:** Any Quarkus apps within your project should be updated to include the `com.boozallen.aissemble:aissemble-quarkus-bom:${version.aissemble}` within the `<dependencyManagement>` section of their respective `pom.xml` instead
3. Run `./mvnw clean install` and resolve any manual actions that are suggested
2. Run `./mvnw clean install` and resolve any manual actions that are suggested
- **NOTE:** This will update any aiSSEMBLE dependencies in 'pyproject.toml' files automatically
4. Repeat the previous step until all manual actions are resolved
3. Repeat the previous step until all manual actions are resolved

# What's Changed
- `pyproject.toml` files updated to allow for Python version `>=3.8`.
- All SageMaker modules and corresponding references were removed in commit 8ce393f.
- The Service Discovery module and corresponding references were removed in commit d10db5d0.
- Leveraging a new version of Booz Allen Licenses. Booz Allen projects may find some minor updates to header file text
that better conforms to some strict linting standards.
_to be auto-generated when published_
2 changes: 1 addition & 1 deletion bom/aissemble-quarkus-bom/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>bom</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>aissemble-quarkus-bom</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion bom/bom-instantiation/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>bom</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>bom-instantiation</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion bom/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<groupId>com.boozallen.aissemble</groupId>
<artifactId>build-parent</artifactId>
<relativePath>../build-parent/pom.xml</relativePath>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>bom</artifactId>
Expand Down
4 changes: 2 additions & 2 deletions build-parent/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>aissemble-root</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>build-parent</artifactId>
Expand All @@ -22,7 +22,7 @@
</distributionManagement>

<properties>
<version.aissemble>1.10.0-SNAPSHOT</version.aissemble>
<version.aissemble>1.11.0-SNAPSHOT</version.aissemble>

<!-- **************** -->
<!-- Plugins versions -->
Expand Down
2 changes: 1 addition & 1 deletion build-support/aissemble-enforcer-extension/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>build-support</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>
<artifactId>aissemble-enforcer-extension</artifactId>

Expand Down
2 changes: 1 addition & 1 deletion build-support/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>aissemble-root</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>build-support</artifactId>
Expand Down
6 changes: 2 additions & 4 deletions cucumber-report-aggregator/pom.xml
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>aissemble-root</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>cucumber-report-aggregator</artifactId>
Expand Down
4 changes: 2 additions & 2 deletions docs/antora.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: aissemble
title: aiSSEMBLE
version: 1.10.0
display_version: 1.10.0-SNAPSHOT
version: 1.11.0
display_version: 1.11.0-SNAPSHOT
prerelease: true
nav:
- modules/ROOT/nav.adoc
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/ROOT/attachments/HardwareDetailsRunFacet.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.10.0-SNAPSHOT/HardwareDetailsRunFacet.json",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.11.0-SNAPSHOT/HardwareDetailsRunFacet.json",
"$defs": {
"HardwareDetailsRunFacet": {
"allOf": [
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/ROOT/attachments/HyperparameterRunFacet.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.10.0-SNAPSHOT/HyperparameterRunFacet.json",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.11.0-SNAPSHOT/HyperparameterRunFacet.json",
"$defs": {
"HyperparameterRunFacet": {
"allOf": [
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/ROOT/attachments/MLflowRunFacet.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.10.0-SNAPSHOT/MLflowRunFacet.json",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.11.0-SNAPSHOT/MLflowRunFacet.json",
"$defs": {
"MLflowRunFacet": {
"allOf": [
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.10.0-SNAPSHOT/PerformanceMetricRunFacet.json",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.11.0-SNAPSHOT/PerformanceMetricRunFacet.json",
"$defs": {
"PerformanceMetricRunFacet": {
"allOf": [
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.10.0-SNAPSHOT/SourceCodeDirectoryJobFacet.json",
"$id": "https://boozallen.github.io/aissemble/openlineage/schemas/1.11.0-SNAPSHOT/SourceCodeDirectoryJobFacet.json",
"$defs": {
"SourceCodeDirectoryJobFacet": {
"allOf": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-alerting</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>extensions-alerting-slack</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-alerting</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>extensions-alerting-teams</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-alerting/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>extensions-alerting</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-data-delivery</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>aissemble-extensions-data-delivery-spark-py</artifactId>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "aissemble-extensions-data-delivery-spark-py"
version = "1.10.0.dev"
version = "1.11.0.dev"
description = "Contains the core Python functionality of data delivery for Spark"
authors = ["aiSSEMBLE Baseline Community <[email protected]>"]
readme = "README.md"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-data-delivery</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>extensions-data-delivery-spark-neo4j</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-data-delivery</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>extensions-data-delivery-spark-postgres</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-data-delivery</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>extensions-data-delivery-spark</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-data-delivery/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>
<packaging>pom</packaging>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-data-lineage</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>
<artifactId>extensions-data-lineage-http-consumer-service</artifactId>
<name>aiSSEMBLE::Extensions::Data Lineage::Http Consumer Service</name>
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-data-lineage/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>
<packaging>pom</packaging>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-docker</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>aissemble-configuration-store</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>extensions-docker</artifactId>
<groupId>com.boozallen.aissemble</groupId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<packaging>docker-build</packaging>
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-docker/aissemble-fastapi/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-docker</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>aissemble-fastapi</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>extensions-docker</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>aissemble-hive-service</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>aissemble-jenkins</artifactId>
<version>1.10.0-SNAPSHOT</version>
<version>1.11.0-SNAPSHOT</version>
</parent>

<artifactId>aissemble-jenkins-agent</artifactId>
Expand Down
Loading

0 comments on commit b48bad6

Please sign in to comment.