Make dependency updates more efficient

[ThorbenLindhauer]

This issue was imported from JIRA:

Field	Value
JIRA Link	CAM-14890
Reporter	@toco-cam
Has restricted visibility comments	false

---

Acceptance Criteria (Required on creation):

• Assess where we spend time to perform dependency updates
• The time is reduced as much as possible within reasonable effort

Hints (optional):

• E.g. evaluate dependency check tools (that other teams use) to (semi-)automate the license checks. Find a good tradeoff between time saved and probability of mistakes

Links:

PRs

Preview Give feedback

1. feat: add SBOM-based dependency diff workflow automation-platform-github-actions#14
   +0
   
2. https://github.com/camunda/infra-core/pull/6629

Rollout PRs

Preview Give feedback

1. chore(project): add new dependency check workflow #3799
   ci:no-build +1
   
2. fix(github): reference reusable workflow correctly #3823
   bot:java-dependency-check ci:no-build +2
   
3. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1085
4. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1075
5. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1076
6. https://github.com/camunda/camunda-bpm-platform-ee/pull/866
7. https://github.com/camunda/camunda-bpm-platform-ee-maintenance/pull/444
8. https://github.com/camunda/camunda-bpm-platform-ee-maintenance/pull/441
9. https://github.com/camunda/camunda-bpm-platform-ee-maintenance/pull/442
10. chore(project): add new dependency check workflow camunda-connect#84
    +0
    
11. chore(project): add new dependency check workflow camunda-spin#212
    +0
    
12. chore(project): add new dependency check workflow camunda-commons#37
    +0
    
13. chore(project): add new dependency check workflow camunda-template-engines-jsr223#32
    +0
    
14. https://github.com/camunda/camunda-bpm-rpa-bridge-ee/pull/90

[ThorbenLindhauer]

Experiment for automatically detecting licenses:

1. Run mvn org.codehaus.mojo:license-maven-plugin:2.0.0:aggregate-download-licenses in the root directory of the repo.
2. Compare license book to generated report

Findings:

• Most licenses are detected correctly
• Lists both licenses for dual-licensed libraries (e.g. logback-classic, h2) => needs education
• Deps not included in report:
  • jackson-dataformat-csv
• Deps with mismatching license:
  • jcip-annotations (reported: Apache; license book: Creative commons) => this is very likely a mistake in the license book
  • jakarta.activation-api (reported: EDL 1.0; license book: BSD-3-Clause)
  • jersey-* dependencies (reported: ten licenses, cannot be accurate)
  • javassist (reported: three licenses, not sure if accurate)
  • asm-* (reported: BSD, license book: BSD-3-Clause)

[ThorbenLindhauer]

https://www.mojohaus.org/license-maven-plugin/aggregate-add-third-party-mojo.html and its configuration are quite powerful, e.g. the parameter templateFile can be used to create a JSON document (or any other text format).

[ThorbenLindhauer]

Same experiment as above using https://github.com/CycloneDX/cyclonedx-maven-plugin/:

Dependencies not detected:

• jackson-dataformat-csv
  • is only used in EE repo which I didn't scan
• juel-api, juel-impl, juel-spi
  • was removed recently and replaced with jakarta.el-api, which is detected correctly
• [email protected] (only 2.3.6 is found)
  • wasn't part of the Maven Reactor because I built with Java 8

Incorrect licenses:

• jcip-annotations (reported as Apache, is Creative Commons Attribution in license book)
  • seems to be a mistake in the license book, it is Apache according to https://github.com/stephenc/jcip-annotations; the book itself (https://jcip.net/) is probably CCA
• jakarta.activation (reported as BSD-3-Clause, is EDL 1.0 in the license book)
• jaxb-impl 2.3.6 (reported as BSD-3-Clause, is EDL 1.0 in the license book)
• jakarta.xml.bind-api (reported as BSD-3-Clause, is EDL 1.0 in the license book)
  • In the POM.xml, EDL 1.0 is declared. In the license header, BSD-3-Clause is declared. Maybe Maven reads this comment when detecting the license. The license texts of the two licenses are 99% identical (note that https://www.eclipse.org/org/documents/edl-v10.php also gives BSD-3 Clause as the SPDX identifier for the EDL 1.0 license)
  • CycloneDX's Java component maps these licenses to each other, where BSD-3-Clause is the canonical name, see https://github.com/CycloneDX/cyclonedx-core-java/blob/cyclonedx-core-java-7.3.2/src/main/resources/license-mapping.json#L45
  • The other EDL 1.0 vs BSD-3-clause probably have a similar reason
• Jersey dependencies: reports 10 licenses (including the correct one)
  • A parent POM of the jersey artifacts declares of all these licenses, whereas the license header in the POM file clearly mentions dual-licensing under EPL 2.0 and GPL 2.0 with classpath exception. It seems that this is a list of licenses of third-party dependencies that the Jersey project shades/packages into various artifacts (see comments in the code; Jersey license clarification eclipse-ee4j/jersey#5004 (comment)).
• jboss-jaxb-api_2.3_spec (reported as BSD-3-Clause, is Apache 2.0 in the license book)
  • The license book content is incorrect, it is EDL 1.0 (and reports as BSD-3-Clause for similar reasons as above)
• asm-* (reported as BSD-4-clause, is BSD-3-Clause in the license book)
  • The POM for the asm 7.1 dependencies (e.g. asm-analysis) declares the license as

    <licenses>
      <license>
        <name>BSD</name>
        <url>http://asm.ow2.org/license.html</url>
      </license>
    </licenses>
    

  • Newer versions use BSD-3-Clause in the name attribute. CycloneDX interprets BSD as the original BSD-4-Clause license: https://github.com/CycloneDX/cyclonedx-core-java/blob/cyclonedx-core-java-7.3.2/src/main/resources/license-mapping.json#L73
• wildfly-* (included in the report but without license; LGPL 2.1 in license book)
  • Not clear why this is not detected, it is declared correctly in the parent POM

The plugin gets the license information from Maven (I assume that this means it retrieves whatever is declared in the pom.xml of the Maven artifact). See https://github.com/CycloneDX/cyclonedx-maven-plugin/blob/cyclonedx-maven-plugin-2.7.9/src/main/java/org/cyclonedx/maven/DefaultModelConverter.java#L297 and calling code.

[ThorbenLindhauer]

Idea for the tool chain:

• Generate an aggregate SBOM for the base and head commit using mvn org.cyclonedx:cyclonedx-maven-plugin:2.7.9:makeAggregateBom
• Parse the SBOM. Perform a diff, starting with every Camunda module in the reactor, compute and diff the transitive hull of (third-party) dependencies (using the dependency graph; note that we use the same dependencies with different versions in various modules, so simply detecting that two third-party dependencies have different versions before and after doesn't quite cut it)
• For every module with changes, render the changed versions along with their licenses and instructions (based on FOSS policy) what to do

Side notes:

• There is a cyclonedx-cli that has a diffing feature, but it is very limited (note that the JSON output format has much more details than the default plain text one)
• I considered parsing the SBOM with CycloneDX JS library, but deserialization is not a feature yet (feat: Deserializer/Denormalizer CycloneDX/cyclonedx-javascript-library#86).
• We can consider running a second scan for test dependencies (see https://github.com/CycloneDX/cyclonedx-maven-plugin/#default-values for how to configure scopes in the Maven plugin)
• I noticed that in the aggregate BOM created by the Maven plugin, sometimes test dependencies show up if they are in the compile (or another included scope) somewhere else (e.g. junit shows up as a dependency of camunda-xml-model where it is a test dependency, which is likely caused by it being in compile scope in the engine module)
• SBOM JSON reference: https://cyclonedx.org/docs/1.4/json/#bomFormat

[ThorbenLindhauer]

Test PRs:

• Jackson update
• Spring Boot update
• EE Jackson downgrade

Breakdown of remaining tasks:

• Use proper Maven settings so that there are no more artifact transfer failures
  • Reuse the maven settings.xml from java-dependency-tree-diff: https://github.com/camunda/java-dependency-tree-diff/blob/enterprise/maven-settings.xml
  • We can use that in both open and closed source repos. There is no issue with Artifactory credentials leaking on external PRs as for those the vault access secrets are not accessible (confer https://docs.github.com/en/actions/security-guides/encrypted-secrets#using-encrypted-secrets-in-a-workflow)
  • Not mirroring all downloads to Artifactory but using Maven Central as much as possible seems to significantly reduce runtime (25 vs 10 minutes on one example PR)
• Make sure the workflow works in a private/internal repo (e.g. EE)
  • In the EE repo, I see warnings as in this comment
  • This is because newer Maven versions use a mirror declaration (maven-default-http-blocker) in the global settings XML that redirects any http repository requests to the non-existent 0.0.0.0. Nevertheless, the artifacts can be resolved by other declared repositories, so that is not a problem.
  • Note that http repositories are not declared in our code but in some third-party POMs, e.g. javax.enterprise:cdi-api:1.0-SP4
  • The problem could be solved in the following ways:
    • Use the internal Artifactory group as a mirror for everything => Maven Central is atm much faster and significantly reduces the time of the build (see above)
    • Override the mirror declaration in the build's settings => Could work, but not sure about precedence. Also, it may not be obvious which repository we would use as the mirror
    • Update the third-party dependencies => a lot of effort
  • Accordingly, we are accepting the warnings for the time being
• Analyze oddities and decide if they need a fix (e.g. same dependency versions may be reported as a change)
• Resolve all open TODOs in the code
  • Unresolved code issue: The reusable workflow cannot checkout its own repository in the current github ref. The corresponding property github.job_workflow_sha that is documented to provide this does not work.
• Review tests and test coverage
• Resolve warnings in test output
• Review and if necessary improve file structure of the shared workflow
• Let the workflow be triggered by a label
• Improve performance if reasonable
  • Build time is roughly 12 minutes in the platform mono repo, which should be the one that takes longest. Downloading Maven dependencies has the largest performance impact (e.g. the build runs much faster locally with a populated local Maven repo)
  • Maven build parallelization via -T parameter is not useful, becasue the makeAggregateBom goal does all of its work in the root module (the remaining modules are skipped) and -T only parallelizes by module
  • Doubling the number of threads for artifact download (via -Dmaven.artifact.threads=10) only has a slight positive effect (10:30 minutes vs 12:00 minutes build time)
  • Breadth-first resolution of POM files (which in turn would parallelize POM downloads; via https://stackoverflow.com/questions/32299902/parallel-downloads-of-maven-artifacts) is not available by default, because Maven is currently still on version 3.8.8 in the ubuntu-latest runner (https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#project-management)
  • Using dependency caching has two drawbacks:
    1. By default, the cache key is based on the hashes of the POM files, i.e. any POM change invalidates the cache. However, the nature of this workflow is to apply when the POM has changed. This problem is solvable one way or the other.
    2. PR builds can only access the caches of the main branch, the base branch, and its own branch. They cannot access caches from other PRs. In consequence, we would have to keep caches for the master branch (e.g. by introducing an additional workflow that creates the cache periodically on the master branch). This would add quite some extra complexity to our github workflows
  • In consequence, I am not implementing any of the suggestions. It is ok to wait for ~15 minutes for this analysis. Changing depdendencies doesn't happen too often.
• Add CI that runs the unit tests on merge and PRs
• Enable Vault access for all repositories
• Prepare PRs for all repositories
• Add the bot label to all repositories
• Communicate changes to the team (will happen on team meeting 10.10.2023)
• Document how to debug unexpected behavior (and potentially write code that allows to easily run the diff for two given SBOM files)
• Document known limitations
  • Cases
    • When a dependency is declared in multiple versions and pinned to one by Maven, the SBOM plugin may report the declared (and overridden), not the actual version of the dependency
• Make a follow-up for retiring the current workflow (won't do; there is a bit of value to still have the other workflow and the team can make this decision further down the road)

[ThorbenLindhauer]

Blocked HTTP repository example output:

Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Downloading from camunda-public-repository: https://artifacts.camunda.com/artifactory/public/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Downloading from maven-default-http-blocker: http://0.0.0.0/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Progress (1): 622 B
                   
Downloaded from camunda-public-repository: https://artifacts.camunda.com/artifactory/public/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml (622 B at 6.1 kB/s)
Downloading from JBoss public: https://repository.jboss.org/nexus/content/groups/public/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Progress (1): 622 B
                   
Downloading from sonatype-oss-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml (622 B at 2.3 kB/s)
Progress (1): 622 B
                   
Downloaded from sonatype-oss-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml (622 B at 1.3 kB/s)
[WARNING] Could not transfer metadata org.camunda.bpm.webapp:camunda-webapp-root:7.20.0-SNAPSHOT/maven-metadata.xml from/to maven-default-http-blocker (http://0.0.0.0/): transfer failed for http://0.0.0.0/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/camunda-webapp-root-7.20.0-20230818.070330-58.pom
Progress (1): 4.1/10 kB
Progress (1): 7.8/10 kB
Progress (1): 10 kB    
                   
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/camunda-webapp-root-7.20.0-20230818.070330-58.pom (10 kB at 42 kB/s)
[WARNING] org.camunda.bpm.webapp:camunda-webapp-root:7.20.0-SNAPSHOT/maven-metadata.xmlfailed to transfer from http://0.0.0.0/ during a previous attempt. This failure was cached in the local repository and resolution will not be reattempted until the update interval of maven-default-http-blocker has elapsed or updates are forced. Original error: Could not transfer metadata org.camunda.bpm.webapp:camunda-webapp-root:7.20.0-SNAPSHOT/maven-metadata.xml from/to maven-default-http-blocker (http://0.0.0.0/): transfer failed for http://0.0.0.0/org/camunda/bpm/webapp/camunda-webapp-root/7.20.0-SNAPSHOT/maven-metadata.xml

[ThorbenLindhauer]

Repos that we will roll out the workflow to:

camunda/camunda-bpm-platform
camunda/camunda-bpm-platform-maintenance
camunda/camunda-bpm-platform-ee
camunda/camunda-bpm-platform-ee-maintenance
camunda/camunda-connect
camunda/camunda-spin
camunda/camunda-commons
camunda/camunda-template-engines-jsr223
camunda/camunda-bpm-rpa-bridge-ee

[ThorbenLindhauer]

@danielkelemen I have added the 7.20 PRs and corrected the workflow references. Please re-review.

[ThorbenLindhauer]

❓Should we also include the older but still active branches for spin, commons and connect? Or it's unnecessary.

Forgot to respond here. I decided to not do backports there to save some effort. Anyone who needs it can still do it when they work with such a branch.