Tool to generate BOM file #22417

pallavit · 2021-06-21T17:33:17Z

The tool generates the BOM in the following way:

Picks up all the eligible SDK dependencies from the versioning_client.txt file.
Create a map of all dependencies these libraries bring in (non-transitively) and flatten them as a tree map of dependency name (groupid:artifactId) ->version ->(parent dependencies that brought in this dependency i.e. all the ancestors).
For all the libraries with multiple versions (conflicting situations) picks the latest version as the winner, and removes the other versions and their respective parent dependencies as non-eligible dependencies.
Gets the new list of eligible dependencies from step(3) and step(1).
And rerun step(2) to get the complete set to be put in the BOM.
Run the validation step to ensure BOM resolution results in no conflicts.
Write the new BOM.

TODO:

Have better reporting\logging. Currently it logs to the console, all the libraries that had to be dropped and the reason why they were dropped.
Have a better algo to pick the winner in the conflicting list. As of now, this algo picks the highest version which may be the right choice for core libraries but may not be true for say libraries like communication-common etc.

pallavit · 2021-07-07T15:01:58Z

While creating dependency tree for the external BOMs - I came across - https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-bom/2.11.3/jackson-bom-2.11.3.pom - which uses properties to version control the different dependencies - and it made me think that this is a good way to automatically move all our SDK dependencies to latest core (have the core library version defined as properties in parent POM, and then make all SDKs use that property as the version, shipping core would then entail updating the property and all libraries will automatically pick that up). Any reason - why this approach won't work, should not be used?

/cc: @srnagar , @alzimmermsft , @JonathanGiles

In reply to: 875680300

pallavit · 2021-07-07T19:27:51Z

Discussed offline with @srnagar. We tried a similar approach in the past where we added dependencyManagement to the parent POM but soon got into trouble because of it. Multiple hierarchial dependency - so what we have is what we stay with!

In reply to: 875680300

weshaggard · 2021-07-08T18:49:42Z

eng/bomgenerator/buildAndRun.cmd

+set "versioningClientFileLocation=%~dp0..\versioning\version_client.txt"
+set "bomPomFileLocation=%~dp0..\..\sdk\boms\azure-sdk-bom\pom.xml"
+set "outputFileLocation=%~dp0..\..\sdk\boms\azure-sdk-bom\newpom.xml"
+mvn clean install && mvn exec:java -Dexec.args="-inputFile=%versioningClientFileLocation% -outputFile=%outputFileLocation% -pomFile=%bomPomFileLocation%"


Out of curiosity is there a reason you decided to write a java based tool for generating this? Not that it matters too much but most of the other versioning tools are written in python so this would need to duplicate a chunk of that logic. Do we think we should move more of the versioning tooling to java? #Resolved

There were three main reasons 1. I rely on a library shrinkwrap that basically emulates maven assembly resolution to create the dependency map for each library. I also depend on maven reader and writer library which is again available for java. (I believe there are solutions that can call java methods from python directly but I did not delve into that much) 2. This can in future become a plug-in and be part of a build step. 3. My vested interest in getting more comfy in Java.

When you say duplication of code - do you mean parsing the files and versions etc, or other things too?

Do we think we should move more of the versioning tooling to java?

This is a tough one. I see why python is the language of choice here? But at the same time for scenarios like BOM having things in java and reuse library\solutions from the ecosystem is beneficial.

Thanks for the context. I don't have an issue with this, especially if this becomes a plugin and produced as part of the maven build.

The duplication I was thinking about is the parsing and understanding of our version files. It isn't overly complicated but there are some interesting rules in there to understand dependency vs current versions as well as our unreleased/beta versions.

One of the biggest reason we use things like python and powershell scripts for our versioning and other engsys tasks is to avoid needing to do a restore and build step before using the tool. Interpreted languages are much better suited for those type of tasks. For that reason it might be kind of difficult to implement all the versioning rules in a java based tool, at least one that lives in this repo. If we did use a java tool for these type of things I'd suggest we move the building/publishing of that tool into another repo.

I think @srnagar also said having a plugin which lived in a repo like azure-sdk would be a better place to put this. How about I make this the first step and use it to polish the tool a little and once it is mature and ready to be turned into a plugin, I can move it a more suited location.

One of the biggest reason we use things like python and powershell scripts for our versioning and other engsys tasks is to avoid needing to do a restore and build step before using the tool. Interpreted languages are much better suited for those type of tasks.

I did not think of this .. thanks!

If you do end up putting this in another repo the right place for it would be https://github.com/Azure/azure-sdk-tools/tree/main/tools.

eng/bomgenerator/pom.xml

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/Main.java

srnagar · 2021-07-09T21:01:30Z

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/BomGenerator.java

+    BomGenerator() {
+    }


This can be removed #Resolved

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyVersionComparator.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/BomGenerator.java

srnagar · 2021-07-10T03:20:05Z

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyAnalyzer.java

+    }
+
+
+    private void makeDependencyInEligible(BomDependency dependency) {


Add logs to indicate why dependencies are marked as ineligible. #Pending

Yes, I have a work-item to add better logging\reporting in the system.

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyAnalyzer.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/Utils.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/BomGenerator.java

eng/bomgenerator/buildAndRun.cmd

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/BomGenerator.java

alzimmermsft · 2021-07-12T15:34:08Z

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyAnalyzer.java

+        return this.bomEligibleDependencies;
+    }
+
+    public Boolean validate() {


Why does this return Boolean when the method call value it is returning is boolean?

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyComparator.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/Main.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/Utils.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/models/BomDependencyNoVersion.java

eng/bomgenerator/samplereport.txt

eng/bomgenerator/buildAndRun.cmd

eng/bomgenerator/pom.xml

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyComparator.java

Incorporate all PR comments

check-enforcer · 2021-07-22T21:05:03Z

This pull request is protected by Check Enforcer.

What is Check Enforcer?

Check Enforcer helps ensure all pull requests are covered by at least one check-run (typically an Azure Pipeline). When all check-runs associated with this pull request pass then Check Enforcer itself will pass.

Why am I getting this message?

You are getting this message because Check Enforcer did not detect any check-runs being associated with this pull request within five minutes. This may indicate that your pull request is not covered by any pipelines and so Check Enforcer is correctly blocking the pull request being merged.

What should I do now?

If the check-enforcer check-run is not passing and all other check-runs associated with this PR are passing (excluding license-cla) then you could try telling Check Enforcer to evaluate your pull request again. You can do this by adding a comment to this pull request as follows:
/check-enforcer evaluate
Typically evaulation only takes a few seconds. If you know that your pull request is not covered by a pipeline and this is expected you can override Check Enforcer using the following command:
/check-enforcer override
Note that using the override command triggers alerts so that follow-up investigations can occur (PRs still need to be approved as normal).

What if I am onboarding a new service?

Often, new services do not have validation pipelines associated with them, in order to bootstrap pipelines for a new service, you can issue the following command as a pull request comment:
/azp run prepare-pipelines
This will run a pipeline that analyzes the source tree and creates the pipelines necessary to build and validate your pull request. Once the pipeline has been created you can trigger the pipeline using the following comment:
/azp run java - [service] - ci

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/BomGenerator.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/Utils.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/Main.java

eng/bomgenerator/src/main/java/com/azure/tools/bomgenerator/DependencyAnalyzer.java

pallavit · 2021-07-27T16:38:13Z

Thank you for the feedback,

pallavit added the azure-sdk-bom Azure Java SDK BOM (Bill of Materials) label Jun 21, 2021

pallavit self-assigned this Jun 21, 2021

ghost added the Azure.Core azure-core label Jun 21, 2021

pallavit force-pushed the pallavit/bomtooling branch from ba0f718 to a0e66c2 Compare July 7, 2021 17:31

pallavit marked this pull request as ready for review July 8, 2021 16:38

pallavit requested review from alzimmermsft, anuchandy, benbp, danieljurek, JimSuplizio, JonathanGiles, srnagar and weshaggard as code owners July 8, 2021 16:38

pallavit force-pushed the pallavit/bomtooling branch from b72d5ad to adfd432 Compare July 8, 2021 16:45

weshaggard reviewed Jul 8, 2021

View reviewed changes

srnagar reviewed Jul 10, 2021

View reviewed changes

alzimmermsft reviewed Jul 12, 2021

View reviewed changes

pallavit force-pushed the pallavit/bomtooling branch from 00e822b to 07b0a5e Compare July 13, 2021 18:41

pallavit commented Jul 13, 2021

View reviewed changes

eng/bomgenerator/samplereport.txt Show resolved Hide resolved

pallavit commented Jul 13, 2021

View reviewed changes

eng/bomgenerator/samplereport.txt Show resolved Hide resolved

alzimmermsft reviewed Jul 16, 2021

View reviewed changes

Add the Bom tooling support

adabb89

Incorporate all PR comments

pallavit force-pushed the pallavit/bomtooling branch from 07b0a5e to adabb89 Compare July 22, 2021 20:54

alzimmermsft reviewed Jul 23, 2021

View reviewed changes

Incorporate CR comments

0e410a7

alzimmermsft approved these changes Jul 23, 2021

View reviewed changes

pallavit merged commit bfba5b1 into Azure:main Jul 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tool to generate BOM file #22417

Tool to generate BOM file #22417

pallavit commented Jun 21, 2021 •

edited

Loading

pallavit commented Jul 7, 2021 •

edited

Loading

pallavit commented Jul 7, 2021

weshaggard Jul 8, 2021 •

edited by pallavit

Loading

pallavit Jul 8, 2021

pallavit Jul 8, 2021 •

edited

Loading

weshaggard Jul 8, 2021

pallavit Jul 8, 2021

pallavit Jul 8, 2021

weshaggard Jul 8, 2021

pallavit Jul 12, 2021

srnagar Jul 9, 2021 •

edited by pallavit

Loading

srnagar Jul 10, 2021 •

edited by pallavit

Loading

pallavit Jul 12, 2021

alzimmermsft Jul 12, 2021

check-enforcer bot commented Jul 22, 2021

pallavit commented Jul 27, 2021

		}


		private void makeDependencyInEligible(BomDependency dependency) {

Tool to generate BOM file #22417

Tool to generate BOM file #22417

Conversation

pallavit commented Jun 21, 2021 • edited Loading

pallavit commented Jul 7, 2021 • edited Loading

pallavit commented Jul 7, 2021

weshaggard Jul 8, 2021 • edited by pallavit Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallavit Jul 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srnagar Jul 9, 2021 • edited by pallavit Loading

Choose a reason for hiding this comment

srnagar Jul 10, 2021 • edited by pallavit Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

check-enforcer bot commented Jul 22, 2021

What is Check Enforcer?

Why am I getting this message?

What should I do now?

What if I am onboarding a new service?

pallavit commented Jul 27, 2021

pallavit commented Jun 21, 2021 •

edited

Loading

pallavit commented Jul 7, 2021 •

edited

Loading

weshaggard Jul 8, 2021 •

edited by pallavit

Loading

pallavit Jul 8, 2021 •

edited

Loading

srnagar Jul 9, 2021 •

edited by pallavit

Loading

srnagar Jul 10, 2021 •

edited by pallavit

Loading