Transitive dependencies creating version conflicts & missing jar verification

[ghubstan]

Description

The gradle build needs to be more strict about excluding transitive dependencies that create version conflicts. (This implies some dependencies might need to be upgraded, and some even downgraded, to align versions of direct and transitive dependencies.)

Version

Commit 17bb7b4

Steps to reproduce

Build Bisq and see multiple versions of some jars in the lib dir.

Expected behaviour

There should not be any version conflicts, and every jar needs to be verified by the gradle-witness.gradle file.

Actual behaviour

There are several transitive dependencies downloaded during the build that conflict with direct dependencies specified in gradle.build, and in the case of the desktop build, they are not being verified by gradle-witness.

Here are duplicate jars with the letters 'grpc' in their name:

grpc-context-1.10.1.jar
grpc-context-1.25.0.jar
grpc-core-1.10.1.jar
grpc-core-1.25.0.jar
grpc-netty-shaded-1.10.1.jar
grpc-netty-shaded-1.25.0.jar
grpc-protobuf-1.10.1.jar
grpc-protobuf-1.25.0.jar
grpc-protobuf-lite-1.10.1.jar
grpc-protobuf-lite-1.25.0.jar
grpc-stub-1.10.1.jar
grpc-stub-1.25.0.jar
opencensus-contrib-grpc-metrics-0.11.0.jar
opencensus-contrib-grpc-metrics-0.21.0.jar

This may not be an urgent problem because the latest jar is loaded at run time. However, that also means that a transitive dependency with a larger version than is specified in the build file is loaded, not the jar which has passed gradle witness verification. For example: the build file specifies

slf4jVersion = '1.7.22'

but

lsof -Pan -p $(pgrep -f BisqAppMain)        ( on linux )

shows me that the transitive dependency slf4j-api-1.7.25.jar is being loaded instead.

Maybe building the shipped, shaded jar eliminates most of this problem, but I new devs like me may spend a lot of time trying to figure out why jar version conflicts like the grpc example I described above exist.

[ghubstan]

I don't think it will be possible to resolve all version conflicts because too many transitive dependencies have too many of their own version conflict problems in their own dependency graphs. But it would be good to minimize them in Bisq as much as practicable.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cd2357]

@ghubstan is this still an issue?

[ghubstan]

Yes, it is. I'm working on an idea to improve this situation in a misq poc.

[cd2357]

Same here, was also looking for alternatives.

Some of the issues with gradle-witness I found:

• it doesn't check transitive dependencies (Transient dependencies are not verified. signalapp/gradle-witness#33)
• it doesn't verify plugins, only dependencies (Remove jcenter repository #5497 (comment))
• your issue described above
• far fewer hashes in gradle-witness.gradle than actual dependencies used in the build

Seems to give a false sense of security

Related older thread: #1897 (comment)

Main solutions I came across so far:

• https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin
  • From their README: Checksum Dependency Plugin is probably the first plugin that is able to verify Gradle Plugins and
    that is able to use PGP for trust-based verification.
• Verify hashes and PGP signatures for dependencies and plugins gradle/gradle#10443

cc @bisq-network/bisq-devs : If you have other ideas, add as comments below.

[ghubstan]

The gradle-witness plugin is becoming obsolete (and possibly) incompatible with newer versions of gradle. There is a built in verification task in newer gradle versions: https://docs.gradle.org/6.6.1/userguide/dependency_verification.html
It will generate an xml file with checksums for almost all dependencies in a big project; you can verify those generated checksums and add missing checksums.

I don't know if it can verify plugin jars.

[cd2357]

@ghubstan that's probably the best way forward.

Looks like it can even do plugins:

This feature can be used for:

• detecting compromised dependencies
• detecting compromised plugins
• detecting tampered dependencies in the local dependency caches

Let me know if I can help.

[ghubstan]

Let me know if I can help.

I think it just works if you add verification-metadata.xml to the project root's gradle directory.

<?xml version="1.0" encoding="UTF-8"?>
<verification-metadata>
   <configuration>
      <verify-metadata>true</verify-metadata>
      <verify-signatures>false</verify-signatures>
    </configuration>
</verification-metadata>

[cd2357]

Thanks, I added a PR for the only Misq repo I am aware of: https://github.com/chimp1984/misq/pull/8

[ghubstan]

We will always be dealing with the version conflicts, but I appreciate the issue house cleaning you're doing.