Create a custom Gradle task to list all dependencies

[jakubgs]

There is a terrible hack hidden in how we generate the nix/deps/gradle/deps.json which is consumed by the Gradle dependencies derivation which is responsible for providing Gradle with all necessary dependencies for builds. This hack is this AWK script:

status-mobile/nix/deps/gradle/gradle_parser.awk

Lines 17 to 46 in 8c358d4

	# Gradle outputs dependencies in groups defined by configurations.
	# Those configurations are words followed by a dash and a description.
	# There's also a special 'classpath' configuration we want.
	/^(classpath|[a-zA-Z0-9]+)( - .*)?$/ {
	# Ignore configurations starting with 'test'
	if (tolower($1) ~ /^test/) {
	next
	}
	# Lines after configuration name list packages
	for (getline line; line != ""; getline line) {
	# React Native is provided by node_modules
	if (line ~ "com.facebook.react:react-native") { continue }
	# Example: +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.50
	if (findPackage(line, "--- ([^ :]+):([^ :]+):([^ :]+)$")) {
	continue
	}
	# Example: +--- androidx.lifecycle:lifecycle-common:{strictly 2.0.0} -> 2.0.0 (c)
	if (findPackage(line, "--- ([^ :]+):([^ :]+):[^:]+ -> ([^ :]+) ?(\\([*c]\\))?$")) {
	continue
	}
	# Example: +--- com.android.support:appcompat-v7:28.0.0 -> androidx.appcompat:appcompat:1.0.2
	if (findPackage(line, "--- [^ :]+:[^ :]+:[^ ]+ -> ([^ :]+):([^ :]+):([^ :]+)$")) {
	continue
	}
	}
	}

Which parses the output of a Gradle call that prints tree of dependencies which looks like this:

debugCompileClasspath - Resolved configuration for compilation for variant: debug
+--- com.facebook.react:react-native:+ -> 0.63.5
|    +--- com.facebook.infer.annotation:infer-annotation:0.11.2
|    |    \--- com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2
|    +--- com.facebook.yoga:proguard-annotations:1.14.1
|    +--- javax.inject:javax.inject:1
|    +--- androidx.appcompat:appcompat:1.0.2
|    |    +--- androidx.annotation:annotation:1.0.0
|    |    +--- androidx.core:core:1.0.1
|    |    |    +--- androidx.annotation:annotation:1.0.0
|    |    |    +--- androidx.collection:collection:1.0.0
|    |    |    |    \--- androidx.annotation:annotation:1.0.0

The result of parsing this file is nix/deps/gradle/deps.list which is then turned into nix/deps/gradle/deps.urls which in turn is used to create nix/deps/gradle/deps.json.

This is not a good way of generating this list of dependencies. And is possibly the cause of some dependencies being missed in the React Native upgrade PR, specifically:

• [WIP] React native upgrade from version 0.63.3 to 0.71.4 ? #15203 (comment)
  • org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin:1.6.10
• [WIP] React native upgrade from version 0.63.3 to 0.71.4 ? #15203 (comment)
  • com.android.tools.lint:lint-gradle:30.3.0

Which is why we need a more proper solution, in the form of a custom Gradle task that can print out the list of dependencies as it is without the need for parsing disgusting trees with AWK. Such a solution would most probably be close to a task like this:

task printDependencies {
    doLast {
        def depsSet = new TreeSet<String>()

        configurations.all { config ->
            if (!config.name.toLowerCase().startsWith('test')) {
                config.allDependencies.all { dep ->
                    if (dep.group && dep.name && dep.version) {
                        depsSet.add("${dep.group}:${dep.name}:${dep.version}")
                    }
                }
            }
        }

        depsSet.each { println it }
    }
}

Which was found in some StackOverflow by @siddarthkay .

[jakubgs]

It is worth noting that Gradle often picks a newer version of a package over an older one when resolving dependencies:

+--- com.facebook.react:react-native:+ -> 0.63.5
|    +--- com.facebook.infer.annotation:infer-annotation:0.11.2
|    |    \--- com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2

In this case we see that com.google.code.findbugs:jsr305 is required at 3.0.1 but 3.0.2 will be used in its stead.
This usually means that the version definition allows this, and there are other dependencies that pull in the newer verison.

It would be good to support this, as it would allow us to keep the Gradle dependencies derivation on the smaller side.
Every dependency we pull in that is not necessary will make fetching dependencies slower, and take up more space locally.

Now, this is not absolutely necessary, if it turns out to be difficult, but it would be good to have.

[alwx]

I tried multiple different approaches, not very successfully, but here are some results:

1. The printDependencies task from above works but it shows only the direct dependencies and doesn't return any sub-dependencies (here is the class definition: https://github.com/gradle/gradle/blob/master/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/dependencies/DefaultExternalModuleDependency.java)

2. However, I found out that there is a Project Report Plugin that can generate a complete dependency report, and it can be added this way:
   

   status-mobile/android/build.gradle

   Line 41 in 33055ff

   task allDeps(type: DependencyReportTask) {}

It seems to show everything we need but in a tree-like structure which means parsing it with the same script again, and that's something we want to avoid.

We can try to utilize some code from DependencyReportTask and use it to get all the dependencies. Here is its documentation: https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/diagnostics/DependencyReportTask.html

All the current results can be found here: #15539

[alwx]

I'm currently checking if something can be extracted from DependencyReportTask or some code from it can be used separately in another task to generate a plain list of dependencies (see DepReport.groovy file in my PR)

[alwx]

Going to temporarily pause all the work to focus on more urgent UI issues for 0.23.0.

Current results:
Haven't really managed to generate a list of dependencies but pretty sure it's not hard to achieve that by utilizing the code from DependencyReportTask and rewriting it that way so that it will be generating a plain list instead of a tree. It requires some Java knowledge and I can check it out once I finish with the current issues but feel free to take this issue and work on it in the meantime.

[jakubgs]

Thanks for looking into it.

[siddarthkay]

Hey! @alwx : I'm assigning this to myself to give it a shot.

[jakubgs]

I believe the key to finding the missing dependencies are these files:

~/work/status-mobile/node_modules develop 45s
 > ag '\[libraries\]'
@react-native/gradle-plugin/gradle/libs.versions.toml
9:[libraries]

react-native/gradle/libs.versions.toml
42:[libraries]

They appear to be loaded somewhere to provide required versions of packages:

node_modules/@react-native/gradle-plugin/gradle/libs.versions.toml

[versions]
agp = "8.1.1"
gson = "2.8.9"
guava = "31.0.1-jre"
javapoet = "1.13.0"
junit = "4.13.2"
kotlin = "1.8.0"

[libraries]
kotlin-gradle-plugin = { module = "org.jetbrains.kotlin:kotlin-gradle-plugin", version.ref = "kotlin" }
android-gradle-plugin = { module = "com.android.tools.build:gradle", version.ref = "agp" }
gson = { module = "com.google.code.gson:gson", version.ref = "gson" }
guava = { module = "com.google.guava:guava", version.ref = "guava" }
javapoet = { module = "com.squareup:javapoet", version.ref = "javapoet" }
junit = {module = "junit:junit", version.ref = "junit" }

[plugins]
kotlin-jvm = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }

[jakubgs]

Apparently Gradle can load additional catalogs of dependencies:

dependencyResolutionManagement {
    versionCatalogs {
        // declares an additional catalog, named 'testLibs', from the 'test-libs.versions.toml' file
        testLibs {
            from(files('gradle/test-libs.versions.toml'))
        }
    }
}

https://docs.gradle.org/current/userguide/platforms.html#ex-declaring-additional-catalogs

But I cannot find a reference to such a line.

[jakubgs]

Something is loading the libs attribute somewhere:

  implementation(libs.kotlin.gradle.plugin)
  implementation(libs.android.gradle.plugin)

  implementation(libs.gson)
  implementation(libs.guava)
  implementation(libs.javapoet)

https://github.com/status-im/status-mobile/blob/ac186e27dec5a661e3f33561624bb4eb85b04c58/node_modules/@react-native/gradle-plugin/build.gradle.kts#L43-L48

Which is then loaded in our Gradle config:

status-mobile/android/settings.gradle

Line 21 in ac186e2

includeBuild('../node_modules/@react-native/gradle-plugin')

But why is it not evaluated together with other deps?