Android: Analyzer fails because Gradle cannot choose between multiple project variants

[MarcelBochtler]

When analysing a fairly large Android project, we often see issues with Gradle's dependency resolutions.

To reproduce:
Clone Owncloud's Android App: https://github.com/owncloud/android

Run the Analyzer: ort --info --stacktrace analyze -i android/ -o android/ort/analyzer

This causes the Analyzer to produce the following error:

14:10:57.164 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.analyzer.managers.GradleDependencyHandler - Unresolved: ModuleVersionResolveException: Could not resolve project :owncloudDomain.
Caused by: AmbiguousConfigurationSelectionException: The consumer was configured to find a usage of 'kotlin-api' of a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'common'. However we cannot choose between the following variants of project :owncloudDomain:
  - debugApiElements
  - debugRuntimeElements
  - ktlint
  - releaseApiElements
  - releaseRuntimeElements
All of them match the consumer attributes:
  - Variant 'debugApiElements' capability owncloud-android:owncloudDomain:unspecified declares an API of a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm':
      - Unmatched attributes:
          - Provides attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'debug' but the consumer didn't ask for it
          - Provides attribute 'com.android.build.api.attributes.VariantAttr' with value 'debug' but the consumer didn't ask for it
  - Variant 'debugRuntimeElements' capability owncloud-android:owncloudDomain:unspecified declares a runtime of a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm':
      - Unmatched attributes:
          - Provides attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'debug' but the consumer didn't ask for it
          - Provides attribute 'com.android.build.api.attributes.VariantAttr' with value 'debug' but the consumer didn't ask for it
  - Variant 'ktlint' capability owncloud-android:owncloudDomain:unspecified:
      - Unmatched attributes:
          - Provides its dependencies declared externally but the consumer didn't ask for it
          - Doesn't say anything about its usage (required a usage of 'kotlin-api')
          - Doesn't say anything about org.jetbrains.kotlin.platform.type (required 'common')
  - Variant 'releaseApiElements' capability owncloud-android:owncloudDomain:unspecified declares an API of a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm':
      - Unmatched attributes:
          - Provides attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'release' but the consumer didn't ask for it
          - Provides attribute 'com.android.build.api.attributes.VariantAttr' with value 'release' but the consumer didn't ask for it
  - Variant 'releaseRuntimeElements' capability owncloud-android:owncloudDomain:unspecified declares a runtime of a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm':
      - Unmatched attributes:
          - Provides attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'release' but the consumer didn't ask for it
          - Provides attribute 'com.android.build.api.attributes.VariantAttr' with value 'release' but the consumer didn't ask for it

[MarcelBochtler]

A possible way to fix this would be to offer an optional Analyzer option to select the variant which should be analysed.

[sschuberth]

A possible way to fix this would be to offer an optional Analyzer option to select the variant which should be analysed.

A similar (package-manager-specific) option (in .ort.yml) could then also be used to address #1774 by specifying the Maven profile to use.

[tsteenbe]

@sschuberth @MarcelBochtler Did you just enabled the Analyzer to use labels? Could we instead use a specific label keys to specify gradle variant/maven profile to analyze?

[sschuberth]

Whatever mechanism we use, IMO it needs to be one that is configurable along with the project's source code. We could also allow to set labels in .ort.yml actually, so things like the "application type" (which AFAIK HERE tracks as labels) could also be configured in the project itself, rather than a belonging Jira ticket or Confluence page.

[sschuberth]

I'll edit this comment to collect a list of other Gradle dependency analysis tools, to look at how they're avoiding the AmbiguousConfigurationSelectionException and AmbiguousVariantSelectionException (ticked ones have been looked at):

• https://docs.gradle.org/current/userguide/project_report_plugin.html#project_report_plugin (use the JSON embedded into the htmlDependencyReport)
• https://github.com/gradle/github-dependency-extractor
• https://github.com/sonatype-nexus-community/scan-gradle-plugin
  • This change seems to solve the same problem that we have. Also see this comment.
• https://github.com/CycloneDX/cyclonedx-gradle-plugin
• https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin
• https://github.com/kronicle-tech/dependencies-file-gradle-plugin
• https://github.com/cashapp/licensee
• https://github.com/uklance/gradle-dependency-export

Additional information:

• Understanding variant selection in Gradle
• There's also an upstream Gradle issue about this.
• Kotlin Usages of Gradle Variant-Aware Dependency Management
• Accessing the resolution result programmatically

[sschuberth]

I just confirmed that the GradleInspector can successfully analyze https://github.com/owncloud/android, see the attached analyzer result. So this specific issue can be closed.

[brueggenthies-ams]

@sschuberth after switching to the new Gradle-Inspector i could also successfully analyze my android project. However i struggle to scan through the project, because the sources for all androidx.xyz libraries from the Android Jetpack Project cannot be resolved. This is probably due to bad metadata in the maven artifact.

Because the AndroidX Libraries are developed inside a big mono repo, you cannot match a specific library version against a revision inside the repository. The sources are provided as jars for each library version.

Is it correct to add these information via the curations.yml file, to prevent failing source downloads?

[sschuberth]

However i struggle to scan through the project, because the sources for all androidx.xyz libraries from the Android Jetpack Project cannot be resolved.

Also see #5105 in that context.

This is probably due to bad metadata in the maven artifact.

Correct.

Because the AndroidX Libraries are developed inside a big mono repo, you cannot match a specific library version against a revision inside the repository

That's not completely true, as you could leverage the VcsInfo's path property to point at the subdirectory in a monorepo containing the source code of that library:

ort/model/src/main/kotlin/VcsInfo.kt

Lines 43 to 47 in 15e33b2

	/**
	* The path inside the VCS to take into account. If the VCS supports checking out only a subdirectory, only this
	* path is checked out.
	*/
	val path: String = ""

Is it correct to add these information via the curations.yml file, to prevent failing source downloads?

Yes, I'd probably curate author and concluded license, and enable the skipConcluded scanner option as suggested here.

[ksg97031]

I'll edit this comment to collect a list of other Gradle dependency analysis tools, to look at how they're avoiding the AmbiguousConfigurationSelectionException and AmbiguousVariantSelectionException (ticked ones have been looked at):

• https://docs.gradle.org/current/userguide/project_report_plugin.html#project_report_plugin (use the JSON embedded into the htmlDependencyReport)

• https://github.com/gradle/github-dependency-extractor

• https://github.com/sonatype-nexus-community/scan-gradle-plugin

  • This change seems to solve the same problem that we have. Also see this comment.

• https://github.com/CycloneDX/cyclonedx-gradle-plugin

• https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin

• https://github.com/kronicle-tech/dependencies-file-gradle-plugin

• https://github.com/cashapp/licensee

• https://github.com/uklance/gradle-dependency-export

Additional information:

• Understanding variant selection in Gradle
• There's also an upstream Gradle issue about this.
• Kotlin Usages of Gradle Variant-Aware Dependency Management
• Accessing the resolution result programmatically

Hello sschuberth, I'm looking for a way to choose build variants in the analyzer to speed up processing. Currently, we can't select build variants, so are you suggesting we use gradle-inspector instead?

[sschuberth]

Currently, we can't select build variants, so are you suggesting we use gradle-inspector instead?

Using gradle-inspector will also not allow you to limit analysis to specified build variants for performance reasons; that would go against one of the fundamental principles of ORT to not "hide" information from the user. Instead, we always analyze everything, and later on allow to filter down information (e.g. via scope excludes and other means) to what is relevant in the respective context.

However, if performance is such a big issue in your case, one could think about a feature that indeed really omits build variants (or scopes, as a more general term) from analysis, but clearly documents as part of the analyzer result what was not analyzed. If you're interested in such a feature, please file a new issue.