Add explicit DuplicatesStrategy as required by gradle 7+ (#470) #487

tom-haines · 2021-04-13T02:00:30Z

See gradle issue Make it an error to not handle duplicates explicitly: gradle/gradle#15759

Previous Gradle releases used a default 'include' strategy for
copy specs: if a duplicate was found, then Gradle would silently
include the duplicate, potentially overwriting the previous entry.

In Gradle 5, we introduced a warning whenever duplicates are found
and that no explicit deduplication strategy is set. 
** With Gradle 7, this now becomes an error. **

(emphasis added)

In ProtobufExtract.groovy, I've added DuplicatesStrategy.INCLUDE explicitly (which was the previous behaviour of protobuf plugin).

If you run the test (added to travis) without this change:

/gradlew test --tests com.google.protobuf.gradle.ProtobufKotlinDslGradle7PluginTest --stacktrace --info

you can replicate the failure that occurs under gradle 7.

With the change, the gradle 7 issue is remedied. This has no change to existing functionality of the plugin: it just makes it compatible with gradle 7 by declaring the behaviour explicitly.

voidzcy

The change LGTM, although we may want to reuse existing tests to cover this for Gradle 7.

tom-haines · 2021-04-16T04:28:49Z

src/test/groovy/com/google/protobuf/gradle/ProtobufKotlinDslCopySpecTest.groovy

+ */
+@CompileDynamic
+class ProtobufKotlinDslCopySpecTest extends Specification {
+  private static final List<String> GRADLE_VERSIONS = ["5.6", "6.0", "6.7.1", "7.0"]


Added tests against full gradle suite

tom-haines · 2021-04-16T04:30:37Z

testProjectBase/build_base.gradle

@@ -29,10 +29,10 @@ dependencies {
  protobuf files("ext/")
  testProtobuf files("lib/protos-test.tar.gz")

-  compile protobufDep
-  testCompile 'junit:junit:4.12'
+  implementation protobufDep


This change allows ProtobufJavaPluginTest suite to run under gradle 7, but issues relating to copy dirs are causing gradle 7 :jar task to fail (failure is in test itself, not connected to plugin). It requires some refactoring to enable 7.0 testing for ProtobufJavaPluginTest

But ProtobufJavaPluginTest doesn't include Gradle 7 now, and ProtobufKotlinDslCopySpecTest is independent. So this change doesn't seem to be necessary now, does it?

It is not necessary, no. It just means when someone does look at gradle 7, the tests compile, so thought it was better to make the fix preemptively as it's backwards compatible to gradle 5.6.
Should I revert it?

tom-haines · 2021-04-16T04:36:56Z

@voidzcy I've rebased the change with a much lighter test project that tests against gradle 5.6 - 7.0.
I had a look at updating testProjectKotlinDslBase but there are test suite issues connected to the jar task, which looks like it might require some refactoring in order to prevent the merge/fold of the projects from having duplicates (which jar task complains about).

Anyway, please see updated commit, and let me know re any other improvements.

voidzcy · 2021-04-16T09:46:09Z

testProjectKotlinDslCopySpec/build.gradle.kts

+plugins {
+  java
+  id("java-library")
+  id("com.google.protobuf").version("0.8.15")


This doesn't seem right. You shouldn't specify the 0.8.15 version. Otherwise, it's not actually testing the plugin built from the project (tests still pass, which means the change isn't really get covered. Or the Gradle TestKit just ignores the version while still injecting the plugin built from code to the classpath of the test build? I am not sure, try delete this version setting and confirm you are really testing the code instead of the plugin pulled from Maven).

Thanks for spotting. This is an interesting one, I will investigate it.

I have pushed a sanity check to confirm that the test fails under 0.8.15 - pushed here:
https://github.com/tom-haines/protobuf-gradle-plugin/tree/example-project/prove_test_fail

voidzcy · 2021-04-16T09:47:09Z

testProjectKotlinDslCopySpec/src/main/proto/google/type/money.proto

+  // If `units` is negative, `nanos` must be negative or zero.
+  // For example $-1.75 is represented as `units`=-1 and `nanos`=-750,000,000.
+  int32 nanos = 3;
+}


nit: leave an empty line at the end of file.

voidzcy · 2021-04-16T09:51:30Z

testProjectKotlinDslCopySpec/settings.gradle.kts

@@ -0,0 +1,8 @@
+pluginManagement {
+
+    repositories {


This can be put directly into the repositories{} closure in the buildscript.

voidzcy · 2021-04-16T09:56:31Z

testProjectBase/build_base.gradle

@@ -29,10 +29,10 @@ dependencies {
  protobuf files("ext/")
  testProtobuf files("lib/protos-test.tar.gz")

-  compile protobufDep
-  testCompile 'junit:junit:4.12'
+  implementation protobufDep


But ProtobufJavaPluginTest doesn't include Gradle 7 now, and ProtobufKotlinDslCopySpecTest is independent. So this change doesn't seem to be necessary now, does it?

…ncies to build.gradle.kts (google#470)

tom-haines · 2021-04-16T11:00:01Z

@voidzcy
To confirm test is effective, see failed travis build 644 re above 78c6183.

Then see travis build 645, which should succeed via commit above cf0419b re-applying fix.

You were right about the classpath / test kit ignoring the version definition. Removed version number in any event.

All identified should be remedied, let me know if anything further.

voidzcy

LGTM

voidzcy · 2021-04-16T17:42:47Z

Thanks for your contribution!

ejona86 · 2021-06-03T19:28:57Z

I think this was a mistake and should be reverted. Gradle is changing its behavior because if you have duplicate files they may not be identical and then you just get "random" results; you won't actually know what file you are using. Duplicate files indicate there is a problem that needs to be addressed.

I investigated an instance of this in gRPC, and after some poking, it looks like this was happening due to a bug in the plugin in my case. It seems we'll need to fix the bug in the plugin before reverting this. If a user encounters this issue after we fix the bug, then that means their dependencies are accidentally duplicating files and should be addressed.

I'll reopen #470 and document the bug I discovered.

tom-haines mentioned this pull request Apr 13, 2021

Duplicated .protos in multi-project build #470

Closed

voidzcy approved these changes Apr 15, 2021

View reviewed changes

Add default DuplicatesStrategy as required by gradle 7+ (google#470)

de7190b

tom-haines force-pushed the 470-duplicate-strategy-grd7 branch from f08fcba to de7190b Compare April 16, 2021 04:27

tom-haines commented Apr 16, 2021

View reviewed changes

Include CopySpecTest in travis build (google#470)

b58604f

voidzcy reviewed Apr 16, 2021

View reviewed changes

tom-haines added 5 commits April 16, 2021 18:29

Revert syntax refresh of build_base; remove version; and move depende…

0f6cb57

…ncies to build.gradle.kts (google#470)

Allow travis to confirm test failure (google#470)

9ed07a1

Apply fix to confirm travis build succeeds (google#470)

ac2ae3e

Use conflicting path to build-in proto3 type

78c6183

Address nit and apply fix to confirm travis success

cf0419b

voidzcy approved these changes Apr 16, 2021

View reviewed changes

voidzcy merged commit 76a1922 into google:master Apr 16, 2021

tom-haines deleted the 470-duplicate-strategy-grd7 branch April 22, 2021 17:45

voidzcy mentioned this pull request May 5, 2021

Not compatible with Gradle 7.0 #493

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add explicit DuplicatesStrategy as required by gradle 7+ (#470) #487

Add explicit DuplicatesStrategy as required by gradle 7+ (#470) #487

tom-haines commented Apr 13, 2021

voidzcy left a comment

tom-haines Apr 16, 2021

tom-haines Apr 16, 2021

voidzcy Apr 16, 2021

tom-haines Apr 16, 2021

tom-haines commented Apr 16, 2021

voidzcy Apr 16, 2021 •

edited

Loading

tom-haines Apr 16, 2021

tom-haines Apr 16, 2021

voidzcy Apr 16, 2021

voidzcy Apr 16, 2021

tom-haines Apr 16, 2021

voidzcy Apr 16, 2021

tom-haines commented Apr 16, 2021

voidzcy left a comment

voidzcy commented Apr 16, 2021

ejona86 commented Jun 3, 2021

Add explicit DuplicatesStrategy as required by gradle 7+ (#470) #487

Add explicit DuplicatesStrategy as required by gradle 7+ (#470) #487

Conversation

tom-haines commented Apr 13, 2021

voidzcy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tom-haines commented Apr 16, 2021

voidzcy Apr 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tom-haines commented Apr 16, 2021

voidzcy left a comment

Choose a reason for hiding this comment

voidzcy commented Apr 16, 2021

ejona86 commented Jun 3, 2021

voidzcy Apr 16, 2021 •

edited

Loading