Provide testing helper to verify actual incremental processor type

[JakeWharton]

I'm not sure the best way to do this. If the processor is using @SupportedAnnotationTypes you could validate in your processor. Otherwise, there are a lot of options. Error-prone check, but that only covers Java and not Kotlin. Some kind of test artifact that checks the values returned by getSupportedAnnotationTypes()?

[tbroyer]

I think the best way is to actually run the processor in Gradle (see the tests in this project; and you can use the Gradle Tooling API to do that in non-Gradle projects, as the Gradle TestKit is not available as a library on its own).

The problem with doing that in an annotation processor, or Error-prone check, is that this will only work if the processor being compiled uses @SupportedAnnotationTypes rather than overriding getSupportedAnnotationTypes() and the annotation is available in the classpath (or possibly the processor classpath, as a fallback), but if the annotation is on the classpath, chances are that the processor actually overrides getSupportedAnnotationTypes() to use the annotation's .class.getCanonicalName().

Things could be done in a unit-test, but because getSupportedAnnotationTypes() could, at least theoretically, depend on the ProcessingEnvironment, this needs to be done with an actual CompilationTask; and this would still not check that the processor will actually run incrementally.

There could be some kind of test artifact to help run Gradle projects and check for incrementality, but there'd still be some work needed in each project to "expose" their artifacts to the Gradle builds… For Maven, maybe a "gradle-invoker-maven-plugin" à la "maven-invoker-plugin" would be better…

[tbroyer]

Looking at the Tooling API more closely, I see it provides even more information about annotation processors than TestKit (see https://docs.gradle.org/5.6.4/javadoc/org/gradle/tooling/events/task/java/JavaCompileTaskOperationResult.html), so I think it deserves a test artifact! Not sure when I'll have time to work on this though…

[tbroyer]

Fwiw, former Gradle employees (now running Nokee.dev) are redistributing Gradle API and Gradle TestKit JARs to the Central Repository. They can be used from non-Gradle builds to drive Gradle builds and check the build logs for whether the processor under test is actually incremental.
AutoValue now uses this to prevent regressions: google/auto@c8a5c19