Zinc invalidates all java sources in mixed projects with any scala source change

[eatkins]

steps

• publish zinc locally using the latest HEAD of develop
• publish sbt with the zinc version set to 1.4.0-SNAPSHOT
• run sbt -Dsbt.version=1.4.0-SNAPSHOT in the sbt repo
• create a new file main/src/main/scala/Foo.scala

object Foo {
  val x = 1
}

• run compile in the shell
• change val x = 1 to val x = 2 in Foo.scala
• run compile again

problem

zinc invalidates all 14 java sources
This not-only recompiles all the java classes but also triggers a second cycle which compiles more scala files:

sbt:sbtRoot> compile
[info] Formatting 1 source in mainProj / Compile using scalafmt
[info] Compiling 1 Scala source and 14 Java sources to /Users/ethanatkins/work/oss/sbt/sbt/main/target/scala-2.12/classes ...
[info] done compiling
[info] Compiling 3 Scala sources and 14 Java sources to /Users/ethanatkins/work/oss/sbt/sbt/main/target/scala-2.12/classes ...
[info] done compiling

expectation

Java sources are only invalidated when their source or binary dependencies change.

notes

[debug] > initialChanges = InitialChanges(Changes(added = Set(), removed = Set(), changed = Set(${BASE}/main/src/main/scala/Foo.scala), unmodified = ...),Set(),Set(),API Changes: Set())                                                         
[debug]                                                                                                                                
[debug] Initial source changes:                                                                                                        
[debug]         removed:Set()                                                                                                          
[debug]         added: Set()                                                                                                           
[debug]         modified: Set(${BASE}/main/src/main/scala/Foo.scala)                                                                   
[debug] Invalidated products: Set()                                                                                                    
[debug] External API changes: API Changes: Set()                                                                                                                                                                                                                                
[debug] Modified binary dependencies: Set()                                                                                            
[debug] Initial directly invalidated classes: Set(Foo)                                                                                 
[debug]                                                                                                                                                                                                                                                                         
[debug] Sources indirectly invalidated by:                                                                                             
[debug]         product: Set()                                                                                                                                                                                                                                                  
[debug]         binary dep: Set()                                                                                                      
[debug]         external source: Set()                                                                                                                                                                                                                                          
[debug] All initially invalidated classes: Set(Foo)                                                                                                                                                                                                                             
[debug] All initially invalidated sources:Set(${BASE}/main/src/main/scala/Foo.scala, ${BASE}/main/src/main/java/sbt/internal/ScalaReflectClassLoader.java, ${BASE}/main/src/main/java/sbt/internal/ClassLoadingLock.java, ${BASE}/main/src/main/java/sbt/internal/FullScalaLoade
r.java, ${BASE}/main/src/main/java/sbt/internal/JLineLoader.java, ${BASE}/main/src/main/java/sbt/internal/BottomClassLoader.java, ${BASE}/main/src/main/java/sbt/internal/TestInterfaceLoader.java, ${BASE}/main/src/main/java/sbt/internal/ReverseLookupClassLoader.java, ${BAS
E}/main/src/main/java/sbt/internal/XMainClassLoader.java, ${BASE}/main/src/main/java/sbt/internal/FlatLoader.java, ${BASE}/main/src/main/java/sbt/internal/ClassLoaderClose.java, ${BASE}/main/src/main/java/sbt/internal/ManagedClassLoader.java, ${BASE}/main/src/main/java/sb
t/internal/ScalaLibraryClassLoader.java, ${BASE}/main/src/main/java/sbt/internal/LayeredClassLoader.java, ${BASE}/main/src/main/java/sbt/internal/MetaBuildLoader.java)                                                                                                         
[debug] Initial set of included nodes: Foo                                                                                             
[debug] compilation cycle 1                                                                                                                                                                                                                                                     
[info] Compiling 1 Scala source and 14 Java sources to /Users/ethanatkins/work/oss/sbt/sbt/main/target/scala-2.12/classes ...  

[eed3si9n]

That's currently a known/intentional behavior for pipelining. Is pipelining turned on?

[eatkins]

I don't know how to turn pipelining on so I assume it has always been off in the projects I test. At any rate, #868 fixes the issue I described above.

[eatkins]

Out of curiosity, when will compilation run with pickle java on? The behavior of always invalidating and recompiling all java sources has the potential to completely break incremental compilation in some projects which would offset much or even all of the performance gains of pipelining.

[eed3si9n]

Pipelining was on by default for a short while until I turned it off in #843.

Out of curiosity, when will compilation run with pickle java on? The behavior of always invalidating and recompiling all java sources has the potential to completely break incremental compilation in some projects which would offset much or even all of the performance gains of pipelining.

Java pickling would be on when -Ypickle-java is in scalacOptions, which sbt would do when you have pipelining on (after sbt/sbt#5703).

It's not a perfect situation. Java compilation tends to be relatively fast, and hopefully name hashing (includes API change comparison) would catch the invalidation from spreading. Further optimization is left to Lightbend as an exercise.

[eatkins]

Got it. My experience seems to be that name hashing does not prevent the invalidation from spreading though. In the example I described above in the sbt project, it did two cycles even though only Foo.scala actually changed and no other file depended on Foo.scala.