Optimize import/export resolution (#5700)

This change adds serialization and deserialization of library bindings.
In order to be functional, one needs to first generate IR and
serialize bindings using `--compiled <path-to-library>` command. The bindings
will be stored under the library with `.bindings` suffix.
Bindings are being generated during `buildEngineDistribution` task, thus not
requiring any extra steps.

When resolving import/exports the compiler will first try to load
module's bindings from cache. If successful, it will not schedule its
imports/exports for immediate compilation, as we always did, but use the
bindings info to infer the dependent modules.

The current change does not make any optimizations when it comes to
compiling the modules, yet. It only delays the actual
compilation/loading IR from cache so that it can be done in bulk.
Further optimizations will come from this opportunity such as parallel
loading of caches or lazily inferring only the necessary modules.

Part of #5568 work.

CHANGELOG.md

@@ -597,6 +597,7 @@
 - [Engine can now recover from serialization failures][5591]
 - [Use sbt runEngineDistribution][5609]
 - [Update to GraalVM 22.3.1][5602]
+- [Cache library bindings to optimize import/export resolution][5700]
 
 [3227]: https://github.com/enso-org/enso/pull/3227
 [3248]: https://github.com/enso-org/enso/pull/3248
@@ -693,6 +694,7 @@
 [5591]: https://github.com/enso-org/enso/pull/5591
 [5609]: https://github.com/enso-org/enso/pull/5609
 [5602]: https://github.com/enso-org/enso/pull/5602
+[5700]: https://github.com/enso-org/enso/pull/5700
 
 # Enso 2.0.0-alpha.18 (2021-10-12)
 

build.sbt

@@ -1512,7 +1512,6 @@ lazy val runtime = (project in file("engine/runtime"))
   .dependsOn(graph)
   .dependsOn(pkg)
   .dependsOn(`edition-updater`)
-  .dependsOn(`library-manager`)
   .dependsOn(`connected-lock-manager`)
   .dependsOn(syntax.jvm)
   .dependsOn(`syntax-rust-definition`)

docs/runtime/ir-caching.md

@@ -207,7 +207,7 @@ other than the fact that they may be seeing the actual files on disk.
 Integrity Checking does not check the situation when the cached module imports a
 module which cache has been invalidated. For example, module `A` uses a method
 `foo` from module `B` and a successful compilation resulted in IR cache for both
-`A` and `B`. Later, someone modifed module `B` by renaming method `foo` to
+`A` and `B`. Later, someone modified module `B` by renaming method `foo` to
 `bar`. If we only compared source hashes, `B`'s IR would be re-generated while
 `A`'s would be loaded from cache, thus failing to notice method name change,
 until a complete cache invalidation was forced.
@@ -228,6 +228,21 @@ There are two main elements that need to be tested as part of this feature.
   runtime options for debugging, but also constructing the `DistributionManager`
   on context creation (removing `RuntimeDistributionManager`).
 
+### Import/Export caching of bindings
+
+Import and export resolution is one of the more expensive elements in the
+initial pipeline. It is also the element which does not change for the releases
+library components as we do not expect users to modify them. During the initial
+compilation stage we iteratively parse/load cached ir, do import resolution on
+the module, followed by export resolution, and repeat the process with any
+dependent modules discovered in the process. Calculating such transitive closure
+is an expensive and repeatable process. By caching bindings per library we are
+able to skip that process completely and discover all necessary modules of the
+library in a single pass.
+
+The bindings are serialized along with the library caches in a file with a
+`.bindings` suffix.
+
 ## Future Directions
 
 Due to the less than ideal platform situation we're in, we're limited to using

engine/language-server/src/main/scala/org/enso/languageserver/libraries/CompilerBasedDependencyExtractor.scala

@@ -37,7 +37,7 @@ class CompilerBasedDependencyExtractor(logLevel: LogLevel)
       importedLibraries.toSet
     }
 
-    val sourcesImports = pkg.listSources.toSet.flatMap(findImportedLibraries)
+    val sourcesImports = pkg.listSources().toSet.flatMap(findImportedLibraries)
     val itself         = pkg.libraryName
 
     // Builtins need to be removed from the set of the dependencies, because

engine/language-server/src/main/scala/org/enso/languageserver/libraries/handler/LibraryPreinstallHandler.scala

@@ -223,7 +223,7 @@ class LibraryPreinstallHandler(
         publishedLibraryProvider = config.publishedLibraryCache,
         edition                  = edition,
         preferLocalLibraries     = preferLocalLibraries,
-        versionResolver          = LibraryResolver(config.localLibraryProvider),
+        libraryResolver          = LibraryResolver(config.localLibraryProvider),
         dependencyExtractor      = config.installerConfig.dependencyExtractor
       )
     } yield Tools(installer, dependencyResolver)

engine/language-server/src/test/scala/org/enso/languageserver/websocket/json/LibrariesTest.scala

@@ -609,9 +609,11 @@ class LibrariesTest extends BaseServerTest {
             .loadPackage(cachedLibraryRoot.location.toFile)
             .get
         pkg.name shouldEqual "Bar"
-        pkg.listSources.map(
-          _.file.getName
-        ) should contain theSameElementsAs Seq("Main.enso")
+        pkg
+          .listSources()
+          .map(
+            _.file.getName
+          ) should contain theSameElementsAs Seq("Main.enso")
 
         assert(
           Files.exists(cachedLibraryRoot / LibraryManifest.filename),