-
Notifications
You must be signed in to change notification settings - Fork 323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Store IR caches in suitable format to read them "on demand" #8207
Merged
+3,218
−702
Merged
Changes from all commits
Commits
Show all changes
102 commits
Select commit
Hold shift + click to select a range
61d2d01
Framework for persisting objects into ByteBuffer
JaroslavTulach a59e9b7
Turning Location into record
JaroslavTulach 999bea6
Testing simple serde of Location
JaroslavTulach f8ff83a
Enable debugging in runtime-parser tests
JaroslavTulach c845572
Test persistance of composed classes
JaroslavTulach 95298f4
Allow access to Truffle API from runtime-parser/test
JaroslavTulach 89df854
Encapsulating Persistance lookup
JaroslavTulach 236cc08
Deserialize nested objects
JaroslavTulach 04b97ef
Don't serialize UUIDs
JaroslavTulach 3fe9e33
Adjusting imports to new location of Location
JaroslavTulach 7a7f9d9
Turning IdentifiedLocation into record
JaroslavTulach 98f720b
Save some memory by using UUID directly without Some(UUID) overhead
JaroslavTulach c549943
Making the tests compiler again
JaroslavTulach 400b151
Using own Persistance Input and Output types
JaroslavTulach b583697
Measure the expected size of the serialized data
JaroslavTulach e26c83f
Storing objects inline lowers the size of the stream
JaroslavTulach d2afbab
References don't need identification of the object type
JaroslavTulach aa131e5
Support for serializing Scala List
JaroslavTulach de85e83
Keep the API stream based
JaroslavTulach 2df600b
Using Persistance to store IR
JaroslavTulach 9260903
Support null values in writeObject
JaroslavTulach f52c7e1
Can persist some ir.Module
JaroslavTulach a25bfb6
Use class loader of the language when searching for provides
JaroslavTulach ff2c9ac
Log also runtime exceptions
JaroslavTulach 704ac72
Don't swallow exceptions - log them
JaroslavTulach 7e6e9b5
Can read IR for (some) import statements
JaroslavTulach 1f1d797
Can read (some) export statements
JaroslavTulach c8c8366
Support for lazy Seq
JaroslavTulach cd12f51
Using lazy Seq to delay loading of method body
JaroslavTulach 719993a
Persistance for Name.MethodReference
JaroslavTulach 0740709
Persistance for imports.Polyglot
JaroslavTulach 9721fbb
Persistance for Function.Lambda
JaroslavTulach db4f8f2
Annotation processor for @Persistable to automate Persistance of simp…
JaroslavTulach 14c2282
Able to persist signatures
JaroslavTulach f888c4f
Able to persist IR for Standard.Visualization
JaroslavTulach 3391d6e
Persistance has writeObject and readObject static methods
JaroslavTulach bab61a1
Using Persistance to store MapToBindings
JaroslavTulach 6d15515
Library BindingsMaps get serialized without issues
JaroslavTulach b05ec17
Persistance of MetadataStorage
JaroslavTulach 4a5809a
More of persistance for metadata
JaroslavTulach 5413e2f
Persist all caches
JaroslavTulach a76a9bb
PersistanceMap is singleton
JaroslavTulach 2d1b62a
Formatting
JaroslavTulach 2d60832
Lazy BindingsMap
JaroslavTulach 5aacca7
Caching read objects
JaroslavTulach 73d56e0
Avoid empty objects by appending trailing zero
JaroslavTulach e56d043
Warn when a cache cannot be loaded
JaroslavTulach d4a1460
Manual persistance of AliasAnalysis.Graph and co.
JaroslavTulach 44d1af1
Java formatting
JaroslavTulach 0a1f325
Merge remote-tracking branch 'origin/develop' into wip/jtulach/Persis…
JaroslavTulach 3ed31ac
Making Function.Lambda body list lazy
JaroslavTulach da5ecdc
IDE complained about missing Double conversion
JaroslavTulach f2886a2
More case classes elligible for serialization
JaroslavTulach 0f77ac0
Add missing import to address #8289
JaroslavTulach 20f5ae8
Merging with CompilerContext based serialization
JaroslavTulach 4a314ef
Support for services associated with Persistance.Input
JaroslavTulach 0363603
Making runtime tests compile again and SerdeTest pass
JaroslavTulach 5e52147
Perform restoreFromSerialization on demand, not in a batch
JaroslavTulach 56663af
No need for cross module links: either the module deserializes OK or …
JaroslavTulach 78bab90
Modules loaded from cache shall have no errors
JaroslavTulach b66bfa1
Store Method.Explicit.isStatic information in caches to avoid materia…
JaroslavTulach 177c44e
Delay construction of RuntimeFunction until really needed
JaroslavTulach ef84e1d
Serializing errors.ImportExport now fails
JaroslavTulach 95ff6a0
Compare IR before and after serialization test
JaroslavTulach 5cb4b68
Make sure the richest constructor (that includes all variables) is us…
JaroslavTulach 98b95c4
Adding persistance for two more IR types
JaroslavTulach 68ed2a7
Use @Persistable annotation for more classes
JaroslavTulach 7c562b4
Turn linkage errors into AssertionError
JaroslavTulach 0f21b80
prepareForSerialization just before writing MetadataStorage down
JaroslavTulach bb1bc94
Using ir.duplicate(keepLocations=false...) to eliminate locations
JaroslavTulach b172c85
Avoid prepareForSerialization calls that modify the IR. Do the conver…
JaroslavTulach 120ac13
Make sure Scala's Seq is stored inline
JaroslavTulach cacf9f0
Merge remote-tracking branch 'origin/develop' into wip/jtulach/Persis…
JaroslavTulach 2122da8
Introducing writeReplace and readResolve functions
JaroslavTulach 497e300
Either convert AbstractModule or stop deserialization
JaroslavTulach 0fec07d
Fix exhaustiveness check
hubertp aad192b
Treat List as Seq for purposes of the test
JaroslavTulach 385b3b8
Merge branch 'wip/jtulach/Persistance_6100' of enso:enso-org/enso int…
JaroslavTulach c40c7c4
scalafmt
JaroslavTulach 79482a1
Moving Persistance classes and annotation processor into their own pr…
JaroslavTulach b7f2e09
Lookup library shall only be used at compile time
JaroslavTulach 76859b9
Splitting implementation away from the API classes
JaroslavTulach f5238b0
legal
hubertp 5e8d880
bring back more precise patterns
hubertp 05be437
Adding tests to persistance project
JaroslavTulach 2231152
Basic documentation of the org.enso.persistance package
JaroslavTulach 3586c33
More code examples
JaroslavTulach 3370745
more legal
hubertp a4048a6
Documenting the @Persistable annotation
JaroslavTulach 5300790
one more legal change
hubertp 73b63f3
Shorter package names
JaroslavTulach 2c1a2c1
Merge branch 'wip/jtulach/Persistance_6100' of enso:enso-org/enso int…
JaroslavTulach dbf62e2
Complete javadoc for all API elements
JaroslavTulach 7c9c23c
Merging with GraalVM for JDK21 effort
JaroslavTulach 54a0471
Updating the IR caches documentation
JaroslavTulach b22f5eb
Documenting PersistableProcessor
JaroslavTulach 7d23bd2
ServiceLoader doesn't want to load classes from Module.isNamed() - le…
JaroslavTulach 4ff07ca
Formatting build.sbt
JaroslavTulach 0fbcdc3
Avoid legal changes due to lookup library
JaroslavTulach b1ecbdf
Giving the lazy sequence a real IrLazySeq name
JaroslavTulach 36f1d7e
Adding LF at EOF
JaroslavTulach 13ae27e
It's a Seq
JaroslavTulach File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -254,6 +254,8 @@ lazy val buildNativeImage = | |
lazy val enso = (project in file(".")) | ||
.settings(version := "0.1") | ||
.aggregate( | ||
`persistance-dsl`, | ||
`persistance`, | ||
`interpreter-dsl`, | ||
`interpreter-dsl-test`, | ||
`json-rpc-server-test`, | ||
|
@@ -1074,6 +1076,35 @@ lazy val searcher = project | |
.dependsOn(testkit % Test) | ||
.dependsOn(`polyglot-api`) | ||
|
||
lazy val `persistance` = (project in file("lib/java/persistance")) | ||
.settings( | ||
version := "0.1", | ||
frgaalJavaCompilerSetting, | ||
Compile / javacOptions := ((Compile / javacOptions).value), | ||
libraryDependencies ++= Seq( | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion, | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test | ||
) | ||
) | ||
.dependsOn(`persistance-dsl` % Test) | ||
|
||
lazy val `persistance-dsl` = (project in file("lib/java/persistance-dsl")) | ||
.settings( | ||
version := "0.1", | ||
frgaalJavaCompilerSetting, | ||
Compile / javacOptions := ((Compile / javacOptions).value ++ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can do |
||
// Only run ServiceProvider processor and ignore those defined in META-INF, thus | ||
// fixing incremental compilation setup | ||
Seq( | ||
"-processor", | ||
"org.netbeans.modules.openide.util.ServiceProviderProcessor" | ||
)), | ||
libraryDependencies ++= Seq( | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided" | ||
) | ||
) | ||
|
||
lazy val `interpreter-dsl` = (project in file("lib/scala/interpreter-dsl")) | ||
.settings( | ||
version := "0.1", | ||
|
@@ -1087,7 +1118,7 @@ lazy val `interpreter-dsl` = (project in file("lib/scala/interpreter-dsl")) | |
)), | ||
libraryDependencies ++= Seq( | ||
"org.apache.commons" % "commons-lang3" % commonsLangVersion, | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion, | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided", | ||
JaroslavTulach marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"com.google.guava" % "guava" % guavaVersion exclude ("com.google.code.findbugs", "jsr305") | ||
) | ||
) | ||
|
@@ -1373,23 +1404,24 @@ lazy val runtime = (project in file("engine/runtime")) | |
scalacOptions += "-Ymacro-annotations", | ||
scalacOptions ++= Seq("-Ypatmat-exhaust-depth", "off"), | ||
libraryDependencies ++= jmh ++ jaxb ++ circe ++ GraalVM.langsPkgs ++ Seq( | ||
"org.apache.commons" % "commons-lang3" % commonsLangVersion, | ||
"org.apache.tika" % "tika-core" % tikaVersion, | ||
"org.graalvm.polyglot" % "polyglot" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.sdk" % "polyglot-tck" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-dsl-processor" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-tck" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-tck-common" % graalMavenPackagesVersion % "provided", | ||
"org.scalacheck" %% "scalacheck" % scalacheckVersion % Test, | ||
"org.scalactic" %% "scalactic" % scalacticVersion % Test, | ||
"org.scalatest" %% "scalatest" % scalatestVersion % Test, | ||
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % Benchmark, | ||
"org.typelevel" %% "cats-core" % catsVersion, | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test, | ||
"org.hamcrest" % "hamcrest-all" % hamcrestVersion % Test, | ||
"org.slf4j" % "slf4j-nop" % slf4jVersion % Benchmark | ||
"org.apache.commons" % "commons-lang3" % commonsLangVersion, | ||
"org.apache.tika" % "tika-core" % tikaVersion, | ||
"org.graalvm.polyglot" % "polyglot" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.sdk" % "polyglot-tck" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-dsl-processor" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-tck" % graalMavenPackagesVersion % "provided", | ||
"org.graalvm.truffle" % "truffle-tck-common" % graalMavenPackagesVersion % "provided", | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided", | ||
"org.scalacheck" %% "scalacheck" % scalacheckVersion % Test, | ||
"org.scalactic" %% "scalactic" % scalacticVersion % Test, | ||
"org.scalatest" %% "scalatest" % scalatestVersion % Test, | ||
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % Benchmark, | ||
"org.typelevel" %% "cats-core" % catsVersion, | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test, | ||
"org.hamcrest" % "hamcrest-all" % hamcrestVersion % Test, | ||
"org.slf4j" % "slf4j-nop" % slf4jVersion % Benchmark | ||
), | ||
// Add all GraalVM packages with Runtime scope - we don't need them for compilation, | ||
// just provide them at runtime (in module-path). | ||
|
@@ -1454,6 +1486,7 @@ lazy val runtime = (project in file("engine/runtime")) | |
.dependsOn(`common-polyglot-core-utils`) | ||
.dependsOn(`edition-updater`) | ||
.dependsOn(`interpreter-dsl`) | ||
.dependsOn(`persistance-dsl` % "provided") | ||
.dependsOn(`library-manager`) | ||
.dependsOn(`logging-truffle-connector`) | ||
.dependsOn(`polyglot-api`) | ||
|
@@ -1468,31 +1501,42 @@ lazy val `runtime-parser` = | |
.settings( | ||
frgaalJavaCompilerSetting, | ||
instrumentationSettings, | ||
commands += WithDebugCommand.withDebug, | ||
fork := true, | ||
Test / javaOptions ++= Seq( | ||
JaroslavTulach marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"-Dgraalvm.locatorDisabled=true", | ||
s"--upgrade-module-path=${file("engine/runtime/build-cache/truffle-api.jar").absolutePath}" | ||
), | ||
libraryDependencies ++= Seq( | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test, | ||
"org.scalatest" %% "scalatest" % scalatestVersion % Test | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test, | ||
"org.scalatest" %% "scalatest" % scalatestVersion % Test, | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided" | ||
) | ||
) | ||
.dependsOn(syntax) | ||
.dependsOn(`syntax-rust-definition`) | ||
.dependsOn(`persistance`) | ||
.dependsOn(`persistance-dsl` % "provided") | ||
|
||
lazy val `runtime-compiler` = | ||
(project in file("engine/runtime-compiler")) | ||
.settings( | ||
frgaalJavaCompilerSetting, | ||
instrumentationSettings, | ||
libraryDependencies ++= Seq( | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test, | ||
"org.scalatest" %% "scalatest" % scalatestVersion % Test, | ||
"com.lihaoyi" %% "fansi" % fansiVersion | ||
"junit" % "junit" % junitVersion % Test, | ||
"com.github.sbt" % "junit-interface" % junitIfVersion % Test, | ||
"org.scalatest" %% "scalatest" % scalatestVersion % Test, | ||
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided", | ||
"com.lihaoyi" %% "fansi" % fansiVersion | ||
) | ||
) | ||
.dependsOn(`runtime-parser`) | ||
.dependsOn(pkg) | ||
.dependsOn(`polyglot-api`) | ||
.dependsOn(editions) | ||
.dependsOn(`persistance-dsl` % "provided") | ||
|
||
lazy val `runtime-instrument-common` = | ||
(project in file("engine/runtime-instrument-common")) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be unnecessary