Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store IR caches in suitable format to read them "on demand" #8207

Merged
merged 102 commits into from
Nov 19, 2023
Merged

Store IR caches in suitable format to read them "on demand" #8207

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
61d2d01
Framework for persisting objects into ByteBuffer
JaroslavTulach Nov 1, 2023
a59e9b7
Turning Location into record
JaroslavTulach Nov 1, 2023
999bea6
Testing simple serde of Location
JaroslavTulach Nov 1, 2023
f8ff83a
Enable debugging in runtime-parser tests
JaroslavTulach Nov 2, 2023
c845572
Test persistance of composed classes
JaroslavTulach Nov 2, 2023
95298f4
Allow access to Truffle API from runtime-parser/test
JaroslavTulach Nov 2, 2023
89df854
Encapsulating Persistance lookup
JaroslavTulach Nov 2, 2023
236cc08
Deserialize nested objects
JaroslavTulach Nov 2, 2023
04b97ef
Don't serialize UUIDs
JaroslavTulach Nov 2, 2023
3fe9e33
Adjusting imports to new location of Location
JaroslavTulach Nov 2, 2023
7a7f9d9
Turning IdentifiedLocation into record
JaroslavTulach Nov 2, 2023
98f720b
Save some memory by using UUID directly without Some(UUID) overhead
JaroslavTulach Nov 2, 2023
c549943
Making the tests compiler again
JaroslavTulach Nov 2, 2023
400b151
Using own Persistance Input and Output types
JaroslavTulach Nov 2, 2023
b583697
Measure the expected size of the serialized data
JaroslavTulach Nov 2, 2023
e26c83f
Storing objects inline lowers the size of the stream
JaroslavTulach Nov 2, 2023
d2afbab
References don't need identification of the object type
JaroslavTulach Nov 2, 2023
aa131e5
Support for serializing Scala List
JaroslavTulach Nov 2, 2023
de85e83
Keep the API stream based
JaroslavTulach Nov 2, 2023
2df600b
Using Persistance to store IR
JaroslavTulach Nov 2, 2023
9260903
Support null values in writeObject
JaroslavTulach Nov 2, 2023
f52c7e1
Can persist some ir.Module
JaroslavTulach Nov 2, 2023
a25bfb6
Use class loader of the language when searching for provides
JaroslavTulach Nov 2, 2023
ff2c9ac
Log also runtime exceptions
JaroslavTulach Nov 3, 2023
704ac72
Don't swallow exceptions - log them
JaroslavTulach Nov 3, 2023
7e6e9b5
Can read IR for (some) import statements
JaroslavTulach Nov 3, 2023
1f1d797
Can read (some) export statements
JaroslavTulach Nov 3, 2023
c8c8366
Support for lazy Seq
JaroslavTulach Nov 3, 2023
cd12f51
Using lazy Seq to delay loading of method body
JaroslavTulach Nov 3, 2023
719993a
Persistance for Name.MethodReference
JaroslavTulach Nov 3, 2023
0740709
Persistance for imports.Polyglot
JaroslavTulach Nov 3, 2023
9721fbb
Persistance for Function.Lambda
JaroslavTulach Nov 3, 2023
db4f8f2
Annotation processor for @Persistable to automate Persistance of simp…
JaroslavTulach Nov 3, 2023
14c2282
Able to persist signatures
JaroslavTulach Nov 8, 2023
f888c4f
Able to persist IR for Standard.Visualization
JaroslavTulach Nov 8, 2023
3391d6e
Persistance has writeObject and readObject static methods
JaroslavTulach Nov 8, 2023
bab61a1
Using Persistance to store MapToBindings
JaroslavTulach Nov 8, 2023
6d15515
Library BindingsMaps get serialized without issues
JaroslavTulach Nov 9, 2023
b05ec17
Persistance of MetadataStorage
JaroslavTulach Nov 9, 2023
4a5809a
More of persistance for metadata
JaroslavTulach Nov 9, 2023
5413e2f
Persist all caches
JaroslavTulach Nov 9, 2023
a76a9bb
PersistanceMap is singleton
JaroslavTulach Nov 9, 2023
2d1b62a
Formatting
JaroslavTulach Nov 9, 2023
2d60832
Lazy BindingsMap
JaroslavTulach Nov 10, 2023
5aacca7
Caching read objects
JaroslavTulach Nov 10, 2023
73d56e0
Avoid empty objects by appending trailing zero
JaroslavTulach Nov 10, 2023
e56d043
Warn when a cache cannot be loaded
JaroslavTulach Nov 10, 2023
d4a1460
Manual persistance of AliasAnalysis.Graph and co.
JaroslavTulach Nov 10, 2023
44d1af1
Java formatting
JaroslavTulach Nov 10, 2023
0a1f325
Merge remote-tracking branch 'origin/develop' into wip/jtulach/Persis…
JaroslavTulach Nov 10, 2023
3ed31ac
Making Function.Lambda body list lazy
JaroslavTulach Nov 12, 2023
da5ecdc
IDE complained about missing Double conversion
JaroslavTulach Nov 13, 2023
f2886a2
More case classes elligible for serialization
JaroslavTulach Nov 13, 2023
0f77ac0
Add missing import to address #8289
JaroslavTulach Nov 13, 2023
20f5ae8
Merging with CompilerContext based serialization
JaroslavTulach Nov 13, 2023
4a314ef
Support for services associated with Persistance.Input
JaroslavTulach Nov 13, 2023
0363603
Making runtime tests compile again and SerdeTest pass
JaroslavTulach Nov 13, 2023
5e52147
Perform restoreFromSerialization on demand, not in a batch
JaroslavTulach Nov 13, 2023
56663af
No need for cross module links: either the module deserializes OK or …
JaroslavTulach Nov 13, 2023
78bab90
Modules loaded from cache shall have no errors
JaroslavTulach Nov 13, 2023
b66bfa1
Store Method.Explicit.isStatic information in caches to avoid materia…
JaroslavTulach Nov 13, 2023
177c44e
Delay construction of RuntimeFunction until really needed
JaroslavTulach Nov 14, 2023
ef84e1d
Serializing errors.ImportExport now fails
JaroslavTulach Nov 14, 2023
95ff6a0
Compare IR before and after serialization test
JaroslavTulach Nov 15, 2023
5cb4b68
Make sure the richest constructor (that includes all variables) is us…
JaroslavTulach Nov 15, 2023
98b95c4
Adding persistance for two more IR types
JaroslavTulach Nov 15, 2023
68ed2a7
Use @Persistable annotation for more classes
JaroslavTulach Nov 15, 2023
7c562b4
Turn linkage errors into AssertionError
JaroslavTulach Nov 16, 2023
0f21b80
prepareForSerialization just before writing MetadataStorage down
JaroslavTulach Nov 16, 2023
bb1bc94
Using ir.duplicate(keepLocations=false...) to eliminate locations
JaroslavTulach Nov 16, 2023
b172c85
Avoid prepareForSerialization calls that modify the IR. Do the conver…
JaroslavTulach Nov 16, 2023
120ac13
Make sure Scala's Seq is stored inline
JaroslavTulach Nov 16, 2023
cacf9f0
Merge remote-tracking branch 'origin/develop' into wip/jtulach/Persis…
JaroslavTulach Nov 16, 2023
2122da8
Introducing writeReplace and readResolve functions
JaroslavTulach Nov 16, 2023
497e300
Either convert AbstractModule or stop deserialization
JaroslavTulach Nov 16, 2023
0fec07d
Fix exhaustiveness check
hubertp Nov 16, 2023
aad192b
Treat List as Seq for purposes of the test
JaroslavTulach Nov 16, 2023
385b3b8
Merge branch 'wip/jtulach/Persistance_6100' of enso:enso-org/enso int…
JaroslavTulach Nov 16, 2023
c40c7c4
scalafmt
JaroslavTulach Nov 16, 2023
79482a1
Moving Persistance classes and annotation processor into their own pr…
JaroslavTulach Nov 16, 2023
b7f2e09
Lookup library shall only be used at compile time
JaroslavTulach Nov 17, 2023
76859b9
Splitting implementation away from the API classes
JaroslavTulach Nov 17, 2023
f5238b0
legal
hubertp Nov 17, 2023
5e8d880
bring back more precise patterns
hubertp Nov 17, 2023
05be437
Adding tests to persistance project
JaroslavTulach Nov 17, 2023
2231152
Basic documentation of the org.enso.persistance package
JaroslavTulach Nov 17, 2023
3586c33
More code examples
JaroslavTulach Nov 17, 2023
3370745
more legal
hubertp Nov 17, 2023
a4048a6
Documenting the @Persistable annotation
JaroslavTulach Nov 17, 2023
5300790
one more legal change
hubertp Nov 17, 2023
73b63f3
Shorter package names
JaroslavTulach Nov 17, 2023
2c1a2c1
Merge branch 'wip/jtulach/Persistance_6100' of enso:enso-org/enso int…
JaroslavTulach Nov 17, 2023
dbf62e2
Complete javadoc for all API elements
JaroslavTulach Nov 17, 2023
7c9c23c
Merging with GraalVM for JDK21 effort
JaroslavTulach Nov 18, 2023
54a0471
Updating the IR caches documentation
JaroslavTulach Nov 18, 2023
b22f5eb
Documenting PersistableProcessor
JaroslavTulach Nov 18, 2023
7d23bd2
ServiceLoader doesn't want to load classes from Module.isNamed() - le…
JaroslavTulach Nov 18, 2023
4ff07ca
Formatting build.sbt
JaroslavTulach Nov 18, 2023
0fbcdc3
Avoid legal changes due to lookup library
JaroslavTulach Nov 18, 2023
b1ecbdf
Giving the lazy sequence a real IrLazySeq name
JaroslavTulach Nov 18, 2023
36f1d7e
Adding LF at EOF
JaroslavTulach Nov 18, 2023
13ae27e
It's a Seq
JaroslavTulach Nov 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 69 additions & 25 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,8 @@ lazy val buildNativeImage =
lazy val enso = (project in file("."))
.settings(version := "0.1")
.aggregate(
`persistance-dsl`,
`persistance`,
`interpreter-dsl`,
`interpreter-dsl-test`,
`json-rpc-server-test`,
Expand Down Expand Up @@ -1074,6 +1076,35 @@ lazy val searcher = project
.dependsOn(testkit % Test)
.dependsOn(`polyglot-api`)

lazy val `persistance` = (project in file("lib/java/persistance"))
.settings(
version := "0.1",
frgaalJavaCompilerSetting,
Compile / javacOptions := ((Compile / javacOptions).value),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be unnecessary

libraryDependencies ++= Seq(
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion,
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test
)
)
.dependsOn(`persistance-dsl` % Test)

lazy val `persistance-dsl` = (project in file("lib/java/persistance-dsl"))
.settings(
version := "0.1",
frgaalJavaCompilerSetting,
Compile / javacOptions := ((Compile / javacOptions).value ++
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can do javacOptons ++= Seq() the same way it is done for libraryDependencies

// Only run ServiceProvider processor and ignore those defined in META-INF, thus
// fixing incremental compilation setup
Seq(
"-processor",
"org.netbeans.modules.openide.util.ServiceProviderProcessor"
)),
libraryDependencies ++= Seq(
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided"
)
)

lazy val `interpreter-dsl` = (project in file("lib/scala/interpreter-dsl"))
.settings(
version := "0.1",
Expand All @@ -1087,7 +1118,7 @@ lazy val `interpreter-dsl` = (project in file("lib/scala/interpreter-dsl"))
)),
libraryDependencies ++= Seq(
"org.apache.commons" % "commons-lang3" % commonsLangVersion,
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion,
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided",
JaroslavTulach marked this conversation as resolved.
Show resolved Hide resolved
"com.google.guava" % "guava" % guavaVersion exclude ("com.google.code.findbugs", "jsr305")
)
)
Expand Down Expand Up @@ -1373,23 +1404,24 @@ lazy val runtime = (project in file("engine/runtime"))
scalacOptions += "-Ymacro-annotations",
scalacOptions ++= Seq("-Ypatmat-exhaust-depth", "off"),
libraryDependencies ++= jmh ++ jaxb ++ circe ++ GraalVM.langsPkgs ++ Seq(
"org.apache.commons" % "commons-lang3" % commonsLangVersion,
"org.apache.tika" % "tika-core" % tikaVersion,
"org.graalvm.polyglot" % "polyglot" % graalMavenPackagesVersion % "provided",
"org.graalvm.sdk" % "polyglot-tck" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-dsl-processor" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-tck" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-tck-common" % graalMavenPackagesVersion % "provided",
"org.scalacheck" %% "scalacheck" % scalacheckVersion % Test,
"org.scalactic" %% "scalactic" % scalacticVersion % Test,
"org.scalatest" %% "scalatest" % scalatestVersion % Test,
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % Benchmark,
"org.typelevel" %% "cats-core" % catsVersion,
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test,
"org.hamcrest" % "hamcrest-all" % hamcrestVersion % Test,
"org.slf4j" % "slf4j-nop" % slf4jVersion % Benchmark
"org.apache.commons" % "commons-lang3" % commonsLangVersion,
"org.apache.tika" % "tika-core" % tikaVersion,
"org.graalvm.polyglot" % "polyglot" % graalMavenPackagesVersion % "provided",
"org.graalvm.sdk" % "polyglot-tck" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-dsl-processor" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-tck" % graalMavenPackagesVersion % "provided",
"org.graalvm.truffle" % "truffle-tck-common" % graalMavenPackagesVersion % "provided",
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided",
"org.scalacheck" %% "scalacheck" % scalacheckVersion % Test,
"org.scalactic" %% "scalactic" % scalacticVersion % Test,
"org.scalatest" %% "scalatest" % scalatestVersion % Test,
"org.graalvm.truffle" % "truffle-api" % graalMavenPackagesVersion % Benchmark,
"org.typelevel" %% "cats-core" % catsVersion,
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test,
"org.hamcrest" % "hamcrest-all" % hamcrestVersion % Test,
"org.slf4j" % "slf4j-nop" % slf4jVersion % Benchmark
),
// Add all GraalVM packages with Runtime scope - we don't need them for compilation,
// just provide them at runtime (in module-path).
Expand Down Expand Up @@ -1454,6 +1486,7 @@ lazy val runtime = (project in file("engine/runtime"))
.dependsOn(`common-polyglot-core-utils`)
.dependsOn(`edition-updater`)
.dependsOn(`interpreter-dsl`)
.dependsOn(`persistance-dsl` % "provided")
.dependsOn(`library-manager`)
.dependsOn(`logging-truffle-connector`)
.dependsOn(`polyglot-api`)
Expand All @@ -1468,31 +1501,42 @@ lazy val `runtime-parser` =
.settings(
frgaalJavaCompilerSetting,
instrumentationSettings,
commands += WithDebugCommand.withDebug,
fork := true,
Test / javaOptions ++= Seq(
JaroslavTulach marked this conversation as resolved.
Show resolved Hide resolved
"-Dgraalvm.locatorDisabled=true",
s"--upgrade-module-path=${file("engine/runtime/build-cache/truffle-api.jar").absolutePath}"
),
libraryDependencies ++= Seq(
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test,
"org.scalatest" %% "scalatest" % scalatestVersion % Test
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test,
"org.scalatest" %% "scalatest" % scalatestVersion % Test,
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided"
)
)
.dependsOn(syntax)
.dependsOn(`syntax-rust-definition`)
.dependsOn(`persistance`)
.dependsOn(`persistance-dsl` % "provided")

lazy val `runtime-compiler` =
(project in file("engine/runtime-compiler"))
.settings(
frgaalJavaCompilerSetting,
instrumentationSettings,
libraryDependencies ++= Seq(
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test,
"org.scalatest" %% "scalatest" % scalatestVersion % Test,
"com.lihaoyi" %% "fansi" % fansiVersion
"junit" % "junit" % junitVersion % Test,
"com.github.sbt" % "junit-interface" % junitIfVersion % Test,
"org.scalatest" %% "scalatest" % scalatestVersion % Test,
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided",
"com.lihaoyi" %% "fansi" % fansiVersion
)
)
.dependsOn(`runtime-parser`)
.dependsOn(pkg)
.dependsOn(`polyglot-api`)
.dependsOn(editions)
.dependsOn(`persistance-dsl` % "provided")

lazy val `runtime-instrument-common` =
(project in file("engine/runtime-instrument-common"))
Expand Down
1 change: 1 addition & 0 deletions distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Expand_Objects_Helpers.enso
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import project.Extensions.Prefix_Name.Prefix_Name
import project.Internal.Fan_Out
import project.Internal.Java_Exports
import project.Internal.Java_Problems
import project.Internal.Widget_Helpers
from project.Internal.Java_Exports import make_inferred_builder

## PRIVATE
Expand Down
Loading
Loading