SerializationManager attempts to create a system thread without context available

[wdanilo]

This task is automatically imported from the old Task Issue Board and it was originally created by Hubert Plociniczak.
Original issue is here.

---

Reported crash

Feb 03 11:57:30 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:57:30.265Z] [org.enso.languageserver.runtime.ContextEventsListener] Unable to find suggestion for [(Standard.Base.Data.Range.Extensions,Standard.Base.Data.Numbers.Integer,up_to)].
Feb 03 11:57:30 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:57:30.277Z] [org.enso.languageserver.runtime.ContextEventsListener] Unable to find suggestion for [(Standard.Base.Data.Range,Standard.Base.Data.Range.Range,to_vector)].
Feb 03 11:57:31 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:57:31.221Z] [org.enso.languageserver.runtime.ContextEventsListener] Unable to find suggestion for [(Standard.Base.Data.Vector,Standard.Base.Data.Vector.Vector,map)].
Feb 03 11:57:31 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:57:31.221Z] [org.enso.languageserver.runtime.ContextEventsListener] Unable to find suggestion for [(Standard.Base.Data.Vector,Standard.Base.Data.Vector.Vector,sort)].
Feb 03 11:57:40 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:57:40.852Z] [enso] Visualisation thread interrupted cc52c38b-5fde-4c6d-89af-8e696c32f65b.
Feb 03 11:57:41 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:57:41.418Z] [enso] Visualisation thread interrupted cc52c38b-5fde-4c6d-89af-8e696c32f65b.
Feb 03 11:58:03 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:58:03.828Z] [enso] Execution of function main interrupted.
Feb 03 11:58:12 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:58:12.808Z] [enso] Visualisation evaluation failed: Method `from_rows` of Table.Value could not be found..
Feb 03 11:58:13 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:58:13.777Z] [enso] Visualisation thread interrupted 9e82d797-7e7f-4c2c-b19a-ee9292d44385.
Feb 03 11:58:18 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:58:18.389Z] [org.enso.languageserver.requesthandler.visualisation.DetachVisualisationHandler] Request [Number(62)] timed out.
Feb 03 11:58:23 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:58:23.389Z] [akka.actor.LocalActorRef] Message [org.enso.languageserver.requesthandler.RequestTimeout$] from Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-DetachVisualisation-0d780658-7ac3-48a4-b782-bed9c725e380#-105406540] to Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-DetachVisualisation-0d780658-7ac3-48a4-b782-bed9c725e380#-105406540] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-DetachVisualisation-0d780658-7ac3-48a4-b782-bed9c725e380#-105406540] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: Exception in thread "Thread-9" java.lang.IllegalStateException: There is no current context available.
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotEngineException.illegalState(PolyglotEngineException.java:135)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.requireContext(PolyglotContextImpl.java:720)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.graalvm.truffle/com.oracle.truffle.polyglot.EngineAccessor$EngineImpl.createLanguageSystemThread(EngineAccessor.java:1940)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.graalvm.truffle/com.oracle.truffle.api.TruffleLanguage$Env.createSystemThread(TruffleLanguage.java:1847)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.graalvm.truffle/com.oracle.truffle.api.TruffleLanguage$Env.createSystemThread(TruffleLanguage.java:1813)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.enso.compiler.SerializationManager.$anonfun$pool$1(SerializationManager.scala:54)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:623)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:912)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at java.base/java.lang.Thread.run(Thread.java:829)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]:         at org.graalvm.truffle/com.oracle.truffle.polyglot.SystemThread.run(SystemThread.java:65)
Feb 03 11:58:29 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:58:29.211Z] [enso] Visualisation evaluation failed: Method `Data` of Main could not be found..
Feb 03 11:59:12 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:59:12.109Z] [org.enso.languageserver.requesthandler.visualisation.AttachVisualisationHandler] Request [Number(96)] timed out.
Feb 03 11:59:17 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:17.108Z] [akka.actor.LocalActorRef] Message [org.enso.languageserver.requesthandler.RequestTimeout$] from Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-AttachVisualisation-fa89be4f-be94-43c6-a4ae-dc7f3db884e4#691987073] to Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-AttachVisualisation-fa89be4f-be94-43c6-a4ae-dc7f3db884e4#691987073] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-AttachVisualisation-fa89be4f-be94-43c6-a4ae-dc7f3db884e4#691987073] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Feb 03 11:59:36 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [error] [2023-02-03T11:59:36.318Z] [org.enso.languageserver.requesthandler.visualisation.AttachVisualisationHandler] Request [Number(120)] timed out.
Feb 03 11:59:40 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:40.976Z] [org.enso.languageserver.protocol.binary.BinaryConnectionController] Connection closed [86.21.172.249].
Feb 03 11:59:41 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:41.318Z] [akka.actor.LocalActorRef] Message [org.enso.languageserver.requesthandler.RequestTimeout$] from Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-AttachVisualisation-0452468c-59b5-4803-9452-89652d81da92#608537295] to Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-AttachVisualisation-0452468c-59b5-4803-9452-89652d81da92#608537295] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://language-server/user/json-connection-controller-5d4529ad-9655-4ce9-a43b-09ed8f53f0be/request-handler-AttachVisualisation-0452468c-59b5-4803-9452-89652d81da92#608537295] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.053Z] [org.enso.languageserver.protocol.json.JsonConnectionController] Initializing resources.
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.054Z] [org.enso.languageserver.boot.resource.DirectoriesInitialization] Initializing directories...
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.054Z] [org.enso.languageserver.boot.resource.DirectoriesInitialization] Initialized directories.
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.054Z] [org.enso.languageserver.boot.resource.RepoInitialization] Initializing suggestions repo [***/suggestions.db]...
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.06Z] [org.enso.languageserver.boot.resource.RepoInitialization] Initialized Suggestions repo [***/suggestions.db].
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.06Z] [org.enso.languageserver.boot.resource.RepoInitialization] Initializing versions repo [***/suggestions.db]...
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [warn] [2023-02-03T11:59:50.071Z] [org.enso.languageserver.search.SuggestionsHandler] Received unknown message: class org.enso.languageserver.event.InitializedEvent$SuggestionsRepoInitialized$
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.071Z] [org.enso.languageserver.boot.resource.RepoInitialization] Initialized Versions repo [***/suggestions.db].
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.075Z] [org.enso.languageserver.boot.resource.TruffleContextInitialization] Initializing Runtime context...
Feb 03 11:59:50 ip-172-31-11-182.eu-west-1.compute.internal enso_runtime[903154]: [info] [2023-02-03T11:59:50.076Z] [org.enso.languageserver.boot.resource.TruffleContextInitialization] Initialized Runtime context.

Appears to have been added in the recent GraalVM upgrade.

Related discussion: https://discord.com/channels/401396655599124480/1069632659317325844/1071038218616709171

Comments:

Similar to #184390155 (Hubert Plociniczak - Feb 3, 2023)

---

I changed https://github.com/enso-org/enso/blob/develop/engine/runtime/src/main/scala/org/enso/compiler/SerializationManager.scala#L54 from `env.createThread` to `env.createSystemThread`, because native image of engine failed at runtime with "IllegalStateException: Multi threaded access requested by thread Thread[Polyglot-enso-0,5,main] but is not allowed for language(s) js". An excerpt from my local notes is in the attachment, might be helpful. (Pavel Marek - Feb 3, 2023)

---

Changing the aforementioned line back to `env.createThread` might fix the issue. But we should make sure that the native image still runs after that. If there is no time for that, possibly postponing fixes for the native image for later time. (Pavel Marek - Feb 3, 2023)

---

[hubertp]

Duplicate of #5037