Deadlock in NI build wrt. classloading: "Image generator watchdog detected no activity" #7850

JaroslavTulach · 2023-09-20T07:18:29Z

There seem to be a deadlock in classloading of compiler passes caused by heavily parallel initialization of class constructors (e.g. <cinit>) during build of engine-runner/buildNativeImage.

Two builds showing the deadlock:

LambdaConsolidate$ vs. IgnoredBindings$
BindingAnalysis$ vs. GlobalNames$.

In both cases there are two threads initializing the compiler passes objects, each in different order:

"ForkJoinPool-2-worker-9" Id=76 in RUNNABLE
 at org.enso.compiler.pass.resolve.GlobalNames$.<clinit>(GlobalNames.scala:50)
 at org.enso.compiler.pass.analyse.TailCall$.<clinit>(TailCall.scala:58)
 at org.enso.compiler.pass.desugar.NestedPatternMatch$.<clinit>(NestedPatternMatch.scala:87)
 at org.enso.compiler.pass.optimise.ApplicationSaturation$.<clinit>(ApplicationSaturation.scala:37)
 at org.enso.compiler.pass.desugar.LambdaShorthandToLambda$.<clinit>(LambdaShorthandToLambda.scala:56)
 at org.enso.compiler.pass.resolve.IgnoredBindings$.<clinit>(IgnoredBindings.scala:47)
 at org.enso.compiler.pass.optimise.LambdaConsolidate$.<clinit>(LambdaConsolidate.scala:65)
 at org.enso.compiler.pass.analyse.DemandAnalysis$.<clinit>(DemandAnalysis.scala:46)
 at org.enso.compiler.pass.analyse.DataflowAnalysis$.<clinit>(DataflowAnalysis.scala:57)
 at org.enso.compiler.pass.desugar.SectionsToBinOp$.<clinit>(SectionsToBinOp.scala:36)
 at org.enso.compiler.pass.analyse.AliasAnalysis$.<clinit>(AliasAnalysis.scala:81)
 at org.enso.compiler.pass.desugar.FunctionBinding$.<clinit>(FunctionBinding.scala:46)
 at org.enso.compiler.pass.desugar.GenerateMethodBodies$.<clinit>(GenerateMethodBodies.scala:52)
 at org.enso.compiler.pass.resolve.DocumentationComments$.<clinit>(DocumentationComments.scala:42)
 at org.enso.compiler.pass.resolve.ModuleAnnotations$.<clinit>(ModuleAnnotations.scala:24)
 at org.enso.compiler.pass.desugar.ComplexType$.<clinit>(ComplexType.scala:56)
 at org.enso.compiler.pass.analyse.CachePreferenceAnalysis$.<clinit>(CachePreferenceAnalysis.scala:39)
 at [email protected]/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
 at [email protected]/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1155)
 at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.ensureClassInitialized(ClassInitializationSupport.java:168)

vs.

"ForkJoinPool-2-worker-12" Id=79 in RUNNABLE
at org.enso.compiler.pass.analyse.BindingAnalysis$.<clinit>(BindingAnalysis.scala:35)
at [email protected]/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
at [email protected]/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1155)
at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.ensureClassInitialized(ClassInitializationSupport.java:168)

These singletons that depend on other singletons seem to be dangerous. I am not sure how to convince scala to generate deadlock free code. @hubertp, please advice!

The text was updated successfully, but these errors were encountered:

hubertp · 2023-09-20T09:31:32Z

We don't have circular dependencies between passes - there is even a check for that. But passes can depend on the same pass (in this case BindingAnalysis is the usual culprit).

JaroslavTulach · 2023-09-20T11:02:48Z

What the BindingAnalysis<cinit> is blocked then? Another failed build:

JaroslavTulach · 2023-09-20T14:18:34Z

What does your disassembler says? Mine is showing:

enso$ /graalvm/bin/javap -c -cp ./engine/runtime/target/scala-2.13/classes/ 'org.enso.compiler.pass.analyse.BindingAnalysis$'
public final class org.enso.compiler.pass.analyse.BindingAnalysis$ implements org.enso.compiler.pass.IRPass,scala.Product {
  public static final org.enso.compiler.pass.analyse.BindingAnalysis$ MODULE$;

  public static {};
    Code:
       0: new           #2                  // class org/enso/compiler/pass/analyse/BindingAnalysis$
       3: dup
       4: invokespecial #64                 // Method "<init>":()V
       7: putstatic     #66                 // Field MODULE$:Lorg/enso/compiler/pass/analyse/BindingAnalysis$;
      10: getstatic     #66                 // Field MODULE$:Lorg/enso/compiler/pass/analyse/BindingAnalysis$;
      13: invokestatic  #70                 // InterfaceMethod org/enso/compiler/pass/IRPass.$init$:(Lorg/enso/compiler/pass/IRPass;)V
      16: getstatic     #66                 // Field MODULE$:Lorg/enso/compiler/pass/analyse/BindingAnalysis$;
      19: invokestatic  #73                 // InterfaceMethod scala/Product.$init$:(Lscala/Product;)V
      22: new           #75                 // class scala/collection/immutable/$colon$colon
      25: dup
      26: getstatic     #80                 // Field org/enso/compiler/pass/desugar/ComplexType$.MODULE$:Lorg/enso/compiler/pass/desugar/ComplexType$;
      29: new           #75                 // class scala/collection/immutable/$colon$colon
      32: dup
      33: getstatic     #85                 // Field org/enso/compiler/pass/desugar/FunctionBinding$.MODULE$:Lorg/enso/compiler/pass/desugar/FunctionBinding$;
      36: new           #75                 // class scala/collection/immutable/$colon$colon
      39: dup
      40: getstatic     #90                 // Field org/enso/compiler/pass/desugar/GenerateMethodBodies$.MODULE$:Lorg/enso/compiler/pass/desugar/GenerateMethodBodies$;
      43: getstatic     #95                 // Field scala/collection/immutable/Nil$.MODULE$:Lscala/collection/immutable/Nil$;
      46: invokespecial #98                 // Method scala/collection/immutable/$colon$colon."<init>":(Ljava/lang/Object;Lscala/collection/immutable/List;)V
      49: invokespecial #98                 // Method scala/collection/immutable/$colon$colon."<init>":(Ljava/lang/Object;Lscala/collection/immutable/List;)V
      52: invokespecial #98                 // Method scala/collection/immutable/$colon$colon."<init>":(Ljava/lang/Object;Lscala/collection/immutable/List;)V
      55: putstatic     #100                // Field precursorPasses:Lscala/collection/immutable/Seq;
      58: getstatic     #102                // Field bitmap$init$0:B
      61: iconst_1
      62: ior
      63: i2b
      64: putstatic     #102                // Field bitmap$init$0:B
      67: new           #75                 // class scala/collection/immutable/$colon$colon
      70: dup
      71: getstatic     #107                // Field org/enso/compiler/pass/resolve/MethodDefinitions$.MODULE$:Lorg/enso/compiler/pass/resolve/MethodDefinitions$;
      74: new           #75                 // class scala/collection/immutable/$colon$colon
      77: dup
      78: getstatic     #112                // Field org/enso/compiler/pass/resolve/Patterns$.MODULE$:Lorg/enso/compiler/pass/resolve/Patterns$;
      81: getstatic     #95                 // Field scala/collection/immutable/Nil$.MODULE$:Lscala/collection/immutable/Nil$;
      84: invokespecial #98                 // Method scala/collection/immutable/$colon$colon."<init>":(Ljava/lang/Object;Lscala/collection/immutable/List;)V
      87: invokespecial #98                 // Method scala/collection/immutable/$colon$colon."<init>":(Ljava/lang/Object;Lscala/collection/immutable/List;)V
      90: putstatic     #114                // Field invalidatedPasses:Lscala/collection/immutable/Seq;
      93: getstatic     #102                // Field bitmap$init$0:B
      96: iconst_2
      97: ior
      98: i2b
      99: putstatic     #102                // Field bitmap$init$0:B
     102: return

while this method is being executed, it has a lock on the class initializer - other class can access any field of BindingsAnalysis$. However, it is clear that the pass is accessing static fields of other classes (which are also behind a class loading lock). Clearly there is an access to GenerateMethodBodies$.MODULE$ - a class which participated in a deadlock just recently.

JaroslavTulach · 2023-09-20T14:26:12Z

We don't have circular dependencies between passes

In class org.enso.compiler.pass.desugar.GenerateMethodBodies$ there is

    33: getstatic     #117                // Field org/enso/compiler/pass/desugar/FunctionBinding$.MODULE$:Lorg/enso/compiler/pass/desugar/FunctionBinding$;

and in class class org.enso.compiler.pass.desugar.FunctionBinding$ there is

      81: getstatic     #208                // Field org/enso/compiler/pass/desugar/GenerateMethodBodies$.MODULE$:Lorg/enso/compiler/pass/desugar/GenerateMethodBodies$;

There is a cyclic dependency (of the <cinit> initializers) and it is causing the native image build to get stuck!

JaroslavTulach · 2023-09-28T08:37:55Z

Another failure: https://github.com/enso-org/enso/actions/runs/6335322564/job/17206297931?pr=7909#step:10:1293

GitHub
Make sure Integer can be treated as Float · enso-org/enso@5b2d670
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

Akirathan · 2023-10-02T08:11:43Z

Three more failures:

GitHub
[gui2] Removing nodes · enso-org/enso@411e363
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

GitHub
[gui2] Shortcut manager + shortcuts for selecting nodes · enso-org/enso@938190e
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

…7943) Fixes #7850 by computing other passes lazily.

JaroslavTulach added p-high Should be completed in the next sprint -ci -compiler labels Sep 20, 2023

JaroslavTulach assigned hubertp Sep 20, 2023

github-project-automation bot added this to Issues Board Sep 20, 2023

github-project-automation bot moved this to ❓New in Issues Board Sep 20, 2023

JaroslavTulach mentioned this issue Sep 20, 2023

Transient issues in backend tests #7775

Closed

25 tasks

enso-bot bot mentioned this issue Sep 21, 2023

Methods with type parameter are reported as errorneous #6885

Closed

jdunkerley moved this from ❓New to 📤 Backlog in Issues Board Sep 26, 2023

JaroslavTulach mentioned this issue Oct 2, 2023

Avoid classloading deadlock by delaying computations of other passes #7943

Merged

2 tasks

JaroslavTulach assigned JaroslavTulach and unassigned hubertp Oct 2, 2023

JaroslavTulach moved this from 📤 Backlog to 👁️ Code review in Issues Board Oct 2, 2023

mergify bot closed this as completed in #7943 Oct 2, 2023

mergify bot pushed a commit that referenced this issue Oct 2, 2023

Avoid classloading deadlock by delaying computations of other passes (#…

515e660

…7943) Fixes #7850 by computing other passes lazily.

github-project-automation bot moved this from 👁️ Code review to 🟢 Accepted in Issues Board Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deadlock in NI build wrt. classloading: "Image generator watchdog detected no activity" #7850

Deadlock in NI build wrt. classloading: "Image generator watchdog detected no activity" #7850

JaroslavTulach commented Sep 20, 2023

hubertp commented Sep 20, 2023

JaroslavTulach commented Sep 20, 2023 •

edited

Loading

JaroslavTulach commented Sep 20, 2023 •

edited

Loading

JaroslavTulach commented Sep 20, 2023

JaroslavTulach commented Sep 28, 2023 •

edited by unfurl-links bot

Loading

Akirathan commented Oct 2, 2023 •

edited

Loading

Deadlock in NI build wrt. classloading: "Image generator watchdog detected no activity" #7850

Deadlock in NI build wrt. classloading: "Image generator watchdog detected no activity" #7850

Comments

JaroslavTulach commented Sep 20, 2023

hubertp commented Sep 20, 2023

JaroslavTulach commented Sep 20, 2023 • edited Loading

JaroslavTulach commented Sep 20, 2023 • edited Loading

JaroslavTulach commented Sep 20, 2023

JaroslavTulach commented Sep 28, 2023 • edited by unfurl-links bot Loading

Akirathan commented Oct 2, 2023 • edited Loading

JaroslavTulach commented Sep 20, 2023 •

edited

Loading

JaroslavTulach commented Sep 20, 2023 •

edited

Loading

JaroslavTulach commented Sep 28, 2023 •

edited by unfurl-links bot

Loading

Akirathan commented Oct 2, 2023 •

edited

Loading