pub serve hangs forever after code change on 0.13.0

[ewot]

Pub serve (both from cli and from editor) works on first access. But after changing a dart file and reloading the app the browser hangs forever on loading the .dart or .dart.js file depending on if using Dartium or Chrome.

To reproduce:

1. run the hello_world.dart example from Dart editor
2. wait until initialized and running in Dartium
3. change some whitespace in hello_world.dart
4. reload i dartium -> hangs forever on request GET hello_world.dart

This behaviour started after upgrading from 0.12 and adding the angular transformer to pubspec.yaml.

I'm running on Ubuntu 14.04 64bit and Dart SDK version 1.6.0-dev.4.0.

[taisph]

I'm seeing the same behavior on Ubuntu 14.04 64bit and Dart SDK version 1.5.3.

[ewot]

One workaround for me was to turn off the angular transformer and set DI to use dynamic type factories:

  Module.DEFAULT_REFLECTOR = new DynamicTypeFactories();
  applicationFactory()
      .addModule(new MainModule())
      .run();

I guess this generates huge output but at least the turn-around time is fast while developing. Hopefully there is a way to have the transformers off while running from editor and turn them on while running from cli, but I don't know how.

[taisph]

I can confirm setting Module.DEFAULT_REFLECTOR = new DynamicTypeFactories(); stops pub serve from hanging on reload and re-runs (without explicitly stopping pub serve first). I didn't remove the angular transformer though.

[vicb]

Things you can try:

• run a pub upgrade (or upgarde via the editor) and report if this changes anything,
• run pub serve --verbose from the command line rather than from the IDE (to see if the issue is related to angular or the editor).

[taisph]

@vicb I just tried that and it hangs with no apparent error. The trigger seems to be changing and saving a dart-file. I assume it should say MSG : Build completed successfully after the save as it does when editing and saving an html-file but it never does.

Reloading the browser page shows a few requests and then just sits there waiting for "main.dart".

pub serve output hitting reload in browser window:

FINE: Instance of 'BarbackServer' GET /packages/browser/dart.js
    | Loading browser|lib/dart.js
FINE: Instance of 'BarbackServer' GET /packages/browser/dart.js
    | getAssetById(browser|lib/dart.js) returned
FINE: Instance of 'BarbackServer' GET /index.html
    | Loading angular_dart_attach_test|web/index.html
FINE: Instance of 'BarbackServer' GET /index.html
    | getAssetById(angular_dart_attach_test|web/index.html) returned
MSG : [web] GET /packages/browser/dart.js → browser|lib/dart.js
MSG : [web] GET /index.html → angular_dart_attach_test|web/index.html
FINE: Instance of 'BarbackServer' GET /main.dart
    | Loading angular_dart_attach_test|web/main.dart
FINE: Instance of 'BarbackServer' GET /packages/browser/dart.js
    | Loading browser|lib/dart.js
FINE: Instance of 'BarbackServer' GET /packages/browser/dart.js
    | getAssetById(browser|lib/dart.js) returned
MSG : [web] GET /packages/browser/dart.js → browser|lib/dart.js

[vicb]

And if you terminate pub serve (ctrl+c) and execute it again ?

[taisph]

Browser says GET http://localhost:8080/main.dart net::ERR_CONNECTION_REFUSED. When I start pub again I can reload the browser properly.

I also noticed that MSG : [web] GET /main.dart → angular_dart_attach_test|web/main.dart never shows when it gets stuck.

[vicb]

I can reproduce the behavior (save + reload = hang). But when I terminate pub serve and relaunch it, it works. This is not an angular issue but a pub issue. Please create a ticket on dartbug.com

[taisph]

I cannot reproduce this when I downgrade angular to 0.12.0. Could you be more specific as to why this is a pub problem because I'm sure they'll throw it right back here given this information.

[vicb]

It is not an angular issue because:

• get a clean 0.13 & execute pub upgrade in the example folder
• execute pub serve & visit localhost:8080/hello_wold.html -> ok
• terminate pub
• update hello_world.dart
• execute pub serve & visit localhost:8080/hello_wold.html -> ok

Then there is not issue with angular.
However if you do not terminate pub before modifying the dart file, it fails.

[ewot]

It might still be a problem with the angular transformer. Removing it from pubspec fixes the problem.

However, I don't know if it is a general problem with transformers in pub or if it is the angular transformer specifically that triggers the issue.

[taisph]

I completely agree that that would point towards pub. I'm just focusing on the changes in Angular's transformer since disabling this seems to "fix" the problem and changes were made here between 0.12.0 and 0.13.0 - while pub itself didn't change. I just know a bug report towards pub would be rejected if I don't provide a near smoking gun. My last bug report against pub throwing an out-of-memory exception while upgrading was rejected "as designed". 😄

Poking around in the angular transformer it would seem like the InjectorGenerator change in lib/transformer.dart is correlated with the deadlock. I'm not experienced enough to completely understand the implementation but removing it seems to prevent the deadlock from happening - I can keep curl-getting the main.dart and I see "build completed successfully" messages when I change dart-files. I'll poke around some more before bothering the Dart team.

[vicb]

@taisph Maybe you can start by creating this dartbug with your findings and say that you're investigating. They can put the ticket on hold waiting for info if necessary.

[taisph]

I've used for too much time on this already. Reported the bug at dartbug.com.

https://code.google.com/p/dart/issues/detail?id=20312

[nex3]

This is probably actually an issue in Angular and/or code_transformers.

Some simple logging indicates that the ExpressionGenerator transformer is what's actually deadlocking here. This transformer uses ResolverTransformer, which does complex synchronization of resources between multiple transformers. This sort of synchronization has been known to cause deadlocks in the past; it must be very careful not to violate barback's assumption that transformers can safely be run in parallel (modulo asset dependencies).

If you can reproduce this with transformers that don't use shared instance state or isolate a simple case where barback is misbehaving, go ahead and file a bug against barback. Until then, though, I think the evidence is on the side of this being an issue with the transformers.

[vicb]

@nex3 Thanks for the info.

As indicated above, there is no problem when the transformers run for the first time (i.e. you terminate pub serve before editing and re-launch it right after).

How would that relate to your comments ?

Could the ResolverTransformer be updated to generate an error on deadlock ?

Given our implementation should we be concerned about transformers running in // ?

If you can reproduce this with transformers that don't use shared instance state or isolate a simple case where barback is misbehaving, go ahead and file a bug against barback.

Any guidelines / tip on how to do that ?

Thanks for your comments.

[nex3]

As indicated above, there is no problem when the transformers run for the first time (i.e. you terminate pub serve before editing and re-launch it right after).

How would that relate to your comments ?

The first run of a transformer is often more straightforward than future runs, since everything's a clean slate; no incremental updates need to be done, either by barback or by code_transformers.

Could the ResolverTransformer be updated to generate an error on deadlock ?

Detecting deadlocks in general is halting-complete. A common workaround is to report an error after some timeout, but this is necessarily imprecise.

Given our implementation should we be concerned about transformers running in // ?

I don't understand what you mean by "in //".

If you can reproduce this with transformers that don't use shared instance state or isolate a simple case where barback is misbehaving, go ahead and file a bug against barback.

Any guidelines / tip on how to do that ?

I often iteratively remove functionality while adding logging to isolate the core of the problem, then see if I can construct that core from simpler components.

[vicb]

by "//" I mean parallel.

I'll try to investigate it. Thanks again for the tips.

[bgourlie]

+1

This is preventing me from upgrading to 0.13. Would be really nice to see this fixed in 0.14.

[vicb]

Any help solving is highly appreciated, thanks.

[taisph]

So far I've narrowed it down to one task in the "visiting" FutureGroup in _performResolve(), resolver_impl.dart in code_transformers that never completes. I haven't figured out which specific task it is yet.

[taisph]

When a dart-file is changed after the initial run, transform.readInputAsString(assetId) (in code_transformers:ResolverImpl:_performResolve()) never returns for assetId angular_dart_attach_test|web/main_generated_type_factory_maps.dart which di generated from main.dart during the initial run. transform says it's an instance of ForeignTransform which I can't seem to find anywhere. Google hints at Barback but I couldn't find the class there so I'm kind of stuck now.

[nex3]

That suggests that whatever's generating web/main_generated_type_factory_maps.dart is blocked on the transform in question.

[taisph]

It may be a kind of race condition. It seems di InjectorGenerator:applyResolver() does the generation but applyResolver isn't called until after ResolverImpl:_performResolve() returns. At least on the initial run.

[taisph]

How deep does the rabbithole go?

Below is the blocking call. It is ForeignTransform:getInput() in the dart sdk. deserializeAsset() is never called. _port.toString() is "SendPort".

    return call(_port, {
      'type': 'getInput',
      'id': serializeId(id)
    }).then(deserializeAsset);

[taisph]

Ok, I think I've nailed the bug now. I need to clean up all my added debug output and test some more then I'll make a PR.

[vicb]

@taisph thanks for your work on this issue! Any ETA for the PR, it would be nice to have it as part of 0.14

[taisph]

I've cleaned up my angular and dart-sdk changes I added in an attempt to follow the callflow. Then I reduced the changes I made to fix the problem to a minimum. When I think of all the logs I've gone through attempting to grasp the inner workings of Angular, Dart, transformers, isolates and what not, the now one-line fix just seems so... ridiculous.

Anyway, I can no longer reproduce the deadlock in any of my angular projects with the fix. I've run scripts/run-test.sh as well and there were no failures as far as I could tell. Maybe the deadlock situation could be added to some of these tests?

[bgourlie]

@taisph Very nice! What a subtle bug... Can you create a pull request so that the angular team can get it into v0.14? It looks like they're trying to release soon!

[taisph]

@bgourlie Done. My forked master had changes I didn't want in the PR so I just had to clean that up as well.

[bgourlie]

@taisph Awesome and much appreciated! I used to have similar issues when I'd clone a repo and work directly on master. Nowadays I keep master synced with upstream and created branches to work on specific issues, keeps things much cleaner.