Managed directories crash

[gregmagolan]

Description of the problem / feature request:

Bazel crash observed that looks related to managed directories when building bazel-schematics integration test which uses managed directories in angular repo while working on PR angular/angular#30627.

//cc @irengrig @alexeagle

Feature requests: what underlying problem are you trying to solve with this feature?

Crash output:

+ ng build
Internal error thrown during build. Printing stack trace: java.lang.IllegalStateException: existing entry for FILE_STATE:[/Users/greg/google/gregmagolan/angular/integration/bazel-schematics/demo]/[node_modules/lodash/toUpper.js] has deps: InMemoryNodeEntry{identity=1753141257, value=null, lastChangedVersion=IntVersion: 18, lastEvaluatedVersion=IntVersion: 38, directDeps=null, reverseDeps=ReverseDeps{reverseDeps=FILE:[/private/var/tmp/_bazel_greg/207489769571918cf75b329f2a9df974]/[external/npm/node_modules/lodash/toUpper.js], singleReverseDep=true, dataToConsolidate=null}, dirtyBuildingState=FullDirtyBuildingState{dirtyState=NEEDS_REBUILDING, signaledDeps=0, externalDeps=0, dirtyDirectDepIndex=0, lastBuildDirectDeps=GroupedList{elements=[REPOSITORY_DIRECTORY:@npm], size=1}, lastBuildValue=RegularFileStateValue{digest=null, size=592, contentsProxy=ctime of 1558673367194 and nodeId of 12316660772}}}
at com.google.common.base.Preconditions.checkState(Preconditions.java:823)
at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.injectValues(AbstractExceptionalParallelEvaluator.java:682)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.injectValues(InMemoryMemoizingEvaluator.java:268)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:180)
at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.evaluate(SkyframeExecutor.java:2863)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.loadTargetPatterns(SkyframeExecutor.java:2708)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.evaluateTargetPatterns(AnalysisPhaseRunner.java:156)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:77)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:145)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

INFO: Elapsed time: 0.738s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
Internal error thrown during build. Printing stack trace: java.lang.IllegalStateException: existing entry for FILE_STATE:[/Users/greg/google/gregmagolan/angular/integration/bazel-schematics/demo]/[node_modules/lodash/toUpper.js] has deps: InMemoryNodeEntry{identity=1753141257, value=null, lastChangedVersion=IntVersion: 18, lastEvaluatedVersion=IntVersion: 38, directDeps=null, reverseDeps=ReverseDeps{reverseDeps=FILE:[/private/var/tmp/_bazel_greg/207489769571918cf75b329f2a9df974]/[external/npm/node_modules/lodash/toUpper.js], singleReverseDep=true, dataToConsolidate=null}, dirtyBuildingState=FullDirtyBuildingState{dirtyState=NEEDS_REBUILDING, signaledDeps=0, externalDeps=0, dirtyDirectDepIndex=0, lastBuildDirectDeps=GroupedList{elements=[REPOSITORY_DIRECTORY:@npm], size=1}, lastBuildValue=RegularFileStateValue{digest=null, size=592, contentsProxy=ctime of 1558673367194 and nodeId of 12316660772}}}
at com.google.common.base.Preconditions.checkState(Preconditions.java:823)
at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.injectValues(AbstractExceptionalParallelEvaluator.java:682)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.injectValues(InMemoryMemoizingEvaluator.java:268)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:180)
at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.evaluate(SkyframeExecutor.java:2863)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.loadTargetPatterns(SkyframeExecutor.java:2708)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.evaluateTargetPatterns(AnalysisPhaseRunner.java:156)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:77)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:145)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
java.lang.IllegalStateException: existing entry for FILE_STATE:[/Users/greg/google/gregmagolan/angular/integration/bazel-schematics/demo]/[node_modules/lodash/toUpper.js] has deps: InMemoryNodeEntry{identity=1753141257, value=null, lastChangedVersion=IntVersion: 18, lastEvaluatedVersion=IntVersion: 38, directDeps=null, reverseDeps=ReverseDeps{reverseDeps=FILE:[/private/var/tmp/_bazel_greg/207489769571918cf75b329f2a9df974]/[external/npm/node_modules/lodash/toUpper.js], singleReverseDep=true, dataToConsolidate=null}, dirtyBuildingState=FullDirtyBuildingState{dirtyState=NEEDS_REBUILDING, signaledDeps=0, externalDeps=0, dirtyDirectDepIndex=0, lastBuildDirectDeps=GroupedList{elements=[REPOSITORY_DIRECTORY:@npm], size=1}, lastBuildValue=RegularFileStateValue{digest=null, size=592, contentsProxy=ctime of 1558673367194 and nodeId of 12316660772}}}
at com.google.common.base.Preconditions.checkState(Preconditions.java:823)
at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.injectValues(AbstractExceptionalParallelEvaluator.java:682)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.injectValues(InMemoryMemoizingEvaluator.java:268)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:180)
at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.evaluate(SkyframeExecutor.java:2863)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.loadTargetPatterns(SkyframeExecutor.java:2708)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.evaluateTargetPatterns(AnalysisPhaseRunner.java:156)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:77)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:145)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
java.lang.IllegalStateException: existing entry for FILE_STATE:[/Users/greg/google/gregmagolan/angular/integration/bazel-schematics/demo]/[node_modules/lodash/toUpper.js] has deps: InMemoryNodeEntry{identity=1753141257, value=null, lastChangedVersion=IntVersion: 18, lastEvaluatedVersion=IntVersion: 38, directDeps=null, reverseDeps=ReverseDeps{reverseDeps=FILE:[/private/var/tmp/_bazel_greg/207489769571918cf75b329f2a9df974]/[external/npm/node_modules/lodash/toUpper.js], singleReverseDep=true, dataToConsolidate=null}, dirtyBuildingState=FullDirtyBuildingState{dirtyState=NEEDS_REBUILDING, signaledDeps=0, externalDeps=0, dirtyDirectDepIndex=0, lastBuildDirectDeps=GroupedList{elements=[REPOSITORY_DIRECTORY:@npm], size=1}, lastBuildValue=RegularFileStateValue{digest=null, size=592, contentsProxy=ctime of 1558673367194 and nodeId of 12316660772}}}
at com.google.common.base.Preconditions.checkState(Preconditions.java:823)
at com.google.devtools.build.skyframe.AbstractExceptionalParallelEvaluator.injectValues(AbstractExceptionalParallelEvaluator.java:682)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.injectValues(InMemoryMemoizingEvaluator.java:268)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:180)
at com.google.devtools.build.skyframe.SequentialBuildDriver.evaluate(SequentialBuildDriver.java:38)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.evaluate(SkyframeExecutor.java:2863)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.loadTargetPatterns(SkyframeExecutor.java:2708)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.evaluateTargetPatterns(AnalysisPhaseRunner.java:156)
at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:77)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:145)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
FAILED: Build did NOT complete successfully (0 packages loaded)
/Users/greg/google/gregmagolan/angular/integration/bazel-schematics/demo/node_modules/@bazel/bazel-darwin_x64/bazel-0.26.0rc7-darwin-x86_64 failed with code 37.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

As this is a flake I only observed once, it could be difficult to reproduce. The steps to run the bazel-schematics integration test on a fresh angular/angular clone of the branch from angular/angular#30627 are:

echo "build --workspace_status_command=\"node ./tools/bazel_stamp_vars.js\"" >> ./.bazelrc
yarn install
./scripts/build-packages-dist.sh

cd integration/bazel-schematics
yarn install
./test.sh

• Optionally before test, modify integration/bazel-schematics/test.sh by changing ng build in testBazel() to ng build --leaveBazelFilesOnDisk and just comment out testNonBazel. The flag prevents angular-cli from deleted the Bazel files it puts on disk after the test is done & you don't want to run the testNonBazel as you only care about the bazel test.

What operating system are you running Bazel on?

OSX

What's the output of bazel info release?

bazel 0.26.0-rc13

[gregmagolan]

Possible solution to try suggested by @irengrig is to add /integration/bazel-schematics/demo to /.bazelignore so that the root bazel worker doesn't read into that directory during the test (which is a nested WORKSPACE). I added that to angular/angular#30627 and running the test in a loop locally to try to induce the crash again.

Update: ran the test around 50 times without the /.bazelignore change and 50 around 50 times with the change on a loop locally without hitting the crash. This makes this flake pretty rare and there is nothing else to go on besides the call stack above.

[irengrig]

Closing as not reproduced any more (as rules_nodejs are using another model of external repository with managed directories); and as I am not planning to work on this.