Unexpected IO Error (Not a directory) on macOS

[RNabel]

Description of the problem / feature request:

Bazel server crashing with IOException when building a descriptor set with --remote_download_outputs=toplevel et al. enabled.

Feature requests: what underlying problem are you trying to solve with this feature?

IOException crash.

Stack trace:

Couldn't build file <target>-descriptor-set.proto.bin: Generating Descriptor Set proto_library <target> failed due to unexpected I/O exception: /private/var/tmp/_bazel_<user>/b6b989c6d3681c6cb6a4a268dfbe97f5/execroot/__main__/bazel-out/<cpu>/bin/<package>/_virtual_imports/<target name>/<relative .proto path with stripped prefix> (Not a directory)
--
  | java.io.IOException: /private/var/tmp/_bazel_<user>/b6b989c6d3681c6cb6a4a268dfbe97f5/execroot/__main__/bazel-out/<cpu>/bin/<package>/_virtual_imports/<target name>/<relative .proto path with stripped prefix> (Not a directory)
  | at com.google.devtools.build.lib.unix.NativePosixFiles.readdir(Native Method)
  | at com.google.devtools.build.lib.unix.NativePosixFiles.readdir(NativePosixFiles.java:283)
  | at com.google.devtools.build.lib.unix.UnixFileSystem.readdir(UnixFileSystem.java:160)
  | at com.google.devtools.build.lib.vfs.Path.readdir(Path.java:384)
  | at com.google.devtools.build.lib.remote.merkletree.DirectoryTreeBuilder.explodeDirectory(DirectoryTreeBuilder.java:135)
  | at com.google.devtools.build.lib.remote.merkletree.DirectoryTreeBuilder.explodeDirectory(DirectoryTreeBuilder.java:128)
  | at com.google.devtools.build.lib.remote.merkletree.DirectoryTreeBuilder.fromActionInputs(DirectoryTreeBuilder.java:99)
  | at com.google.devtools.build.lib.remote.merkletree.DirectoryTreeBuilder.fromActionInputs(DirectoryTreeBuilder.java:47)
  | at com.google.devtools.build.lib.remote.merkletree.MerkleTree.build(MerkleTree.java:123)
  | at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:130)
  | at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:117)
  | at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:87)
  | at com.google.devtools.build.lib.actions.SpawnActionContext.beginExecution(SpawnActionContext.java:41)
  | at com.google.devtools.build.lib.exec.ProxySpawnActionContext.beginExecution(ProxySpawnActionContext.java:60)
  | at com.google.devtools.build.lib.analysis.actions.SpawnAction.beginExecution(SpawnAction.java:332)
  | at com.google.devtools.build.lib.actions.Action.execute(Action.java:123)
  | at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:905)
  | at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1040)
  | at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1011)
  | at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:116)
  | at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:77)
  | at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:589)
  | at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:902)
  | at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:292)
  | at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:454)
  | at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  | at java.base/java.lang.Thread.run(Unknown Source)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This happens flakily.

Flags: remote cache + --remote_download_outputs=toplevel --experimental_inmemory_jdeps_files --experimental_inmemory_dotd_files --nobuild_runfile_links

The proto_library target has direct and transitive dependencies which all use the strip_prefix parameter.

What operating system are you running Bazel on?

macOS 10.14.6

What's the output of bazel info release?

release 2.0.0

Have you found anything relevant by searching the web?

Likely related: #4751, but the stack trace looks very different.

[tetromino]

Sanity check: in your stack trace, are <target>, <user>, <cpu>, <package>, <target name>, and <relative .proto path with stripped prefix> all placeholders to hide internal code; the angle brackets etc. are not literally in the stack trace, right?

[RNabel]

@tetromino Correct, those are placeholders only and there are no angled brackets in the stack trace.

[coeuvre]

I think this is fixed with recent development. Free feel to reopen if it sill happens to you on latest version.