Unexpected IO error (Not a directory) #4751
Comments
|
Is anyone investigating? |
|
I just tried to collect evidence that this is an instance of existing issue, but I cannot find any. But I vaguely remember strange issues with skylark repositories on mac like this one. |
|
Thanks. Marcel, may I assign this to you or is there someone for whom it's a better fit? |
|
Probably the same error is on re2 project in this build: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/174#76e869fc-616e-4893-a2d6-f86403c64dc0 |
|
failed: Unexpected IO error.: /private/var/tmp/_bazel_buildkite/5325b99c04dd286d6e24d5b36b141d80/execroot/com_googlesource_code_re2/external/bazel_tools/tools/test/test-setup.sh (No such file or directory) |
|
I have managed to get a stacktrace for the error: After Bazel crashes, the file in question exists however. The error is hard to reproduce and I have only ever managed to reproduce it when the machine was under load. The error only seems to happen for files from remote repositories. |
|
Some more instances of the error: https://buildkite.com/bazel/publish-bazel-binaries/builds/50#fc59eefc-b8d3-4c53-9854-5df86d2061d1 https://buildkite.com/bazel/bazel-bazel/builds/1058#a5d34e61-0b0e-47f8-a6cf-ddbb4c774628 |
|
The code that triggers this error is with cc: @ulfjack |
|
It's also quite possible that this is a bug in |
|
So far, I think we've only seen this error for executable files, right? I wonder if it's something in MacOS, maybe related to its verification of executable files? |
|
@ulfjack IIUC all output files of an action are marked executable in Bazel? |
|
None of the paths posted so far contain "bazel-out". I'm not sure how exactly the files under external/ come into existence - the paths are related to external repositories, but that doesn't really help us narrow it down. It could be coincidence that we only see it happen for external repositories paths. The remote cache is implicitly sorting the files by name, so maybe it's just the external/ paths always get sorted first or something like that. |
|
I have disabled the |
|
@ulfjack any ideas on how to proceed? |
|
The next thing to do is rule out the likely answers. Does the file exist when we do the call? |
|
Also, if we're writing the file, are we properly closing it afterwards? |
|
I added the following code before constructing the for (ActionInput input : inputMap.values()) {
Path p = execRoot.getRelative(input.getExecPathString());
try {
p.isExecutable();
} catch (IOException e) {
File f = new File(p.getPathString());
report(Event.debug("stacktrace: " + Throwables
.getStackTraceAsString(e) + ", exists: " + f.isFile()));
}
}According to |
|
The following patch makes the error go away, which makes me conclude that it's a problem with both our diff --git a/src/main/java/com/google/devtools/build/lib/remote/TreeNodeRepository.java b/src/main/java/com/google/devtools/build/lib/remote/TreeNodeRepository.java
index 7767cb8c7b..c1a4f34817 100644
--- a/src/main/java/com/google/devtools/build/lib/remote/TreeNodeRepository.java
+++ b/src/main/java/com/google/devtools/build/lib/remote/TreeNodeRepository.java
@@ -39,6 +39,7 @@ import com.google.devtools.build.lib.vfs.Symlinks;
import com.google.devtools.remoteexecution.v1test.Digest;
import com.google.devtools.remoteexecution.v1test.Directory;
import com.google.protobuf.ByteString;
+import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
@@ -372,7 +373,7 @@ public final class TreeNodeRepository {
b.addFilesBuilder()
.setName(entry.getSegment())
.setDigest(DigestUtil.getFromInputCache(input, inputFileCache))
- .setIsExecutable(execRoot.getRelative(input.getExecPathString()).isExecutable());
+ .setIsExecutable(new File(execRoot.getRelative(input.getExecPathString()).getPathString()).canExecute());
}
} else {
Digest childDigest = Preconditions.checkNotNull(treeNodeDigestCache.get(child)); |
buchgr
added
P3
buchgr
added
team-Remote-Exec
|
I have finally found the problem and solution to this bug.The issue is that we always create The solution is to create the path object for an I ll send out a fix. |
|
Wow! Thanks!! 😀 I’m curious: Why does it only sometimes fail though (and mostly on macOS)? Is there a race somewhere that lets this work most of the time even though the path is wrong? |
|
Wow, that's quite impressive. How did you find this bug? Also, it seems to be easy to do the wrong thing and hard to do the right thing, i.e. I would never guess that artifact.getRoot() then another getRoot() and finally getRelative() is the way to go. Do you have any suggestion about simplifying the API or somehow making it easy to create Artifacts the right way? |
buchgr
added
P0
The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
When build without bytes is enabled, we use isExecutable field of OutputFile for intermediate input files. This is achieved by injecting the metadata into the MetadataProvider. The "always mark" was introduced by 3e3b71a which was a workaround for bazelbuild#4751. However, that issue was then fixed by fc44891. There is no reason to keep the workaround which is causing other issues e.g. bazelbuild#12818.
We see this error frequently on macOS. @mhlopko suggested that it's a somewhat known bug in external repositories.
cc: @aehlig @dslomov
The text was updated successfully, but these errors were encountered: