Action input symlink into directory doesn't work with --remote_download_minimal #15678
Labels
P2
We'll consider working on this in future. (Assignee optional)
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
type: bug
Comments
sgowroji
added
type: bug
untriaged
team-Remote-Exec
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Jun 15, 2022
The intended semantics of a symlink as an action input is that only the symlink itself is an input, not the file/directory it points to. However, due to sandboxing limitations, such an action still executes successfully locally, though not remotely. Note that this change is necessary but not sufficient to make these actions compatible with remote execution; we also need to fix bazelbuild#15678.
19 tasks
copybara-service bot
pushed a commit
that referenced
this issue
Jul 25, 2022
…a file, when the input is a directory. Note that SymlinkAction#toArtifact doesn't currently work when the output is a directory, as it will attempt to create a symlink over it. However, some callers cheat by passing in a non-directory output, which works fine as far as SymlinkAction is concerned, but causes subtle issues elsewhere. In the future, SymlinkAction#toArtifact should itself enforce this precondition, but `ctx.actions.symlink` must be fixed first (in a separate CL). This is a necessary step to fix issue #15678. PiperOrigin-RevId: 463049372 Change-Id: I2d072a17a18cf26bc64b5639078fc2954fc3801b
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 15, 2022
…artifact. A `ctx.actions.symlink` whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to a directory that was built remotely and without the bytes, we currently must download the directory contents before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting the input metadata into the output, thus avoiding the download. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that results in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 15, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to a directory that was built remotely and without the bytes, we currently must download the directory contents before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting the input metadata into the output, thus avoiding the download. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that results in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 15, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to a directory that was built remotely and without the bytes, we currently must download the directory contents before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting the input metadata into the output, thus avoiding the download. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that results in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 19, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to a directory that was built remotely and without the bytes, we currently must download the directory contents before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting the input metadata into the output, thus avoiding the download. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that results in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 21, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 21, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 26, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 27, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 28, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 28, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 28, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 28, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 28, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 29, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 29, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 29, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 29, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 29, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Sep 29, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 13, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 18, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 18, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 19, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 19, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to a directory that was built remotely and without the bytes, we currently must download the directory contents before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting the input metadata into the output, thus avoiding the download. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that results in an awful lot of complexity.) Fixes bazelbuild#15678.
tjgq
added a commit
to tjgq/bazel
that referenced
this issue
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description of the bug:
Actions whose inputs contain a symlink into a directory (tree artifact) fail to build remotely with
--remote_download_minimal.What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Minimal repro at https://github.com/tjgq/bazel-symlink-into-dir.
Some notes:
--remote_download_minimal.--disk_cache.Which operating system are you running Bazel on?
Linux
What is the output of
bazel info release?development version
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.Bazel built near head @ d7eaa0b. However, issue also exists in 5.2.0 (i.e., it's not a regression).
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD?Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
The text was updated successfully, but these errors were encountered: