-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inject metadata when creating a filesystem symlink for a non-symlink artifact. #16283
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tjgq
force-pushed
the
symlink-to-dir
branch
2 times, most recently
from
September 15, 2022 13:18
06c9399
to
9bdf1c1
Compare
coeuvre
reviewed
Sep 15, 2022
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Show resolved
Hide resolved
tjgq
force-pushed
the
symlink-to-dir
branch
from
September 15, 2022 15:57
9bdf1c1
to
ae3b0f6
Compare
ShreeM01
added
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
awaiting-user-response
Awaiting a response from the author
labels
Sep 16, 2022
coeuvre
reviewed
Sep 16, 2022
tjgq
force-pushed
the
symlink-to-dir
branch
4 times, most recently
from
September 21, 2022 08:46
2bcd265
to
1396247
Compare
tjgq
changed the title
Inject metadata when calling RemoteActionFileSystem#createSymbolicLink for a non-symlink artifact.
Inject metadata when creating a filesystem symlink for a non-symlink artifact.
Sep 22, 2022
sgowroji
added
awaiting-review
PR is awaiting review from an assigned reviewer
and removed
awaiting-user-response
Awaiting a response from the author
labels
Sep 26, 2022
tjgq
force-pushed
the
symlink-to-dir
branch
from
September 26, 2022 19:06
1396247
to
d476b26
Compare
alexjski
reviewed
Sep 27, 2022
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/AbstractActionInputPrefetcher.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/skyframe/TreeArtifactValue.java
Outdated
Show resolved
Hide resolved
tjgq
force-pushed
the
symlink-to-dir
branch
5 times, most recently
from
September 28, 2022 23:27
de0baf8
to
7f88ca5
Compare
copybara-service bot
pushed a commit
that referenced
this pull request
Sep 29, 2022
This makes it easier to later return different specializations depending on which fields are present, which will be required by #16283. PiperOrigin-RevId: 477674504 Change-Id: Ieec385e923ab4ce0546f8853811df18316a8d884
tjgq
force-pushed
the
symlink-to-dir
branch
2 times, most recently
from
September 29, 2022 10:10
a4d53a7
to
c7bb56e
Compare
coeuvre
reviewed
Sep 29, 2022
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/cache/CompactPersistentActionCache.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/skyframe/TreeArtifactValue.java
Outdated
Show resolved
Hide resolved
src/test/java/com/google/devtools/build/lib/actions/ActionCacheCheckerTest.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/cache/CompactPersistentActionCache.java
Outdated
Show resolved
Hide resolved
tjgq
force-pushed
the
symlink-to-dir
branch
from
September 29, 2022 13:24
c7bb56e
to
5f0a873
Compare
coeuvre
reviewed
Sep 29, 2022
src/test/java/com/google/devtools/build/lib/remote/BuildWithoutTheBytesIntegrationTest.java
Show resolved
Hide resolved
tjgq
force-pushed
the
symlink-to-dir
branch
3 times, most recently
from
September 29, 2022 19:22
4d55045
to
9f84f7e
Compare
aiuto
pushed a commit
to aiuto/bazel
that referenced
this pull request
Oct 12, 2022
This makes it easier to later return different specializations depending on which fields are present, which will be required by bazelbuild#16283. PiperOrigin-RevId: 477674504 Change-Id: Ieec385e923ab4ce0546f8853811df18316a8d884
alexjski
suggested changes
Oct 17, 2022
src/main/java/com/google/devtools/build/lib/actions/ActionInputMap.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
Outdated
Show resolved
Hide resolved
src/test/java/com/google/devtools/build/lib/actions/ActionCacheCheckerTest.java
Outdated
Show resolved
Hide resolved
src/test/java/com/google/devtools/build/lib/actions/ActionCacheCheckerTest.java
Outdated
Show resolved
Hide resolved
src/test/java/com/google/devtools/build/lib/actions/ActionInputMapTest.java
Outdated
Show resolved
Hide resolved
src/test/java/com/google/devtools/build/lib/actions/ActionInputMapTest.java
Outdated
Show resolved
Hide resolved
src/test/java/com/google/devtools/build/lib/actions/ActionInputMapTest.java
Outdated
Show resolved
Hide resolved
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 18, 2022
In #16283, I will add a new data member to this class, which won't be shared with other uses of FileArtifactValue#createProxy. This is a preliminary no-op CL to make the subsequent change simpler. As noted in the PR discussion, the getType() method on the proxy object currently returns REGULAR_FILE instead of DIRECTORY. I intend to fix this in a separate CL since it's unclear whether any callers rely on this. PiperOrigin-RevId: 481913076 Change-Id: I44f721f10230ec782f7a583cdf372972376959c0
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 18, 2022
This will be needed by #16283. PiperOrigin-RevId: 481921235 Change-Id: I619bdbee25b7465ce2e6ad3e384e1ce414d2e770
tjgq
force-pushed
the
symlink-to-dir
branch
2 times, most recently
from
October 18, 2022 15:40
f09c1dd
to
ba654cc
Compare
coeuvre
reviewed
Oct 19, 2022
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
tjgq
force-pushed
the
symlink-to-dir
branch
4 times, most recently
from
October 20, 2022 08:35
e2217af
to
7e31a48
Compare
coeuvre
reviewed
Oct 20, 2022
src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java
Outdated
Show resolved
Hide resolved
coeuvre
approved these changes
Oct 20, 2022
…artifact. A ctx.actions.symlink whose output is a declare_file or declare_directory (as opposed to a declare_symlink) has "copy" semantics, i.e., the output artifact is indistinguishable from the referent except for its name; the symlink is just a filesystem-level optimization to avoid an actual copy, and is transparently resolved when collecting the action output metadata. When the symlink points to an artifact that was built remotely and without the bytes, we currently must download it before executing the symlink action, so that the output metadata can be constructed from the local filesystem. This change short-circuits the filesystem traversal by injecting output metadata, which is identical to the input plus a pointer to the original path. This is used by the prefetcher to avoid downloading the same files multiple times when they're symlinked more than once. (An alternative would have been to teach all of the RemoteActionFileSystem methods to resolve symlinks by patching together the local and remote metadata, but that would have resulted in an awful lot of complexity.) Fixes bazelbuild#15678.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
awaiting-review
PR is awaiting review from an assigned reviewer
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A ctx.actions.symlink whose output is a declare_file or declare_directory
(as opposed to a declare_symlink) has "copy" semantics, i.e., the output
artifact is indistinguishable from the referent except for its name; the
symlink is just a filesystem-level optimization to avoid an actual copy,
and is transparently resolved when collecting the action output metadata.
When the symlink points to an artifact that was built remotely and without the
bytes, we currently must download it before executing the symlink action, so
that the output metadata can be constructed from the local filesystem. This
change short-circuits the filesystem traversal by injecting output metadata,
which is identical to the input plus a pointer to the original path. This is
used by the prefetcher to avoid downloading the same files multiple times when
they're symlinked more than once.
(An alternative would have been to teach all of the RemoteActionFileSystem
methods to resolve symlinks by patching together the local and remote metadata,
but that would have resulted in an awful lot of complexity.)
Fixes #15678.