dvcfs: detach from pipeline outputs #7353
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introducing simple repo tree that combines all workspaces into one structure, and allows us to detach data representation from pipelines. Pre-requisite for dvc-data extraction and using relpaths in dvcfs/repofs.
efiop
changed the title
efiop
changed the title
efiop
commented
Feb 7, 2022
Comment on lines
+186
to
+190
| shortest = self.repo.index.tree.shortest_prefix(key) | ||
| if shortest: | ||
| assert shortest[1][1].isdir | ||
| if len(shortest[0]) <= len(key): | ||
| ret["isdvc"] = True |
There was a problem hiding this comment.
These odd flags and logic will be removed in a followup, when we just use fs and dvcfs separately (with subrepos), instead of passing through tons of odd flags used inappropriately in ls and diff.
efiop
added
refactoring
karajan1001
reviewed
Feb 8, 2022
| @@ -168,7 +168,7 @@ def _build_tree(fs_path, fs, name, **kwargs): | |||
| assert key | |||
| tree.add(key, meta, obj.hash_info) | |||
|
|
|||
| tree_meta.size += meta.size | |||
| tree_meta.size += meta.size or 0 | |||
There was a problem hiding this comment.
Excuse me, why can not just set them default 0? Like in C or Java current None is just like null point in other languages.
There was a problem hiding this comment.
@karajan1001 This is a bit messy, but we have meta set to None for files inside of dvc add dir dirs. There seems to be a value in separating 0 and "not set" state, but you are right, we'll probably set it to defaults in the future.
karajan1001
reviewed
Feb 8, 2022
| if not odb: | ||
| continue | ||
|
|
||
| try: |
There was a problem hiding this comment.
Can reduce one line.
with suppress(FileNotFoundError, ObjectFormatError):
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing simple repo tree that combines all workspaces into one structure,
and allows us to detach data representation from pipelines.
Pre-requisite for dvc-data extraction and using relpaths in dvcfs/repofs.