output: Don't serialize `hash_info` or `meta` in cloud versioning. #8849

daavoo · 2023-01-20T10:18:16Z

Here is how the .dvc file looks for 2.42.0 / this P.R at different steps of the workflow (click details to expand):

Importing with `--version-aware`

$ dvc import-url --version-aware $VERSIONED_BUCKET/data
$ cat data.dvc

2.42.0

#8849

md5: 677db99135cf82ed1b726b6be7abef49
frozen: true
deps:
- md5: a042035f7fb17fc84a42e33787ae4b6a.dir
  size: 13
  nfiles: 3
  path: s3://diglesia-bucket/data
  files:
  - size: 5
    version_id: dQymcW0EvEqy23qfrksrqo0ni2zGtPMG
    etag: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: pd67.5YcvjqDm7jfmiHbpeQsXnprrXtf
    etag: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo
  - size: 4
    version_id: 2ruNQaorqjnsFQZ5cinHsZSHEw7cz26J
    etag: 7361528e901ca2f2f1952a68ad242d79
    relpath: goo
outs:
- md5: b5c587798f2c0f599ef935244b290bf1.dir
  size: 13
  nfiles: 3
  path: data
  push: false

md5: f642080c781ca41a33a3e6bf77959af5
frozen: true
deps:
- files:
  - size: 5
    version_id: dQymcW0EvEqy23qfrksrqo0ni2zGtPMG
    etag: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: pd67.5YcvjqDm7jfmiHbpeQsXnprrXtf
    etag: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo
  - size: 4
    version_id: 2ruNQaorqjnsFQZ5cinHsZSHEw7cz26J
    etag: 7361528e901ca2f2f1952a68ad242d79
    relpath: goo
  path: s3://diglesia-bucket/data
outs:
- md5: b5c587798f2c0f599ef935244b290bf1.dir
  size: 13
  nfiles: 3
  path: data
  push: false

Tracking local folder with `dvc add` and running the first `dvc push`

$ mkdir data
$ echo bar > data/bar
$ echo foo > data/foo
$ dvc add data
$ dvc push -v
$ cat data.dvc

2.42.0

#8849

outs:
- md5: 168fd6761b9c3f2c081085b51fd7bf15.dir
  size: 8
  nfiles: 2
  path: data
  files:
  - size: 4
    version_id: iGoUk9Wlb95nNOw6iPHUrnNWq9E898YO
    etag: c157a79031e1c40f85931829bc5fc552
    md5: c157a79031e1c40f85931829bc5fc552
    relpath: bar
  - size: 4
    version_id: DwITkoYE8ipINKzK2QcGKvlwlqaS4oER
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo

outs:
- path: data
  files:
  - size: 4
    version_id: DhE9omrAq7IXC5wGIIGqruq9JyP_Nwhm
    etag: c157a79031e1c40f85931829bc5fc552
    md5: c157a79031e1c40f85931829bc5fc552
    relpath: bar
  - size: 4
    version_id: Vnu6l9_ECdLGrp6B5iK6YmQTm87UaDzP
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo

Tracking a local update with `dvc add` on that folder

$ echo bar2 > data/bar
$ dvc add data
$ cat data.dvc

2.42.0

#8849

outs:
- md5: 43f06a6517d71d0267bed23cfb1a53fb.dir
  size: 9
  nfiles: 2
  path: data
  files:
  - size: 5
    md5: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: DwITkoYE8ipINKzK2QcGKvlwlqaS4oER
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo

outs:
- path: data
  files:
  - size: 5
    md5: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: Vnu6l9_ECdLGrp6B5iK6YmQTm87UaDzP
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo

After pushing the local update with `dvc push`

$ dvc push 
$ cat data.dvc

2.42.0

#8849

outs:
- md5: 43f06a6517d71d0267bed23cfb1a53fb.dir
  size: 9
  nfiles: 2
  path: data
  files:
  - size: 5
    md5: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
    version_id: QLNufQhPEwlSlNSfauemR5D23RRMSbfF
    etag: 042967ff088f7f436015fbebdcad1f28
  - size: 4
    version_id: DwITkoYE8ipINKzK2QcGKvlwlqaS4oER
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo

outs:
- path: data
  files:
  - size: 5
    md5: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: Vnu6l9_ECdLGrp6B5iK6YmQTm87UaDzP
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo

Tracking a modification in the remote with `dvc update`

$ echo goo > goo
$ aws s3 cp goo $REMOTE/data/goo
$ dvc update data.dvc -v
$ cat data.dvc

2.42.0

#8849

outs:
- md5: b5c587798f2c0f599ef935244b290bf1.dir
  size: 13
  nfiles: 3
  path: data
  files:
  - size: 5
    version_id: QLNufQhPEwlSlNSfauemR5D23RRMSbfF
    etag: 042967ff088f7f436015fbebdcad1f28
    md5: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: DwITkoYE8ipINKzK2QcGKvlwlqaS4oER
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo
  - size: 4
    version_id: xktZ1CcLphMaZheT_O9QrxFpmd9GrgK1
    etag: 7361528e901ca2f2f1952a68ad242d79
    md5: 7361528e901ca2f2f1952a68ad242d79
    relpath: goo

outs:
- path: data
  files:
  - size: 5
    version_id: dQymcW0EvEqy23qfrksrqo0ni2zGtPMG
    etag: 042967ff088f7f436015fbebdcad1f28
    md5: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: pd67.5YcvjqDm7jfmiHbpeQsXnprrXtf
    etag: d3b07384d113edec49eaa6238ad5ff00
    md5: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo
  - size: 4
    version_id: 2ruNQaorqjnsFQZ5cinHsZSHEw7cz26J
    etag: 7361528e901ca2f2f1952a68ad242d79
    md5: 7361528e901ca2f2f1952a68ad242d79
    relpath: goo

daavoo · 2023-01-20T10:18:56Z

dvc/output.py

+            tree = Tree.from_list(self.files, hash_name=self.hash_name)
+            tree.digest()
+            self.hash_info = tree.hash_info


This introduces overhead but need to check how big it is

codecov · 2023-01-20T10:34:22Z

Codecov Report

Base: 93.64% // Head: 93.70% // Increases project coverage by +0.05% 🎉

Coverage data is based on head (9019208) compared to base (61afdca).
Patch coverage: 98.21% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8849      +/-   ##
==========================================
+ Coverage   93.64%   93.70%   +0.05%     
==========================================
  Files         453      453              
  Lines       36101    36146      +45     
  Branches     5237     5239       +2     
==========================================
+ Hits        33808    33871      +63     
+ Misses       1787     1765      -22     
- Partials      506      510       +4

Impacted Files	Coverage Δ
dvc/output.py	`90.26% <94.11%> (+2.84%)`	⬆️
dvc/stage/serialize.py	`94.91% <100.00%> (+2.60%)`	⬆️
tests/func/test_add.py	`98.72% <100.00%> (+0.01%)`	⬆️
tests/unit/output/test_output.py	`100.00% <100.00%> (ø)`
tests/unit/stage/test_serialize_pipeline_lock.py	`100.00% <100.00%> (ø)`
dvc/repo/experiments/queue/celery.py	`86.29% <0.00%> (-1.86%)`	⬇️
dvc/proc/manager.py	`75.26% <0.00%> (-1.08%)`	⬇️
dvc/stage/__init__.py	`94.70% <0.00%> (+0.63%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

dberenbaum · 2023-01-20T21:50:30Z

LGTM 🎉 ! Nice work @daavoo!

Closes #8357

mergify · 2023-01-24T17:11:53Z

⚠️ The sha of the head commit of this PR conflicts with #8829. Mergify cannot evaluate rules on this PR. ⚠️

skshetry · 2023-01-25T11:23:37Z

dvc/output.py

+        if (
+            (not self.IS_DEPENDENCY or self.stage.is_import)
+            and self.hash_info.isdir
+            and (kwargs.get("with_files") or self.files is not None)
+        ):


Any reason to move this upward @daavoo? This might affect the ordering of metadata in .dvc file. We probably want files to appear at the end.

No real reason. Could be moved bellow

Created #8879.

daavoo linked an issue Jan 20, 2023 that may be closed by this pull request

cloud versioning: get rid of .dir files #8357

Closed

daavoo commented Jan 20, 2023

View reviewed changes

daavoo force-pushed the 8357-cloud-versioning-get-rid-of-dir-files branch from 34045d5 to 291418f Compare January 20, 2023 10:21

daavoo force-pushed the 8357-cloud-versioning-get-rid-of-dir-files branch 2 times, most recently from 74f5cc6 to 2138f0c Compare January 20, 2023 14:57

daavoo marked this pull request as ready for review January 20, 2023 15:17

daavoo requested review from a team and dberenbaum January 20, 2023 15:17

daavoo self-assigned this Jan 23, 2023

dberenbaum approved these changes Jan 23, 2023

View reviewed changes

daavoo added A: cloud-versioning Related to cloud-versioned remotes enhancement Enhances DVC labels Jan 24, 2023

daavoo enabled auto-merge (rebase) January 24, 2023 14:29

daavoo disabled auto-merge January 24, 2023 14:29

daavoo enabled auto-merge (rebase) January 24, 2023 14:29

daavoo force-pushed the 8357-cloud-versioning-get-rid-of-dir-files branch from 2138f0c to 28a80df Compare January 24, 2023 14:29

dberenbaum mentioned this pull request Jan 24, 2023

cloud versioning: dvc pull fails after version aware import-url for non version aware s3 remote #8872

Closed

daavoo removed the request for review from a team January 24, 2023 16:56

output: Don't serialize hash_info or meta in cloud versioning.

9019208

Closes #8357

daavoo force-pushed the 8357-cloud-versioning-get-rid-of-dir-files branch from 28a80df to 9019208 Compare January 24, 2023 16:57

daavoo merged commit ea2e5dc into main Jan 24, 2023

daavoo deleted the 8357-cloud-versioning-get-rid-of-dir-files branch January 24, 2023 17:40

skshetry reviewed Jan 25, 2023

View reviewed changes

dberenbaum mentioned this pull request Jan 25, 2023

Dropping .dir #8884

Closed

karajan1001 mentioned this pull request Feb 2, 2023

import-url/import: follow ups for cloud versioning #8464

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

output: Don't serialize `hash_info` or `meta` in cloud versioning. #8849

output: Don't serialize `hash_info` or `meta` in cloud versioning. #8849

daavoo commented Jan 20, 2023 •

edited

Loading

daavoo Jan 20, 2023

codecov bot commented Jan 20, 2023 •

edited

Loading

dberenbaum commented Jan 20, 2023

mergify bot commented Jan 24, 2023

skshetry Jan 25, 2023 •

edited

Loading

daavoo Jan 25, 2023

skshetry Jan 25, 2023

output: Don't serialize hash_info or meta in cloud versioning. #8849

output: Don't serialize hash_info or meta in cloud versioning. #8849

Conversation

daavoo commented Jan 20, 2023 • edited Loading

Importing with --version-aware

Tracking local folder with dvc add and running the first dvc push

Tracking a local update with dvc add on that folder

After pushing the local update with dvc push

Tracking a modification in the remote with dvc update

daavoo Jan 20, 2023

Choose a reason for hiding this comment

codecov bot commented Jan 20, 2023 • edited Loading

Codecov Report

dberenbaum commented Jan 20, 2023

mergify bot commented Jan 24, 2023

skshetry Jan 25, 2023 • edited Loading

Choose a reason for hiding this comment

daavoo Jan 25, 2023

Choose a reason for hiding this comment

skshetry Jan 25, 2023

Choose a reason for hiding this comment

output: Don't serialize `hash_info` or `meta` in cloud versioning. #8849

output: Don't serialize `hash_info` or `meta` in cloud versioning. #8849

daavoo commented Jan 20, 2023 •

edited

Loading

Importing with `--version-aware`

Tracking local folder with `dvc add` and running the first `dvc push`

Tracking a local update with `dvc add` on that folder

After pushing the local update with `dvc push`

Tracking a modification in the remote with `dvc update`

codecov bot commented Jan 20, 2023 •

edited

Loading

skshetry Jan 25, 2023 •

edited

Loading