import-url/import: follow ups for cloud versioning

[dberenbaum]

• For dvc update we can probably get around downloading when it hasn't changed, but that's a separate feature request I think (import-url: use dvc-data index.save() for fetching imports #8249 (comment)_)
  - [ ] For chained import-url imports I think it will not work right now unless the file has been pushed to a DVC remote, but when we get to refactoring regular cached objects to use dvc-data index for push/pull (so everything is unified to use the dvc-data index) this kind of chaining (from original source location) should work out of the box. (import-url: use dvc-data index.save() for fetching imports #8249 (comment)_) Moved to cloud versioning: imports #8789
  - [ ] Add support to import/get from another repo where that repo's remote is cloud-versioned. Moved to cloud versioning: imports #8789
• Don't push dvc import data to remote when remote is configured with worktree = true. (worktree: support push: false #8581)
• With cloud versioning enabled, DVC always returns that it has pushed all files, even for files that aren't pushed (including files that have already been pushed and files with push: false). (worktree push: do not push existing versions #8606)

[dberenbaum]

• For dvc update we can probably get around downloading when it hasn't changed

@pmrowla Thoughts on this? This would be nice to have for initial release, since redownloading on every update seems unrealistic in any real scenario.

[dberenbaum]

@karajan1001 @pmrowla There was a regression that was making update download unchanged datasets even for regular remotes that was fixed in #8752. For cloud-versioned remotes, update still downloads unchanged datasets even after that fix.

[dberenbaum]

We should also check that data added to a worktree remote doesn't redownload on dvc update if nothing changed.

[dberenbaum]

For update after import-url without cloud versioning, here's what I see:

import-url-update.mov

For update after import-url --version-aware, I see that everything gets downloaded again:

import-url-version-aware-update.mov

For a non-imported dataset pushed to a worktree remote, it looks like nothing gets downloaded, but it would be nice to have this be more explicit and match the UI of the first video:

worktree-add-push-update.mov

[dberenbaum]

@karajan1001 Any updates on this?

[karajan1001]

@karajan1001 Any updates on this?

I'm sorry, I'm still debugging what's wrong with my s3 setup. Maybe I should switch to azure or gcp.

[karajan1001]

@dberenbaum, @daavoo A new situation. I found that the problem of duplicated downloading had already been solved in #8849, but after #8882 it became terribly slow (takes several minutes to finish). Maybe I should make the performance problem the next work, but better to open a new PR for it.

And for

For a non-imported dataset pushed to a worktree remote, it looks like nothing gets downloaded, but it would be nice to have this be more explicit and match the UI of the first video:

It still exists.

[daavoo]

but after #8882 it became terribly slow (takes several minutes to finish).

@karajan1001 not sure what is it in the phrase. dvc update after dvc import-url --version-aware ?

Could you share the profile from:

pip install viztracer
dvc update --viztracer --viztracer-depth 12

[karajan1001]

@karajan1001 not sure what is it in the phrase. dvc update after dvc import-url --version-aware ?

Yes, as well as import itself, but maybe it is caused by the network problem on my side,

viztracer.dvc-20230202_225242.json.tar.gz

looks like the time is mostly spent on waiting for responses from the s3 server. And the output is

dvc update --viztracer --viztracer-depth 12 version                                                                                  [ins][23:08:47]
'version.dvc' didn't change, skipping
Loading finish
Total Entries: 1000005

Circular buffer is full, you lost some early data, but you still have the most recent data.
    If you need more buffer, use "viztracer --tracer_entries <entry_number>"
    Or, you can try the filter options to filter out some data you don't need
    use --quiet to shut me up

Use the following command to open the report:
vizviewer /Users/gao/test/cloud_versioning/viztracer.dvc-20230202_231153.json

[dberenbaum]

@karajan1001 I'm not seeing a major performance difference, but could you and @daavoo debug and make sure it's not a problem? We can close the issue once that's addressed.

[karajan1001]

A big difference is that in the new version of dvc-data's get method, we have find in it.

I found today, the downloading problem appears again even in the latest version, looks like #8849 only partially solved it. Need a more detailed investigation.

[pmrowla]

The update after import-url --version-aware downloading files when nothing has changed issue should be fixed in main (underlying cause was also related to the import-url --no-download --version-aware bug)

[karajan1001]

I can confirm that the redownloading problem had been solved. But the performance problem still exists.

[dberenbaum]

@karajan1001 Are there still performance issues to investigate? Otherwise, let's close this issue.

[karajan1001]

@karajan1001 Are there still performance issues to investigate? Otherwise, let's close this issue.

The performance down was because, in the previous version, we didn't consider the s3 fs as a version_aware fs, and after #8882 this bug was fixed, but we still have performance problems on a version_aware S3FileSystem, But as it only affects users under particular network conditions. I recommend that we close this one and open a new p2 one for the performance problem.