Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #563: How to keep data and cache on external drive #732

Closed

Conversation

dashohoxha
Copy link
Contributor

Fix #563
Replaces #565

@shcheklein shcheklein temporarily deployed to dvc-org-pr-732 October 24, 2019 05:04 Inactive
@shcheklein shcheklein temporarily deployed to dvc-org-pr-732 October 24, 2019 12:01 Inactive
@dashohoxha dashohoxha changed the title Fix #563: How to manage data and cache on external drive Fix #563: How to keep data and cache on external drive Oct 24, 2019
@jorgeorpinel

This comment has been minimized.

@dashohoxha

This comment has been minimized.

Copy link
Contributor

@jorgeorpinel jorgeorpinel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't done an extensive check but just looking at the first file in my first commit, lines 40-43 (abd2446#diff-278c4173e3a65d532906f6a613dc7ce0R40-R43) for example are not included in this PR.

I guess there was a reason to so it, but having closed the original PR and opened a new one from a different branch makes it extra time consuming to track and review the continuity in these changes. I tried just doing git cherry-pick to apply my work again, but since the file is located somewhere else, this method does not work.

We'll probably have to manually compare and reapply my changes (combined here: 482e1c7...f345245) vs those in this PR.

@dashohoxha

This comment has been minimized.

@jorgeorpinel

This comment has been minimized.

@jorgeorpinel

This comment has been minimized.

@shcheklein

This comment has been minimized.

@jorgeorpinel jorgeorpinel temporarily deployed to dvc-org-pr-732 October 28, 2019 06:23 Inactive
@jorgeorpinel

This comment has been minimized.

Copy link
Contributor

@jorgeorpinel jorgeorpinel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, a great article, I would just rename the file to something shorter.

Copy link
Member

@shcheklein shcheklein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The biggest concern is that I don't understand the motivation for this combination. It's very very specific one of the few ways organizing this. Why do we pick and when do we pick this one it's not clear.

@dashohoxha

This comment has been minimized.

@jorgeorpinel jorgeorpinel temporarily deployed to dvc-org-pr-732 October 29, 2019 03:25 Inactive
@dashohoxha
Copy link
Contributor Author

I changed the title and rewrote the motivation. I hope it looks better now.

I was tempted again to remove the reference about Windows (https://github.com/iterative/dvc.org/pull/732/files#diff-e67a9ecb076e21d068e0f5559ed64582R30) but I didn't dare. @jorgeorpinel it really seems out of place to me, please think about removing it again.

@shcheklein
Copy link
Member

btw, @dashohoxha how does this one correlate with the new data sharing section?

@dashohoxha
Copy link
Contributor Author

how does this one correlate with the new data sharing section?

I don't see any correlation.
Maybe there is some correlation with the external data management (it could be an example of it), but I think that it deserves to be a standalone HowTo.

we have to make sure that we can read and write the `/data/` directory:

```dvc
$ sudo chown <username>: -R /data/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not right to provide chown, chmod and other commands to manage permissions. They are not actionable anyway. Exact commands you need to run depend on the case. Some general suggestions to check the permissions should be enough?

@@ -0,0 +1,151 @@
# Keep Data and Cache Outside the Project

Sometimes we would like to keep the data and cache outside the project
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure how is it related to being able to search code tbh, could you elaborate?

Sometimes we would like to keep the data and cache outside the project
directory, so that we can search easily the code of the project.

Keeping the data outside of the project, on a fixed absolute path, is also
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a a very tricky case - if multiple projects use the same data how do they switch different versions?

@shcheklein
Copy link
Member

I don't see how does it replace #565 or fixes #563. It's a very specific how-to focused on external dependencies and output on a local machine. Also, it reminds me a lot some of the other data managements sections you put together.

@shcheklein
Copy link
Member

closing this as stale

@shcheklein shcheklein closed this Mar 14, 2020
@jorgeorpinel jorgeorpinel deleted the howto-manage-data-and-cache-on-external-drive branch May 5, 2020 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

how: use DVC when data is stored in an external drive
3 participants