guide: describe the how hash values (md5) are calculated

[efiop]

Tell about how we compute checksums for files, why we dos2unix when computing md5 and how to map the md5 value found in dvcfile to an actual cache file at .dvc/cache

[jorgeorpinel]

More details from Ruslan

Our checksums usually match with md5/md5sum (commands), except in cases where you have text files with windows-style crlf line endings.

[jorgeorpinel]

@efiop can you or someone from core summarize the explanation here or even submit a draft PR for this when you have some time?

We can take it on from there but it would be great if someone already familiar with the exact algorithm to generate file hashes works.

And/or if you can point me to the part of the code where it's done. Thanks!

[efiop]

@jorgeorpinel Sure! I just wonder if this is really p1. Seems like minor internal implementation details. This ticket was created before dvc commit, so it is even less important these days (people used to hack-around to generate a correct md5). I've lowered the priority to p2 for now, let me know if you think differently. Thanks.

[shcheklein]

@jorgeorpinel agreed with Ruslan, not sure why it was bumped to p1.

[jorgeorpinel]

Yeah sure, Idk either. p2 sounds good. Thanks!

[jorgeorpinel]

Hi! I wonder if users need to know this. What's the advantage or need to document it? Seems like an implementation detail. Or perhaps for security reasons (being able to confirm checksums)?

[MetalBlueberry]

In my personal case, we are working with a custom http remote that validates the checksum of the uploaded content. So being able to confirm the checksum is my personal need.

[jorgeorpinel]

This seems to keep coming up in support channels (1). Bumping priority a little.

Also, here's the relevant code: https://github.com/iterative/dvc/blob/aab5dec7f5f6dd16c00942285bfd229c693446b6/dvc/utils/__init__.py#L44

[jorgeorpinel]

Hi @efiop do you think the current content in https://dvc.org/doc/user-guide/project-structure/dvc-files#output-entries is enough? Still not a full explanation of how it's calculated internally but that's probably too deep? If so feel free to close this. Thanks

Cc @MetalBlueberry

[jorgeorpinel]

In fact I'm going to close this as stale for now, but please reopen if you disagree.