Store checksum in journal and verifiy upload for eml-files and others

[dragotin]

Once #2542 is implement, the checksum should be stored in local sync journal. We could use it to verify if an upload is needed, or if just the files metadata (mtime specifically) has changed. This would solve trouble we have with eml files on windows, and this could be a nice first usage case.

Problems: If we detect that only the mtime has changed, we could propagate the new mtime to the server, using PROPPATCH (needs to be verified). But that will probably not change the ETag of the file and also not the ETags of the parent directories. If it would, a client seeing a changed ETag would not understand that only the mtime has changed, and not the content of the file.

Note to self: If the server could do file checksums as well, it should also maintain directory checksums. The checksums only would change if the file contents or the contents of files within a directory changes. The ETag would just be an indication that something changed, either content or metadata. The client would need to consider the checksum to find that out.

[guruz]

It would be good in a way to have it support multiple checksum algos at the same time.
For example Adler32 is only good for error or change detection but not for deduplication (e.g. for #230)

[MTRichards]

Moving to 2.1 to keep 2.0 reasonable size.

[cdamken]

@MorrisJobke

00002839

[dragotin]

@cdamken please elaborate. Pasting numbers and adding people is not enough.

[MorrisJobke]

@cdamken please elaborate. Pasting numbers and adding people is not enough.

I think this is just for reference. I was mentioned, because it became a blue ticket and the number is the backlink to their system ;)

[dragotin]

@MorrisJobke yes, I know, but I would like to know the content of the blue-ticket as well.

[cdamken]

@dragotin The content of the ticket: *.EML files can't be deleted, even they are deleted in one client will reappear again and again, additionally generates a lot of network traffic.

[guruz]

@dragotin @ogoffart @ckamm @moscicki I was in general wondering if we should have a csync_journal table
phash, mtime, checksumtype, checksum that is only lazily filled.
Else we risk taking a lot of time to checksum everything..

[ckamm]

Storing the checksums in the DB will be part of #3735, this ticket can then be about using them to avoid unnecessary uploads.

[ckamm]

This has a couple of steps

• Calculate checksums on upload even if the server doesn't support them, store them in the local db
• If the new checksum is the same as the old one, assume the content is the same.
  • This could either be done while reconciling the local and db trees or in the upload propagation job.
• Do a remote metadata update instead of a full upload.
  • Check that PROPPATCH works

[moscicki]

I would guess you'd want md5 for that. The interesting bit is to see if it
is possible to marry it with adler32 in the server used to check the
transfers.

On Wed, Nov 11, 2015 at 4:40 PM, ckamm [email protected] wrote:

This has a couple of steps

• Calculate checksums on upload even if the server doesn't support
  them, store them in the local db
• If the new checksum is the same as the old one, assume the content
  is the same.
  • This could either be done while reconciling the local and db
    trees or in the upload propagation job.
• Do a remote metadata update instead of a full upload.
  • Check that PROPPATCH works

—
Reply to this email directly or view it on GitHub
#3235 (comment).

---

Best regards,
Kuba

[ckamm]

I've tested PROPPATCH for updating file mtimes on the server, but it currently makes the file etag change (owncloud/core#20474).

@dragotin and @ogoffart agree that in that case we are better off not telling the server about the new mtime, as it'd cause all other clients to re-download the unchanged file.

[ckamm]

Current pull request is #4180

[Dianafg76]

@ckamm what are the steps to reproduce this issue?

[ckamm]

@Dianafg76

• Create a new .eml file locally, wait for it to upload
• Update the mtime of the file without changing the contents (touch command)
• Verify that the client does not re-upload the file. (log file or activity log)
• Verify that the client still uploads the file if the content changes.
• Verify that the client still uploads other files. :)

[Dianafg76]

I tested this issue and is working OK
Desktop v ownCloud-2.1.0.5655-nightly20151126-setup.exe
Server v {"installed":true,"maintenance":false,"version":"8.2.1.4","versionstring":"8.2.1","edition":"Enterprise"}

Closed

[mocolini]

It seems that this bug affects owncloud-client yet.
If I don't remove ".eml" in "HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionPropertySystemPropertyHandlers" the client keep uploading (unchanged) *.eml files.
So, every time windows update itself (and so the ".eml" registry key is set back to default), hundred of .eml files (and MB) are re-uploaded to the server.

EDIT:
O.S.: windows 10 Home
Client version: 2.1.0 (build 5683)

[guruz]

@mocolini Are you saying the content of those .eml files does not change but they are still reuploaded?

[mocolini]

@guruz exactly.
Today windows 10 upgrade itserlf and re-set the ".eml" key in the register (the one I've removed days ago), after that owncloud-client has begun to re-upload all the (unchanged) *.eml files.

[ogoffart]

I think it needs to be uploaded at least once after upgrading before the checksum are stored in the databse and are compared.

Does it re-upload several time the same file?

[mocolini]

To be onest I don't know, 'cause I suddenly remove the ".eml" key .
So the upload is performed just one time to "populate" the db? Am 'I right?
And this happen for each windows PC, right?

[ogoffart]

yes, previous version of the client did not store the checksum in the database. and we only store it after we complete an upload or a download.

[mocolini]

Ok, perfect, I misunderstood the client behavior.
I'll check if the problem persist after complete upload/download, in that case I'll bother you again!
:-)

Cheers
Michelino

[guruz]

Sounds good :)
If you re-added the registry key, you can also try to remove it..

[mocolini]

Sure, I'll let you know