Files with unchanged sizes not syncing

[timg7]

Hello,
I noticed that certain modified files were not being updated during push or pull operations. The modifications were detected when drive was run, and the download/upload appeared to take place, but the actual file was not updated.

• The problem only occurs if the file size has not changed.
• Using the -ignore-checksum=false option causes the files to be updated properly.
• Both push and pull operations are affected
• Tested using small text files (<40 bytes)
• Using "master" branch, downloaded on 25 August 2016 and compiled with Go 1.6
• Platform is Raspbian (Debian) Jessie on a Raspberry Pi 2B (ARMv7)

PS. I really appreciate all the work you do on this project. Many, many thanks!

[odeke-em]

Hello there @timg7, thank you for the question and welcome to drive!

Yes in deed, this is a common case that we deemed so in issue #117, set in stone by CL #128.

Checksumming is turned off by default because for most of the cases, modTime changes and file size changes are sufficient to be able to guess if a file has changed, which is the common case.
Once in a while we'll get to the corner case, like yours.

If we turned on checksumming by default, we'll have accuracy but the overall speed and performance will depreciate unfortunately. Am glad that you were able to diagnose and use -ignore-checksum=false.

Also thank you for the kind words.

[timg7]

Thanks for the reply. Just to be clear: the mod-time is different, and drive detects this and (appears to) transfer the new file. But for some reason the contents of the file on the receiving end don't change.

For example:

tim@srv:~/gdrive/srv $ date > testfile.txt
tim@srv:~/gdrive/srv $ drive push .
Resolving...
+ /srv/testfile.txt
Addition count 1 src: 30.00B
Proceed with the changes? [Y/n]:
 30 / 30 [========================================================================================] 100.00% 1s

Cloud file is created correctly

tim@srv:~/gdrive/srv $ sleep 300; date > testfile.txt
tim@srv:~/gdrive/srv $  drive push .
Resolving...
M /srv/testfile.txt
Modification count 1 src: 30.00B dest: 30.00B
Proceed with the changes? [Y/n]:
 30 / 30 [=========================================================================================] 100.00% 0
tim@srv:~/gdrive/srv $

Cloud file does not receive new content

[odeke-em]

Yap yap, I should have clarified, my apologies; modTime changes are their own operations to avoid reuploading the file. When you do a drive touch, and drive detects just a modTime change, it'll only change the modTime :)

[odeke-em]

@timg7 aside from acknowledging that -ignore-checksum=false which is already documented fixes the issue, I don't think there is anything else we can do for this issue. Unless you think otherwise, please advise me.

Otherwise let's close this issue.

[timg7]

I see. Isn't that rather an "unsafe" policy? I was expecting the file to be transferred if modification-time or file-size change. This is how rsync and many others work.
The Mac & Win Google Drive clients are also able to detect changes that do not affect file-size, though I don't know if they use checksums by default.

[odeke-em]

IMO it is a waste of bandwidth to transfer files when the checksum is the same(pre-condition is that checksums are the same in the varying --ignore-checksum contexts). This is why we have modTime changes being transfers on their own.
If you need proper transfer, please set that flag --ignore-checksum=false, since for most people they'll complain about too much processing and slow startup time. Please see https://www.reddit.com/r/linux/comments/2u7224/official_google_drive_linux_client_screenshots/co5w192 and #117.

Also I was just looking at making a new release and noticed that this issue is a duplicate of #690.

[timg7]

I'm not sure we're on the same wavelength. What I'm getting at is described by kcwu and l3iggs in the #128 discussion:

l3iggs commented on Apr 1, 2015

Ok, then my opinion on how this should work is this: When checking if a transfer needs to be done (for push or pull):
Drive should compare all metadata (not including checksum) available to it for the local and remote files. If any differences are found, then transfer the file.

If the user passes a --checksum flag, compare all file metadata AND the checksum (which will be calculated over every single file in the transfer at this point), if any differences are found, then transfer the file.

[odeke-em]

Oh yes, I got you.

Am just saying that a modTime change doesn't warrant the content to be transferred. Google Drive's backend allows us to sync up certain meta data with code such as this https://github.com/odeke-em/drive/blob/master/src/remote.go#L737-L738

req.SetModifiedDate(true)

without having to transfer the whole file. This would be similar to an atomic touch on your file system.

[jaccarmac]

Just ran into this problem myself. Happy to find that there is a documented solution, but I think that the behavior here has to be improved. At least in my usage pattern of drive, I don't do a push when I intend to do a touch. And doing a push gives me a nice upload indicator, as if my changes to a file are being transmitted to Drive over the wire. If the checksum-checking behavior can't be changed, could a solution such as

1. A warning along the lines of x file's contents are not being updated. Only the last modified time will change. Please run with -ignore-checksum=false to force an upload.
2. A setting that disables these implicit touchs and tells the user to either force a checksum check or touch the remote file to update modification time.

work for this case?

[jaccarmac]

On top of that, I would argue that -force should intuitively do an upload instead of a touch when attached to a push. As of now, using -force doesn't affect the checksumming behavior.