files are uploaded to ceph even when wrong checksum given

[individual-it]

1. setup ceph
2. upload file curl -u uu0:uu0 -X PUT http://localhost/owncloud-core/remote.php/dav/files/uu0/newfile.txt -d "BBBBB" -H "OC-Checksum: SHA1:random" -v
3. check response and response code
4. check also if the file exists

even the response code is correct the file is uploaded
same problem when finally moving chunks to final destination

[ownclouders]

GitMate.io thinks possibly related issues are #69 (Cannot upload a file of 10GB), #100 (original file disappear when upload with overwrite is stoped by files_antivirus), and #68 (Thumbnails are not updated when editing a file).

[PVince81]

The issue is related to the fact that we disabled part files for this storage.

The upload flow is currently as follows:

1. bytes are streamed to the server by a client (here curl)
2. bytes arrive in Sabre\File.php, the final file is opened (not part file as disabled): https://github.com/owncloud/core/blob/stable10/apps/dav/lib/Connector/Sabre/File.php#L200
3. bytes are streamed into the final file and checksum is computed on the fly
4. final file is closed
5. checksum is compared with the header => invalid: https://github.com/owncloud/core/blob/stable10/apps/dav/lib/Connector/Sabre/File.php#L209
6. nothing more. usually we'd delete the part file in case of error but there is none, we likely don't delete the final file as we might have overwritten an existing one. https://github.com/owncloud/core/blob/stable10/apps/dav/lib/Connector/Sabre/File.php#L231

We can't compute the checksum earlier without going through all bytes, would require a staging area like tmpfs.

Possible solutions:

• looking into a way to rollback on object store, maybe using versions ?
  or
• provide a different staging area to compute checksums
  or
• rely on storage to compute checksums and disable our own
  or
• enable part files and deal with the loss of performance due to the final rename (on object store rename is copy + delete)

@DeepDiver1975

[PVince81]

This also means that if there was an existing file, the contents of said file is lost, unless we can tell the object store to restore the old version.

And even in the latter scenario, is this protected by transactions or is there a risk of having any client download the wrong file before the old one is restored ?

[phil-davis]

bytes are streamed into the final file

So while a client is doing upload-overwrite of a (big) file, other clients cannot download the previous content of the file because it has been "zapped" already.

If the previous file content is somehow restored from a saved version or from somewhere, then there will still be a time interval when the client sees the file "disappear" (or be locked). In that time you want to make sure that the client does not locally delete the file. Because if the big upload fails, then the previous content of the file will "reappear" and the client would then have to download it again.

[individual-it]

an other checksum issue #156

[PVince81]

the flow might be slightly different.

@sharidas found recently that we first write the oc_filecache entry and then upload the file to object store from a temporary file. This was in context of https://github.com/owncloud/enterprise/issues/3173 where a 200 status was returned despite an error.

I wonder if the checksum thing might be slightly related and the proposed fix above could help solve the checksum issue as well.

[micbar]

@sharidas Is this fixed by https://github.com/owncloud/enterprise/issues/3173#issuecomment-497626218

[sharidas]

@micbar I will have a look at this issue and will update here.

[sharidas]

While I was trying to reproduce the problem, I did as follows:

• Upload a file and provide wrong checksum to curl

sujith@sujith-ownCloud  /tmp/upload  curl -u admin:admin -T file1.txt  'http://localhost/testing/remote.php/dav/files/admin/test.txt' -H "OC-Checksum: SHA1:random" -v
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /testing/remote.php/dav/files/admin/test.txt HTTP/1.1
> Host: localhost
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
> OC-Checksum: SHA1:random
> Content-Length: 305000000
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Date: Tue, 18 Jun 2019 15:53:55 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Set-Cookie: oc6pg2kojoxa=2c7fod2didfkcp2jcadhfpjgrg; path=/testing; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=cAagj8O5tpBzOd97t2liYx3Cnn8Hy1fBolvOHkYtDF6kx7dIZ8W5w%2BIq20n%2BhEYE1IyBjJv35gi5Zdy%2FnBOfie3iC82Rig2QfHnnho1AzI8FCSX8ISaXsujd7gmfXPgu; path=/testing; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: oc6pg2kojoxa=2l5m5fs6v3q0eihc1r3sveli1a; path=/testing; HttpOnly
< Set-Cookie: cookie_test=test; expires=Tue, 18-Jun-2019 16:53:56 GMT; Max-Age=3600
< Content-Length: 264
< Connection: close
< Content-Type: application/xml; charset=utf-8
< 
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
  <s:message>The computed checksum does not match the one received from the client.</s:message>
</d:error>
* Closing connection 0
 sujith@sujith-ownCloud  /tmp/upload 

The the file was not uploaded and the error points to the reason for failure. This was conducted with files_primary_s3.

Tried without providing checksum detail to curl:

 sujith@sujith-ownCloud  /tmp/upload  curl -u admin:admin -X PUT 'http://localhost/testing/remote.php/dav/files/admin/test.txt' -d "BBBBB"  -v                             
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /testing/remote.php/dav/files/admin/test.txt HTTP/1.1
> Host: localhost
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 5
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 5 out of 5 bytes
< HTTP/1.1 201 Created
< Date: Tue, 18 Jun 2019 15:58:27 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Set-Cookie: oc6pg2kojoxa=475sb10dmb1eup1b86s80j4fee; path=/testing; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=aZLqZA%2BoliJqF%2FNBh2%2BPHatI%2FW%2BYiZ1L%2Bneeo3iu8kOgI0KwaV8sTU2hG3jLZxHAImsse7gXOngVKiKOQvnbrAweVgdwDfIQJ3GgXeZXAQQAlSgApdIFSDLUYlmFMA%2Fa; path=/testing; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: oc6pg2kojoxa=cncm5hhsp04k8qcc86j04vqf1u; path=/testing; HttpOnly
< Set-Cookie: cookie_test=test; expires=Tue, 18-Jun-2019 16:58:27 GMT; Max-Age=3600
< OC-FileId: 00000024oc6pg2kojoxa
< Content-Length: 0
< ETag: "5d090a23cb8ba"
< OC-ETag: "5d090a23cb8ba"
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host localhost left intact
 sujith@sujith-ownCloud  /tmp/upload 

Any pointers here would be helpful, to reproduce the issue.

[sharidas]

Tested with ceph:
Steps done:

• Start the docker docker run -e NETWORK_AUTO_DETECT=4 -e CEPH_DEMO_UID=owncloud -e CEPH_DEMO_ACCESS_KEY=owncloud123456 -e CEPH_DEMO_SECRET_KEY=secret123456 owncloudci/ceph:tag-build-master-jewel-ubuntu-16.04
• copy apps/files_primary_s3//tests/drone/ceph.config.php to config/
• With docker ps -a find the id of the container
• Using docker inspect <id> find the IPAddress of the container
• Edit config/ceph.config.php : modify 'endpoint' => 'http://ceph:80/' with 'endpoint' => 'http://<IP>:80/'
• create the bucket ./occ s3:create-bucket OWNCLOUD --accept-warning

Ran the commands below:

• Using curl

sujith@sujith-ownCloud  /tmp/upload  curl -u admin:admin -X PUT 'http://localhost/testing3/remote.php/dav/files/admin/a.txt' -d "BBBBB" -H "OC-Checksum: SHA1:random"  -v
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /testing3/remote.php/dav/files/admin/a.txt HTTP/1.1
> Host: localhost
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
> OC-Checksum: SHA1:random
> Content-Length: 5
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 5 out of 5 bytes
< HTTP/1.1 400 Bad Request
< Date: Wed, 19 Jun 2019 06:31:32 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Set-Cookie: ocoxyjz9vjn8=lsr2ml22546me4s2idua9cm0ph; path=/testing3; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=zeuRbfPvVeHp%2FxX2M6g%2B6WcNxnSR%2BEMdeGK5ejsFqnaKLWBC38iYp7zA2Tu6dWxdBbmuqsgH8NgplJUFHfSHSiYoaAadG6Wfb%2FaYFPF16Zms06Gv7Gyg2CiFo8Z6kQvp; path=/testing3; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: ocoxyjz9vjn8=s4ntg5fokbb5s8e05lpoev5oni; path=/testing3; HttpOnly
< Set-Cookie: cookie_test=test; expires=Wed, 19-Jun-2019 07:31:32 GMT; Max-Age=3600
< Content-Length: 264
< Connection: close
< Content-Type: application/xml; charset=utf-8
< 
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
  <s:message>The computed checksum does not match the one received from the client.</s:message>
</d:error>
* Closing connection 0
 sujith@sujith-ownCloud  /tmp/upload 

• Another test using curl:

sujith@sujith-ownCloud  /tmp/upload  curl -u admin:admin -T file1.txt  'http://localhost/testing3/remote.php/dav/files/admin/test.txt' -H "OC-Checksum: SHA1:random" -v  
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /testing3/remote.php/dav/files/admin/test.txt HTTP/1.1
> Host: localhost
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
> OC-Checksum: SHA1:random
> Content-Length: 105000000
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Date: Wed, 19 Jun 2019 06:36:44 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Set-Cookie: ocoxyjz9vjn8=j3q8kg13hc789ga625s04ve070; path=/testing3; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=j5JNh49BIn8KYIjU8k2fzbMqDnyw6mPZ8pwjdrONlBIXkSMoPwDIl%2BlFG66WbeXMMnIN0XALVzmxtjg%2FjWf7NyAFDeV%2B%2BnQKeFwf6uOUR8jx3B9L5J0DE7omNTrZguut; path=/testing3; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: ocoxyjz9vjn8=3npobqbdio24o1tp1jt6efg5em; path=/testing3; HttpOnly
< Set-Cookie: cookie_test=test; expires=Wed, 19-Jun-2019 07:36:44 GMT; Max-Age=3600
< Content-Length: 264
< Connection: close
< Content-Type: application/xml; charset=utf-8
< 
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
  <s:message>The computed checksum does not match the one received from the client.</s:message>
</d:error>
* Closing connection 0
 sujith@sujith-ownCloud  /tmp/upload 

Again not able to reproduce :(

[individual-it]

seems to have been fixed by owncloud/core@cbab58b

need to write acceptance tests for that

[dpakach]

The acceptance tests were already there but were skipped because of the issue. They have been unskipped in owncloud/core#35601