[oC7 Beta 1] S3 external storage doesn't work

[ser72]

Expected Behavior
S3 mounted as external storage and accessible

Actual Behavior
S3 is mounted with green light but not accessible

Steps
Configure S3 external storage as such:

Go to ALL FILES page and see "UNKNOWN ERROR"

Go to EXTERNAL STORAGE page and see "UNKNOWN ERROR" remains but Amazon folder is listed:

Click on Amazon folder -- it spins forever attempting to open but never brings up a directory listing:

There are NO logs appearing in the owncloud.log file!!

Environment
Broken in 7.0.2RC1
Broken in oC7 Beta 1
Works in 6.0.5

@butonic @PVince81

[PVince81]

Got the error by debugging, somehow this is not logged:

PDOException: SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect integer value: '' for column 'mtime' at row 1 in /srv/www/htdocs/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Statement.php on line 138

Call Stack:
    0.0011     244120   1. {main}() /srv/www/htdocs/owncloud/index.php:0
    0.0545    4796616   2. OC::handleRequest() /srv/www/htdocs/owncloud/index.php:28
    0.6212   15296408   3. OC\Route\Router->match(???) /srv/www/htdocs/owncloud/lib/base.php:730
    0.6469   15971936   4. call_user_func(???, ???) /srv/www/htdocs/owncloud/lib/private/route/router.php:246
    0.6469   15971968   5. __lambda_func(???) /srv/www/htdocs/owncloud/lib/private/route/router.php:246
    0.6470   15988056   6. require_once('/srv/www/htdocs/owncloud/apps/files/ajax/list.php') /srv/www/htdocs/owncloud/lib/private/route/route.php(135) : runtime-created function:1
    0.6546   16334968   7. OCA\Files\Helper::getFiles(???, ???, ???) /srv/www/htdocs/owncloud/apps/files/ajax/list.php:28
    0.6546   16335016   8. ';

[PVince81]

Hmmmm, it seems that this PR #9509 breaks it for me.
Before that commit I get "signature failed" from my location (which I already got in the past when connecting to a US bucket).
After that commit I also get the "Unknown error".

@ser72 would be good if you could try checking out commit 46ba08a and 687cd7f and compare whether it works for you.

[PVince81]

@ser72 is your test instance empty ?
When debugging I see that the stat() call has empty content for most of the fields, even the mtime is an empty string.

[ser72]

@PVince81

No there are files in the ocsteve bucket. only 1 or 2 but there are files there

[butonic]

@ser72 what database are you using? I just tested with sqlite and get a lot of database is locked errors. I also saw the 'Unknown Error' once and had the endless spinning. After a few reloads the issue seems to have vanished. I'll investigate further. If possible send me the credentials via mail, pls.

[ser72]

@butonic

Postgresql.

[karlitschek]

The database is locked errors are also worth to be debugged. I think we got bug reports like that in the past but could never really reproduce.

[carlaschroder]

FWIW I am experiencing a similar issue.

1. I set up my S3 bucket in the External Storage app and share it with all users. It gives me a green light.
2. It creates a folder in my Files page as it supposed to
3. When I click on the folder it does not display the files in my S3 bucket, but says "Nothing in here. Upload something!"
4. I upload files, which then appear in my S3 dashboard, but not on my oC files page

My oC log says:

Debug webdav Sabre\DAV\Exception\NotAuthenticated: No basic authentication headers were found
Debug core OC_Image->fixOrientation() Orientation: -1
Debug core OC_Image->fixOrientation() No readable file path set.

This is oC 7.0.2 on Ubuntu 14.04, Apache, MariaDB

[butonic]

@karlitschek I can avoid the database is locked errors by commenting \OC_DB::beginTransaction(); and \OC_DB::commit(); in cache/scanner.php. Seeing the file scan of the external storage take forever - live and in a color terminal - reminded me of @PVince81 mentioning long running deletes being the problem and I vaguely remembered @icewind1991 adding the transaction to improve scan performance. Maybe we should disable the transaction on sqlite. @jancborchardt did you by chance use files_external?

@carlaschroder that might be caused by another issue I noticed: the external storage seems to keep the cached files when removing and readding the cache. Even worse the storage filecach seems to be reused when the bucket name is changed, requiring a manual file rescan.

For this issue I think I have a solution, but the tests somehow disagree ... I am on it.

[butonic]

I'm not done yet, but progress can be tracked in https://github.com/owncloud/core/tree/fix_files_external_amazon_s3_stat_call

TODO:

• fix tests (some seem to suffer from timeouts being too short)
• increase version and add update.php to migrate storage id from 'amazon::' . $params['key'] . md5($params['secret']) to 'amazon::' . $params['bucket'] . ':' . $params['key']. Otherwise users will suffer a complete rescan of the s3 bucket and a cluttered filecache.

[butonic]

most of the problems are solved with #10104. I need another review for it, then I'll rebase my work to fix some remaining issues

[butonic]

#11375 now fixes leftover issues after #10104 has been merged and backported.

@icewind1991 @karlitschek @DeepDiver1975 What do we do with the database locked error on sqlite. It seems to be reproducible when scanning an external storage due to the open transaction. Should we disable it for sqlite only?

[jancborchardt]

@butonic if you’re asking if I used files_external when these SQLite locking errors appeared: I’m pretty sure that yes, I did.

[butonic]

Tracking the db locked issue in #11391

[butonic]

fix will be in next build of master and stable7, discussion on sqlite db locking after the EE release in #11391