Issues with missing personal files and no access to shared files after upgrading from 6.0.6 to 7.x

[fleish]

Steps to reproduce

1. upgrade owncloud 6.0.6 using SQLite database to 7.0.3 or 7.0.4 using ubuntu package from http://download.opensuse.org/repositories/isv:/ownCloud:/community:/7.0/xUbuntu_12.04/
2. run occ upgrade
3. (optional - issue occurs in either database type) convert from SQLite to MySQL using occ db:convert-type --all-apps mysql owncloud localhost
4. Login to your owncloud via the web

Expected behaviour

You should be able to see your files & the files that have been shared with you

Actual behaviour

You will not be able to view your files nor any of the files that have been shared with you (for the latter you can see the folders that have been shared with you but you cannot get into them)

NOTE: talked to AnybodyElse on IRC about this and as suggested ran "occ files:scan --all" which fixed the first issue of not being able to see my own files.

Server configuration

Operating system: Ubuntu 12.04.5 LTS

Web server: Apache/2.2.22 (Ubuntu)

Database: SQLite3 or mysql Ver 14.14 Distrib 5.5.40, for debian-linux-gnu (x86_64) using readline 6.2

PHP version: PHP 5.3.10-1ubuntu3.15 with Suhosin-Patch (cli) (built: Oct 29 2014 12:19:04)

ownCloud version: 7.0.4.2

Updated from an older ownCloud or fresh install: updated from 6.0.6

List of activated apps:
Activity
Deleted Files
Documents
Encryption
First Run Wizard
Full Text Search
LDAP user and group backend
PDF Viewer
Pictures
Share Files
Text Editor
Video Viewer

The content of config/config.php:

'**REDACTED**', 'passwordsalt' => '**REDACTED**', 'datadirectory' => '/var/www/owncloud/data', 'dbtype' => 'mysql', 'version' => '7.0.4.2', 'installed' => true, 'ldapIgnoreNamingRules' => false, 'forcessl' => true, 'enable_previews' => false, 'theme' => '', 'maintenance' => false, 'trusted_domains' => array ( 0 => '**REDACTED**', ), 'secret' => '**REDACTED**', 'share_folder' => '/Shared', 'dbname' => 'owncloud', 'dbhost' => 'localhost', 'dbuser' => 'owncloud', 'dbpassword' => '**REDACTED**', ); **Are you using external storage, if yes which one:** NO **Are you using encryption:** YES ### Client configuration **Browser:** Google Chrome Version 39.0.2171.71 (64-bit) **Operating system:** Mac OSX 10.8.5 (also tested in Mac OSX 10.9.5) ### Logs #### Web server error log [Thu Dec 11 21:12:27 2014] [error] [client 50.248.219.14] PHP Fatal error: Cannot redeclare class OC_Search_Provider in /var/www/owncloud/lib/private/search/provider.php on line 5, referer: https://somehost.somedomain.org/index.php/apps/files/?dir=%2F #### ownCloud log (data/owncloud.log) messages seen after the upgrade: 1) This one appears repeatedly. I think it corresponds to me trying to view things that have been shared with me by user with UUID yyy-yyy-yyy-yyy (REDACTED): {"app":"files_sharing","message":"Unable to get mount for shared storage '3' user 'yyy-yyy-yyy-yyy'","level":3,"time":"2014-12-11T21:07:31+00:00"} 2) This one appears just once, seems to happen when trying to view shared files. First error No. 1 is repeated twice, then this message appears, then error No. 1 is repeated many more times: {"app":"PHP","message":"Invalid argument supplied for foreach() at \/var\/www\/owncloud\/lib\/private\/files\/cache\/scanner.php#216","level":3,"time":"2014-12-11T21:07:31+00:00"} 3) This one appears a couple of times: {"app":"PHP","message":"Cannot redeclare class OC_Search_Provider at \/var\/www\/owncloud\/lib\/private\/search\/provider.php#5","level":3,"time":"2014-12-11T21:12:27+00:00"} 4) These appeared once in conjunction with me deleting a file on my computer (REDACTED my UUID xxx-xxx-xxx-xxx) and my sync client pushing it to the server. The file was actually deleted, but the messages were logged: {"app":"Encryption library","message":"No share key found for user \"xxx-xxx-xxx-xxx\" for file \"Test222.txt\"","level":2,"time":"2014-12-11T21:11:47+00:00"} {"app":"core","message":"No preview providers exist","level":3,"time":"2014-12-11T21:11:47+00:00"} {"app":"hook","message":"error while running hook (OC\Preview::post_delete_files): No preview providers","level":3,"time":"2014-12-11T21:11:47+00:00"}

[LukasReschke]

@icewind1991 Any idea?

[fleish]

Updated a few things for accuracy & clarity. Uncommented secret in config file. Updated error message No. 2 as I see when it is happening.

[ghost]

Mhhh, normally the:

Cannot redeclare class OC_Search_Provider

only is shown if some leftover files from a previous install is still there or am i wrong?

[MorrisJobke]

@fleish Is the instance working after the upgrade and before the database migration? (after step 2) Or is it still broken at this point. I ask this to find the cause of your problem (either upgrade code or migration code)

[fleish]

@MorrisJobke sorry if that was unclear, I tried to caveat step #3 to note it happens regardless of the DB migration. I've tried just upgrading the code and leaving the instance on SQLite and the same issue(s) occur

[fleish]

@MorrisJobke or anyone else able to provide some suggestions here? My users are getting restless with daily issues with sqlite locking

[MorrisJobke]

@PVince81 @schiesbn YOu have more experience with the sharing part and maybe can better analyze this one. Thanks

[PVince81]

@fleish please check your oc_storages table.
Are there entries with the format "local::$datadir/$user" and also "home::$user" ? (there should be only one per user, not both). Did you move/change/rename the data dir when upgrading ?

[fleish]

@PVince81 checking in oc_storages pre-upgrade I see only 1 local:: entry as follows:
local::/var/www/owncloud/data/|4

the rest are like:
home::admin|1
home::REDACTED-GUID|2

EDIT: And there is no home:: entry ending in 4

I don't change anything manually during the upgrade, only whatever the apt/dpkg upgrade does and then occ upgrade & occ files:scan --all

[PVince81]

Okay good, so your setup isn't a case of the known "home storage duplication" which is known to make shares disappear.

[PVince81]

Let's check for orphaned shares:
select * from oc_share where item_source not in (select fileid from oc_filecache)

and orphaned storages:
select * from oc_filecache where storage not in (select numeric_id from oc_storages)

Also that error 3) about the search provider class is suspicious. Do you somehow happen to have the search app enabled twice from different sources ?

[fleish]

Ophaned shares:

id|share_with|uid_owner|parent|item_type|item_source|item_target|file_source|file_target|permissions|stime|accepted|expiration|token|mail_send
19|**REDACTED_GUID1**|**REDACTED_GUID2**||file|1721|/1721|1721|/REDACTED_FILENAME 3.zip.gpg|||0|27|1398283250|0|0
20|**REDACTED_GUID3**|**REDACTED_GUID2**||file|1721|/1721|1721|/REDACTED_FILENAME 3.zip.gpg|||0|27|1398283261|0|0
85|**REDACTED_GUID4**|**REDACTED_GUID5**||file|34047|/34047|34047|/REDACTED_FILENAME.pdf|||0|27|1409772053|0|0

Nothing for orphaned storages.

When I check the web UI for apps, I only see Full Text Search once in the list. I'm not sure how else to check for this case.

[PVince81]

I see the error above says error with shared storage 3.

Is there a storage with this numeric_id in the oc_storage?
select * from oc_storages where numeric_id=3

[fleish]

Yes, just one
home::REDACTED_GUID|3

[fleish]

anything else we can try to debug this one?

[PVince81]

Ok let's try and find the shared files that do not appear:
select * from oc_share where file_target like '%FILENAME%' (replace FILENAME with a part of the file or folder name that appears in "Shared with you" but not "All files".
Please post the result and check whether "owner" and "shared_with" fields are correct.
Then, write down the matching "item_source", and look up the file id for it:

select * from oc_filecache where fileid=THEITEMSOURCE. Replace THEITEMSOURCE with the item_source id you found in the first query.

[PVince81]

You said your users are having "database is locked" issues with sqlite ? How many users do you have ?
If more than 1-2, it is best to switch to mysql (you can convert the database with this tool: http://doc.owncloud.org/server/7.0/admin_manual/maintenance/convert_db.html) to avoid database locks.

[fleish]

Just to be clear, I've been running these on the instance with issues that I have rolled back to version 6.

Here are the first 2 items from oc_share. The first one is a folder shared with a group, while the second is a folder shared to an individual user. The GUID & group values look to be correct.

7|IT|**REDACTED_GUID1**||folder|356|/356|356|/IT|||1|31|1393605790|0|0

63|**REDACTED_GUID2**|**REDACTED_GUID3**||folder|7184|/7184|7184|/Some Pathname|||0|31|1403635337|0|1

The corresponding oc_filecache details:
356|3|files/IT|8c9b25373ad9c17c37942f323c355ab0|69|IT|2|1|1421463681|1418193552|0|54b9d0b4b9ad0|564708132|420198229

7184|15|files/documents/Some Pathname|1f8fe0f51b8b8b433d2a65072c8246a9|1573|Some Pathname|2|1|1418404137|1405710069|0|548b212c84f94|3607992|2697910

As for the database locked issues with sqlite. We desperately want to migrate it to mysql, however the DB conversion utility was not introduced until version 7 so we are kind of stuck in a catch 22 until we can get our instance upgraded from 6->7.

[PVince81]

You said there is a user Y who doesn't see the files/folders F shared with them in "All files".
Does user Y see the shares in the "Shared with you" section of the sidebar ?

I was wondering whether the scenario of the missing files match the following:

• User U1 shares "folder" with group G1.
• User U2 (member of G1) shares "folder/subfolder" with user U3
• User U3 is unable to see "subfolder"

If your situation matches the scenario, this could be the issue from #12475

[fleish]

That scenario doesn't match. Clicking on "shared with you" in the sidebar shows the folders that are shared with you ... but trying to access any of them the window just spins for a bit and then redirects to "index.php/apps/files/?dir=%2F"

And owncloud.log shows this where REDACTED_UID1 is the user who shared the files with me
{"app":"files_sharing","message":"Unable to get mount for shared storage '3' user 'REDACTED_UID1'","level":3,"time":"2015-01-20T23:12:01+00:00"}

EDIT: If you have some time we could setup to discuss this in real time on IRC or something that'd be greatly appreciated

[PVince81]

Hmmm, but your symptoms match the other issue. You could still try with the patch #13470

[PVince81]

After applying the patch, the user who reshared with the other one needs to unshare+reshare to make it work.

[fleish]

To confirm, this is a patch for v7? If so, do I apply it after completing the upgrade from 6 to 7? Or after the new files are present but before the occ upgrade is run?

[PVince81]

It doesn't matter when you apply it.
Once applied, it will affect newly created reshares.
So if you had broken reshares you need to unshare + reshare them.

[fleish]

I don't think we have any re-shares so hopefully unsharing/re-sharing won't be required. If that ends up being the case we might as well nuke the whole instance and just start over. I don't see a tests sub-dir for that second file in the patch. Is it just ./lib/private/share/share.php that needs updating?

[PVince81]

Yes, only update "lib/private/share/share.php".

If you didn't have any reshares the fix might still work because it affects the logic that checks "give me all shares that target user XYZ through his groups" (the broken part was "through his groups if the user has more than 1 group assigned"). I'd say try it out with a few broken shares. If they're still broken after the patch, try recreating them.

[fleish]

I've upgraded a test system to OC7, replaced the file, done occ upgrade and am waiting for occ files:scan --all to complete before checking how things turned out this time.

[fleish]

still coming up with the unable to mount shared storage '##' user 'GUID' when trying to access shared folders

[blizzz]

Oh, forget it. As talked it makes sense to take out app sorting. OC_App is required for isType check. Unfortunately, it cannot be mocked because it is a static method. We can do things worse, by introducing a second isType method or adding a wrapper or… making it more ugly and error prone.

[PVince81]

You probably meant to comment on the PR instead ? #13865

[blizzz]

Yupp, sorry 'bout this.

[PVince81]

I also raised #14044 to make sure the bogus "local::" storage cannot be created as a side-effect of other bugs.

[DeepDiver1975]

@PVince81 is there anything we can do for 7.0.5? Seems rather complex - moving to 7.0.6 ???? THX

[jcfischer]

any chance of getting this fixed this week in an official release? We are planning our next upgrade windows on March 7th but need to test it this week. Otherwise we will have to push back the upgrade another 2 weeks

[PVince81]

Here is a summary of the current status:

1. If you are having issues with SQLite + database is locked (@fleish the original reporter):

• closeCursor fix: [stable7] Close cursor early in calculateFolderSize #13832
• additional close cursor for oc_preferences, still needs review for stable7: [Stable7] app upgrade order fix #14040 (might not be needed but would be safer to have)

2. If you have users that lost files and additional bogus storages suddenly appeared in oc_storages (both @fleish had these too, and you too @jcfischer ?)

• stable7 patch, still to be reviewed and merged: [Stable7] app upgrade order fix #14040
  • and its master counterpart on ownCloud upgrade: upgrade all apps in order, load important ones #13865
• stable7 patch for LDAP default user id (just in case) is already on stable7: fix LDAP update routine to OC 7 #13759

[jcfischer]

@PVince81 - yep, case 2) is our problem from a 6.0.6 to 7.0.4 upgrade

[DeepDiver1975]

closeCursor fix, already in stable7/7.0.5

@PVince81 afaik we reverted that because of issues with oracle - right?

[LukasReschke]

I merged No. 2, however, I have no idea if this will get into 7.0.5 or 7.0.6

@karlitschek Any input here?

[PVince81]

@DeepDiver1975 dammit, right. Reverted because of Oracle.

[karlitschek]

@LukasReschke 7.0.5 is still not build. So this would go into 7.0.5

[LukasReschke]

@jcfischer 7.0.5 should work then – or the current stable7 branch.

@karlitschek THX

[jcfischer]

@LukasReschke - is 7.0.5 Enterprise going to be built this week?

[karlitschek]

@jcfischer Yes. At the same time than 7.0.5 community

[PVince81]

@fleish did you have any luck with all the stable7 patches ? See #12801 (comment)

[bboule]

Hey Guys will this be addressed in 7.0.5?

[LukasReschke]

@bboule That question is answered above... But just to rephrase it again: Yes.

[bboule]

I thought so... but I need to set expectations with third partied so I wanted to confirm. Thanks for the help!!

[fleish]

@PVince81 sorry, been busy and forgot to respond. I upgraded via the 7.0.0RC1 -> 7.0.4 path and haven't had time to go back and try it with the proper patches.