[6.0.4] Error during the 'rescan' process.

[cdamken]

Expected behavior

run file scan and finish

Actual behavior

run file scan and error

Steps to reproduce

When I started the first rescan with the command 'php console.php files:scan -all' I got the following error:

PHP Fatal error:  Allowed memory size of 6442450944 bytes exhausted (tried to allocate 513 bytes) in /var/www/html/owncloud/3rdparty/Patchwork/PHP/Shim/Normalizer.php on line 199

I added more RAM to the machine, configured 10GB Memory in the php.ini file, but it didn't help:

PHP Fatal error:  Allowed memory size of 10737418240 bytes exhausted (tried to allocate 513 bytes) in /var/www/html/owncloud/3rdparty/Patchwork/PHP/Shim/Normalizer.php on line 199

Then I tried with setting "-1" to the size in php.ini, but it didn't help. I get this error:

Killed

How can we do this rescan without getting an error?

Server specifications

Redhat 6
PHP 5.5.3
MySQL 5.1.73
Data Storage mounted over NFS

@craigpg

[karlitschek]

@icewind1991 any idea?

[PVince81]

@cdamken does the error always happen in that Normalizer class or does the error message sometimes says something else ?

[PVince81]

Please confirm that no symbolic links have been used in the data directory. These could cause infinite loops in some cases.

[MTRichards]

How many users are on the system?
Scanfiles can be run against a user account, and we can run it that way user by user.

[DeepDiver1975]

@cdamken is the php module intl installed and enabled?

As soon as that module is available we shall not use the fallback to Patchwork/PHP/Shim/Normalizer.php - researching ...

[cdamken]

@PVince81 No symbolic links
@MTRichards active users: arround 5'357, after the first try to scan: 14'007 users (took al LDAP users) but the folders are "empty" (files folder only)
@DeepDiver1975 intl was not installed, but I will install and i will retest again.

[DeepDiver1975]

active users: arround 5'357, after the first try to scan: 14'007 users (took al LDAP users) but the folders are "empty" (files folder only)

okay - perform the scan for only those users which have issues with files which are not in sync

[PVince81]

As usual, please check the oc_storages table to see if there are duplicates (with the local::$datadir/$ldapid/ format and home::$ldapid format)

[cdamken]

We have 4'936'829 files and 756'793 directories

[DeepDiver1975]

We have 4'936'829 files and 756'793 directories

Don't expect that you can scan this in one run - as said above: perform the scan only for single users

[craigpg]

Shouldn't the scanner be able to support re-scanning 5M files / 700k directories without running out of memory? Might need to rethink our use of recursion in the scanner.

[cdamken]

@craigpg 👍

Shouldn't the scanner be able to support re-scanning 5M files / 700k directories without running out of memory? Might need to rethink our use of recursion in the scanner.

ownCloud scanner should not depend of 3th party apps!!

Mainly Important tools should run without other dependencies! What happens with real systems with 5TB and 5000 users?

@DeepDiver1975

Don't expect that you can scan this in one run - as said above: perform the scan only for single users

exactly, when we run .occ scan:files -- all we expect that it won't crash!

[DeepDiver1975]

Afaik the code base improved in this area in oc7

We are talking about how to solve this in the scope of oc6: do it step by step

Read this as a work around to solve the issue right away

[cdamken]

I have 6.0.4 and after the suggestion from @DeepDiver1975 :

We are talking about how to solve this in the scope of oc6: do it step by step

I did:

for f in `ls /data`; do ./occ files:scan $f; done

Works 😄

but I get this message many times:

PHP Fatal error:  Call to a member function getAbsolutePath() on a non-object in /var/www/html/owncloud/lib/private/files/utils/scanner.php on line 54

@blizzz This was supposed to be fix in 6.0.3?: #7216 and #8052

[PVince81]

@cdamken possibly because that ls command also tried to find a user called owncloud.log and index.html ?

[PVince81]

Replacing the recursion as @craigpg suggested would only fix cases where there are very deep folder structures, as calling a function multiple times increases the memory footprint.

A few things are cached in local variables during scanning, maybe these should be cleant up automatically after they're not useful any more.

There's also a "change propagator" that will store a list of parent folders for which the etags need to be updated, whenever a change has been found. Not sure but that could also increase the memory usage in some situations.

CC @icewind1991

[cdamken]

@PVince81

possibly because that ls command also tried to find a user called owncloud.log and index.html ?

i just putted the command with ls as example 😏 sorry.

The exact command was:

find . -maxdepth 1 -type d | cut -d"/" -f2 > /tmp/current-accoutns.txt
 for i in `cat /tmp/current-accoutns.txt` ;do php console.php files:scan $i >> /tmp/rescan-user-output.txt ;done

[icewind1991]

I've tried multiple times to get rid of recursion in the file file scanner but never successfully due to the nature of the problem.
Simplified, scanning a folder involves 3 steps

1. Put the folder itself in the cache
2. Scan the contents of the folder
3. Update the size of the folder

Since step 3 requires step 2 to be completed there is no way to move recursion to the end and do a tail recursion optimalization.

[cdamken]

@icewind1991 How big should be the cache if the user folder is 50Gb?

[PVince81]

@cdamken it's not an issue of file size, but number of files/folders and how many levels subfolders go. Metadata cache.

[cdamken]

@PVince81 Thanks for clearing the question 👍

Btw:

Still scaning..... 4 1/2 hours.... about 151 messages like: #10954 (comment)

appears a new message (only one) between them:

PHP Fatal error:  Maximum execution time of 3600 seconds exceeded in /var/www/html/owncloud/lib/private/files/cache/changepropagator.php on line 72

[cdamken]

@PVince81 @DeepDiver1975 @blizzz Could be that #10954 (comment) happens because the accounts are not in LDAP anymore?

[PVince81]

Not sure, but could be an obsolete user, yes. You can check the oc_storages table and see whether the user in question still has entries there.

[bantu]

I think using recursion is fine if you release all the resources in the right way.

Recursion is difficult to get right as witnessed by this ticket and the apparent fix in #11282. I think we should establish that recursion should be avoided.

[cdamken]

@icewind1991 Only the memory part is fixed in #11282, but the error when scanning specific users appear:

PHP Fatal error:  Call to a member function getAbsolutePath() on a non-object in /var/www/html/owncloud/lib/private/files/utils/scanner.php on line 54

Is there any solution for this part?

[icewind1991]

Only thing I can think of that causes that problem is an invalid user parameter (I get the same error when calling the scan command with a non existing user) and that's "addressed" in #11281

[icewind1991]

@cdamken are you able to retest this with the latest stable7?

[cdamken]

@icewind1991. Sadly it's a production system with oC6.0.5 and can not test it in stable7. Additionally those users exist. we test them individually and comes the same error.

[butonic]

@cdamken please try with manually patching:

From f14a8fe5a241b8127c0f5aeb9dec2d022133b675 Mon Sep 17 00:00:00 2001
From: Robin Appelman <[email protected]>
Date: Wed, 24 Sep 2014 15:48:54 +0200
Subject: [PATCH] Show a proper error message when trying to scan the
 filesystem for a non existing user

---
 apps/files/command/scan.php | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/apps/files/command/scan.php b/apps/files/command/scan.php
index 5fc6325..cd90e3f 100644
--- a/apps/files/command/scan.php
+++ b/apps/files/command/scan.php
@@ -68,7 +68,11 @@ class Scan extends Command {
                        if (is_object($user)) {
                                $user = $user->getUID();
                        }
-                       $this->scanFiles($user, $output);
+                       if ($this->userManager->userExists($user)) {
+                               $this->scanFiles($user, $output);
+                       } else {
+                               $output->writeln("<error>Unknown user $user</error>");
+                       }
                }
        }
 }
-- 
2.1.1

[karlitschek]

@cdamken What is the status here?

[cdamken]

@butonic

Unknown user carlos

But the user exist in LDAP and

ll /mnt/data/carlos/
total 24
drwxr-xr-x  2 apache apache 8192 10. Apr 15:33 cache
drwxr-xr-x 11 apache apache 8192 21. Aug 15:42 files
drwxr-xr-x  6 apache apache 8192 24. Apr 14:48 files_trashbin

and the user carlos can login and use his files.

[butonic]

@blizzz can you do a scan of the filesystem for ldap users on the command line? Just trying to make sure this works ... I really need to set up an ldap server ...

[PVince81]

@butonic docker openldap + phpmyldapadmin: owncloud/administration#8

[PVince81]

I tried it locally and it doesn't work with LDAP.
You need to use the LDAP id of the user instead of the login name.

Have a look at the oc_ldap_user_mapping table.
I have a user "user1" so I did this:

MariaDB [owncloud]> select * from oc_ldap_user_mapping where ldap_dn like '%user1%'
    -> ;
+--------------------------------------------------+--------------------------------------+--------------------------------------+
| ldap_dn                                          | owncloud_name                        | directory_uuid                       |
+--------------------------------------------------+--------------------------------------+--------------------------------------+
| cn=user1,cn=group10,ou=groups,dc=owncloud,dc=com | c411eabc-e28f-1033-8c1a-b3a3a32c79f7 | c411eabc-e28f-1033-8c1a-b3a3a32c79f7 |
+--------------------------------------------------+--------------------------------------+--------------------------------------+

Then pick the owncloud name and use it when scanning:

./occ files:scan c411eabc-e28f-1033-8c1a-b3a3a32c79f7

This works for me.

Ideally the name should automatically be recognized, not sure where that logic is @blizzz ?

[butonic]

I was at first expecting this as well but got distracted by /mnt/data/carlos/ being present. I can confirm @PVince81 that scanning with the userid works.

@cdamken In the browser console you can lookup the current usierid with OC.currentUser. Could it be that /mnt/data/carlos/ is a leftover of an old database user that is now overshadowed by an ldap user with the same name?

[PVince81]

I asked @cdamken per chat to compare the database entries for a working user and a broken user, in oc_storages and oc_ldap_user_mapping.

@butonic's theory could make sense.

[cdamken]

@PVince81 @butonic @blizzz

In the table, the users where the scan works looks like:

+----------------------------------------------------------+---------------+--------------------------------------+
| ldap_dn                                                  | owncloud_name | directory_uuid                       |
+----------------------------------------------------------+---------------+--------------------------------------+
| cn=carlos1,ou=users,ou=neth,ou=id,ou=auth,o=server,c=com | carlos1       | 217f2580-14a8-102f-8191-8b570c3d2851 |
+----------------------------------------------------------+---------------+--------------------------------------+

and the user where the scan doesn't works looks like:

+---------------------------------------------------------+--------------------------------------+--------------------------------------+
| ldap_dn                                                 | owncloud_name                        | directory_uuid                       |
+---------------------------------------------------------+--------------------------------------+--------------------------------------+
| cn=carlos,ou=users,ou=neth,ou=id,ou=auth,o=server,c=com | e32aa52c-171c-1033-995d-018008f37f99 | e32aa52c-171c-1033-995d-018008f37f99 |
+---------------------------------------------------------+--------------------------------------+--------------------------------------+

Somehow the users are added different. Is there anyway to make a rebuild of the table?

[PVince81]

@blizzz can you advice on how to fix the wrong owncloud_name ?
Would deleting these entries recreate them ?
Or should the directory_uid be manually copied over to owncloud_name ?

[blizzz]

@cdamken check the experts tab

[blizzz]

guys, the directory name is retrieved from getHome(), right, and not build together like $datadir . '/' . $uid?

[blizzz]

ok, after discussion with @PVince81, forget my last comment.

[PVince81]

@cdamken please use the buttons on the expert tab to clear the mappings and confirm that it fixed the issue, then we can close this ticket

[cdamken]

@PVince81 @blizzz The problem to add "cn" in expert tab will all shared lost. we have to provide an alternate solution

[blizzz]

@cdamken the following lines of bash return the list of users. Run it from within the owncloud root dir or adjust the paths. To use instead of ls.

ocmysqlus=$(grep dbuser config/config.php  | cut -d"'" -f4)                                
ocmysqlpw=$(grep dbpassword config/config.php  | cut -d"'" -f4)                                
ocmysqldb=$(grep dbname config/config.php  | cut -d"'" -f4)

mysql -u "$ocmysqlus" -p"$ocmysqlpw" -s -e "select owncloud_name from oc_ldap_user_mapping" -D "$ocmysqldb"

[PVince81]

To summarize: the core issue here is that ./occ files:scan --all was taking too much memory. @icewind1991 provided a fix which we don't know whether it was tested on the target system.
The current workaround is to run ./occ files:scan user_id with all user ids.
The user ids are taken from the home directories. Some home dirs don't map with the ldap id for 150 users. So the proposed solution for the core issue is to get the user list from the database instead.

Not sure if the 150 users are really to be considered broken. But I think this would be something for a separate ticket.

@cdamken let us know whether the SQL fix to get the whole user list works.
More ideally would be to know whether @icewind1991's patch fixes the memory issue for ./occ files:scan --all

[cdamken]

After the workaround of @blizzz its a good way to make a full scan for now. I will close this issue. About the changes of the LDAP database and file sharing I will create another issue and provide the needed information.