not all carddav contacts are displayed

[pafcioooo]

I have a nextcloud address book with ca. 2400 entries.
I have installed roundcube carddav plugin.
I have configured default address book for the plugin in plugins/carddav/config.inc.php.
Now address book is displayed but only with ca. 700 entries.

I am not sure if it is connected with the problem but in logs/error log there are following entries while operating on contacts in roundcube:

[05-May-2018 22:32:19 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:32:37 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:32:37 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:37 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:32:54 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:33:31 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645

[blind-coder]

This error happens when you remove the require_always attribute from a preconfigured addressbook, I honestly never tried doing that, because it didn't make sense to me to try.
Can you please add 'require_always' => [], to your preset and give it a try if it syncs correctly? Mind the refresh_time if you do.

[pafcioooo]

I have checked that I had 'require_always'=> true, so I changed it to 'require_always' => [],.
I have also changed refresh_time to 15 minutes with 'refresh_time' => '00:15:00',.
Then I have waited 30 minutes, logout and login, and even restart apache. Nothing helped.
After that I noticed that my own carddav settings in RC gui do not got changed even though I have updated plugins/carddav/config.inc.php file. So then I changed my refresh time in GUI to 15 minutes - but it did not helped either.

Then I have found that in logs/error there is following error entry (I have change real values here to XXXXX):
<g84rp7j0> DB Error: [1366] Incorrect string value: '\xF0\x9F\x8C\xA9' for column 'surname' at row 1 (SQL Query: INSERT INTO carddav_contacts (firstname,surname,showas,email,name,etag,vcard,abook_id,uri,cuid) VALUES ('XXXXX','XXXXX🌩','INDIVIDUAL','XXXXXX@XXXXX','XXXXXX XXXXX🌩','\"9ce5a49d7ef767d9c9b1a97735e33ced\"','BEGIN:VCARD\r\nVERSION:3.0\r\nPRODID:-//Sabre//Sabre VObject 4.1.2//EN\r\nUID:4b778423-88c8-4bb5-868e-ca29c5efa7a0\r\nN:XXXXX🌩;XXXXX;;;\r\nFN:XXXXX XXXXX🌩\r\nREV:20171203T194303Z\r\nTEL;TYPE=cell:+XXXXXX\r\nEMAIL:XXXXX@XXXXXX\r\nPHOTO

So I logged into nextcloud and deleted 🌩 from the contact. Then all contacts got synchronized.

Thanks

PS. I have also noticed that "use modern group" is disabled and it can not be enabled.

[blind-coder]

Which database backend are you using? MySQL, PostgreSQL or SQLite?
Can you share your preset addressbook settings (sans username/password/url)?
Lastly, can you share an anonymized VCF file including the emoji character? This really should work.

[pafcioooo]

ad. 1 I use MySQL
ad.2 Do you mean settings from config.inc.php?
ad. 3 Here you go, but I am not sure if it helps since this is with already deleted emoji character - I did not try to enter it once more. You can see the emoji character in my previous comment.
cb53696f-bc11-4af9-93af-0876bafe2748.zip

This is also interesting since I have another contact with emoji - only different ☁ and it was imported correctly.

I also remember that I had similar issues while importing emails to database and this is know problem with database encoding...

[blind-coder]

Thanks!
Yes, I mean the config.inc.php addressbook preset. The database should be able to handle emoji characters. Or at least RCMCardDAV really should gracefully skip over "broken" VCF entries.

[pafcioooo]

Ad.2 :-)

$prefs['Personal'] = array(
        // required attributes
        'name'         =>  'Personal',
        // will be substituted for the roundcube username
        'username'     =>  '%u',
        // will be substituted for the roundcube password
        'password'     =>  '%p',
        // %u will be substituted for the CardDAV username
        'url'          =>  'https://<main_domain_here>/remote.php/dav/addressbooks/users/%u/contacts',
        'active'       =>  true,
        'readonly'     =>  false,
        'refresh_time' => '00:15:00',
        //'require_always'=> true,
        'require_always' => [],
        //'fixed'        =>  array( 'username' ),
        'hide'        =>  false,
);

You are welcome and I thank you:-)

[pafcioooo]

Just remind myself, this can be helpfull:

• https://stackoverflow.com/questions/30074492/what-is-the-difference-between-utf8mb4-and-utf8-charsets-in-mysql
• https://mathiasbynens.be/notes/mysql-utf8mb4

[blind-coder]

Oh wow, thanks for this! I'll create a dbmigration for that.

[blind-coder]

I've created a hotfix that addresses this issue (at least in parts). I need to do a bit more to properly fix this as right now indices are smaller than the column they index, which can lead to issues if you have really long names/uris/cuids or whatnot. This affects the database structure of all supported databases, though, so I'll need to look a bit deeper here.