add a prefix index to filecache.path, attempt 2

[icewind1991]

Second attempt at #26070, this time with a shorter prefix to hopefully not have the same issues.

---

The reason that filecache.path hasn't had an index added is the mysql limitation of ~1kb for indexeded fields,
which is to small for the path, however mysql supports indexing only the first N bytes of a column instead of the entire column,
allowing us to add an index even if the column is to long.

Because the index doesn't cover the entire column it can't be used in all situations where a normal index would be used, but it does cover the path like 'folder/path/%' queries that are used in various places.

Sqlite and Postgresql don't support prefix indexes, but they also don't have the 1kb limit and DBAL handles the differences in index creation.

Signed-off-by: Robin Appelman [email protected]

[nickvergessen]

Can't remember completely why it failed back then. Maybe we can ask people to setup the index manually on several instances to not fail again.

[skjnldsv]

Code is good, no idea about what @nickvergessen was referring to. Let's wait for another review :)

[nickvergessen]

no idea about what nickvergessen was referring to. Let's wait for another review :)

It 💥 on several servers (including c.nc.c iirc) back then (when attempt one was done). We can try it again, but in case it breaks again we have to revert it again.

[icewind1991]

After some more testing I found that postgresql requires setting some additional options ("operator class") on the index before it can be used for LIKE pattern matching when using a non C locale.

Since dbal does not currently support setting these options I've disabled creating the index on pgsql for now so we can more easily add the correct index later and we don't get stuck with a useless index.

[skjnldsv]

Any news? :)

[nickvergessen]

Seems it's helping lot of times, but e.g. with groupfolders the index is not used:

EXPLAIN SELECT `file`.`fileid`, `storage`, `path`, `path_hash`, `file`.`parent`, `name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `permissions`, `checksum`, `metadata_etag`, `creation_time`, `upload_time` FROM
`oc_filecache` `file` LEFT JOIN `oc_filecache_extended` `fe` ON `file`.`fileid` = `fe`.`fileid` WHERE (`storage` = 330) AND (((`path`  COLLATE utf8mb4_general_ci LIKE '__groupfolders/22/%') OR (`path_hash` = '…')) AND (`name`  COLLATE u
tf8mb4_general_ci LIKE '%…%')) ORDER BY `mtime` desc LIMIT 5;

+------+-------------+-------+--------+-----------------------------------------------------------------------------------------------------+----------+---------+---------------------+------+-------------+
| id   | select_type | table | type   | possible_keys                                                                                       | key      | key_len | ref                 | rows | Extra       |
+------+-------------+-------+--------+-----------------------------------------------------------------------------------------------------+----------+---------+---------------------+------+-------------+
|    1 | SIMPLE      | file  | index  | fs_storage_path_hash,fs_storage_mimetype,fs_storage_mimepart,fs_storage_size,fs_storage_path_prefix | fs_mtime | 8       | NULL                |   76 | Using where |
|    1 | SIMPLE      | fe    | eq_ref | PRIMARY                                                                                             | PRIMARY  | 8       | db_biss.file.fileid |    1 | Using where |
+------+-------------+-------+--------+-----------------------------------------------------------------------------------------------------+----------+---------+---------------------+------+-------------+

[skjnldsv]

Applied on NC production. 1 sec per 100k rows in the filecache. Expecting minimal downtime then.
So far so good!

[icewind1991]

@nickvergessen make sure that you have #28476 and #28608 applied

[skjnldsv]

Let's merge? and backport to 20! 🚀

[skjnldsv]

/backport to stable22

[skjnldsv]

/backport to stable21

[skjnldsv]

/backport to stable20

[icewind1991]

Let's merge?

Counting as approval, merging

[backportbot-nextcloud]

The backport to stable20 failed. Please do this backport manually.

[solracsf]

Some feedback about this PR here: #29377