Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add foreign keys to filecache, share #28529

Closed
wants to merge 2 commits into from
Closed

Conversation

PVince81
Copy link
Contributor

@PVince81 PVince81 commented Jul 28, 2017

Just for fun...

Fixes #13143.

Setting up the DB works but as soon as you try accessing the filesystem 💥

{"reqId":"21UH6QkWsTlqzd3QlFD9","level":4,"time":"2017-07-28T14:45:44+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"webdav","method":"PROPFIND","url":"/owncloud/remote.php/webdav/","message":"Exception: {"Message":"An exception occurred while executing 'INSERT INTO `oc_filecache` (`mimepart`,`mimetype`,`mtime`,`size`,`etag`,`storage_mtime`,`permissions`,`checksum`,`path_hash`,`path`,`parent`,`name`,`storage`) SELECT ?,?,?,?,?,?,?,?,?,?,?,?,? FROM `oc_filecache` WHERE `storage` = ? AND `path_hash` = ? HAVING COUNT(*) = 0' with params ["1", "2", 1501253047, -1, "597b4e18a28aa", 1501253047, 23, "", "d41d8cd98f00b204e9800998ecf8427e", "", -1, "", 1, 1, "d41d8cd98f00b204e9800998ecf8427e"]:\n\nSQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`owncloud`.`oc_filecache`, CONSTRAINT `FK_D3AD7FD63D8E604F` FOREIGN KEY (`parent`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)","Exception":"Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException","Code":0,"Trace":"
#0 /srv/www/htdocs/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(128): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
#1 /srv/www/htdocs/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1015): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'INSERT INTO `oc...', Array)
#2 /srv/www/htdocs/owncloud/lib/private/DB/Connection.php(211): Doctrine\DBAL\Connection->executeUpdate('INSERT INTO `oc...', Array, Array)
#3 /srv/www/htdocs/owncloud/lib/private/DB/Adapter.php(113): OC\DB\Connection->executeUpdate('INSERT INTO `*P...', Array)
#4 /srv/www/htdocs/owncloud/lib/private/DB/Connection.php(249): OC\DB\Adapter->insertIfNotExist('*PREFIX*filecac...', Array, Array)
#5 /srv/www/htdocs/owncloud/lib/private/Files/Cache/Cache.php(267): OC\DB\Connection->insertIfNotExist('*PREFIX*filecac...', Array, Array)
#6 /srv/www/htdocs/owncloud/lib/private/Files/Cache/Cache.php(222): OC\Files\Cache\Cache->insert('', Array)
#7 /srv/www/htdocs/owncloud/lib/private/Files/Cache/Scanner.php(269): OC\Files\Cache\Cache->put('', Array)
#8 /srv/www/htdocs/owncloud/lib/private/Files/Cache/Scanner.php(209): OC\Files\Cache\Scanner->addToCache('', Array, -1)
#9 /srv/www/htdocs/owncloud/lib/private/Files/Cache/Scanner.php(312): OC\Files\Cache\Scanner->scanFile('', 3, -1, NULL, true)
#10 /srv/www/htdocs/owncloud/lib/private/Files/View.php(1302): OC\Files\Cache\Scanner->scan('', false)
#11 /srv/www/htdocs/owncloud/lib/private/Files/View.php(1343): OC\Files\View->getCacheEntry(Object(OC\Files\Storage\Wrapper\Checksum), '', '/admin')
#12 /srv/www/htdocs/owncloud/lib/private/Files/Node/Root.php(182): OC\Files\View->getFileInfo('/admin')
#13 /srv/www/htdocs/owncloud/lib/private/Files/Node/Root.php(353): OC\Files\Node\Root->get('/admin')
#14 /srv/www/htdocs/owncloud/lib/private/Server.php(940): OC\Files\Node\Root->getUserFolder('admin')
#15 /srv/www/htdocs/owncloud/apps/dav/lib/Connector/Sabre/ServerFactory.php(128): OC\Server->getUserFolder()
#16 [internal function]: OCA\DAV\Connector\Sabre\ServerFactory->OCA\DAV\Connector\Sabre\{closure}(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#17 /srv/www/htdocs/owncloud/lib/composer/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Object(Closure), Array)
#18 /srv/www/htdocs/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php(466): Sabre\Event\EventEmitter->emit('beforeMethod', Array)
#19 /srv/www/htdocs/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php(254): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#20 /srv/www/htdocs/owncloud/apps/dav/appinfo/v1/webdav.php(63): Sabre\DAV\Server->exec()
#21 /srv/www/htdocs/owncloud/remote.php(165): require_once('/srv/www/htdocs...')
#22 {main}","File":"/srv/www/htdocs/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php","Line":60,"User":"admin"}"}

@DeepDiver1975 @butonic @tomneedham

@PVince81 PVince81 added this to the backlog milestone Jul 28, 2017
@PVince81 PVince81 self-assigned this Jul 28, 2017
@PVince81
Copy link
Contributor Author

Ok well, turns out we likely can't use "-1" as parent, so would require "NULL" instead.

@PVince81
Copy link
Contributor Author

Fixed, ownCloud seems to work as far as I quickly tested.

It feels so good to be able to delete a parent entry in oc_filecache and see all children go directly... No need for expensive sub-selects.

These foreign keys will however break a few things that need adjsting (on the top of my mind):

  • need to make sure the DB is already sane before creating the foreign keys
  • some scanner code paths might restore parents/children, not sure if it does it in the correct order due to recursion
  • make sure that repair code that detects bad entries doesn't delete the dummy root entries

Now let's see how many of the unit/integration tests will fail...

@PVince81
Copy link
Contributor Author

PVince81 commented Jul 28, 2017

Next up:

  • oc_share.user_id -> oc_accounts.user_id (in the future use account_id instead)
  • mime type ? but without cascading delete
  • oc_mounts.root_id -> oc_filecache.fileid cascading delete
  • oc_mounts.user_id -> oc_accounts.user_id cascading delete
  • oc_mounts.storage_id ->oc_storages.numeric_id cascading delete
  • oc_properties.fileid -> oc_filecache.fileid cascading delete
  • user/group tables
  • oc_share.parent -> oc_share.id
  • also see Introduce foreign keys / cascading delete #13143 (comment)

@PVince81
Copy link
Contributor Author

PVince81 commented Jul 28, 2017

  • in general we should link more things by file id would also allow for cascading deletion
    • files_versions
    • encryption keys

@tomneedham
Copy link
Contributor

10.0.3 or later?

@PVince81
Copy link
Contributor Author

Too dangerous for 10.0.3. I'd see we can try with 11... Will need a lot of testing.

@PVince81
Copy link
Contributor Author

The challenge I see is that many people might have broken filecache entries that we need to repair, so we might need to have #28253 run first. Maybe we'll need more repair steps to delete orphaned items, etc before we can add keys.

@PVince81
Copy link
Contributor Author

and:

  • TODO: remove code that is not needed any more thanks to cascade delete (reduces number of queries)

@PVince81
Copy link
Contributor Author

17:43:33 1) Test\Share20\DefaultShareProviderTest::testDeleteFromSelfGroupUserNotInGroup
17:43:33 Failed asserting that exception of type "Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException" matches expected exception "\OC\Share20\Exception\ProviderException". Message was: "An exception occurred while executing 'INSERT INTO `oc_share` (`share_type`, `share_with`, `uid_owner`, `uid_initiator`, `item_type`, `file_source`, `file_target`, `permissions`) VALUES('1', 'group', 'user1', 'user1', 'file', '1', 'myTarget1', '2')':
17:43:33 
17:43:33 SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`oc_autotest28`.`oc_share`, CONSTRAINT `FK_24BEB63E79D59CC` FOREIGN KEY (`file_source`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)" at
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:59
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1012
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/Connection.php:211
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:208
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/QueryBuilder/QueryBuilder.php:141
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/tests/lib/Share20/DefaultShareProviderTest.php:1934
17:43:33 .
17:43:33 
17:43:33 2) Test\Share20\DefaultShareProviderTest::testDeleteFromSelfGroupDoesNotExist
17:43:33 Failed asserting that exception of type "Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException" matches expected exception "\OC\Share20\Exception\ProviderException". Message was: "An exception occurred while executing 'INSERT INTO `oc_share` (`share_type`, `share_with`, `uid_owner`, `uid_initiator`, `item_type`, `file_source`, `file_target`, `permissions`) VALUES('1', 'group', 'user1', 'user1', 'file', '1', 'myTarget1', '2')':
17:43:33 
17:43:33 SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`oc_autotest28`.`oc_share`, CONSTRAINT `FK_24BEB63E79D59CC` FOREIGN KEY (`file_source`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)" at
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:59
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1012
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/Connection.php:211
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:208
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/QueryBuilder/QueryBuilder.php:141
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/tests/lib/Share20/DefaultShareProviderTest.php:1979
17:43:33 .
17:43:33 
17:43:33 3) Test\Share20\DefaultShareProviderTest::testDeleteFromSelfUserNotRecipient
17:43:33 Failed asserting that exception of type "Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException" matches expected exception "\OC\Share20\Exception\ProviderException". Message was: "An exception occurred while executing 'INSERT INTO `oc_share` (`share_type`, `share_with`, `uid_owner`, `uid_initiator`, `item_type`, `file_source`, `file_target`, `permissions`) VALUES('0', 'user2', 'user1', 'user1', 'file', '1', 'myTarget1', '2')':
17:43:33 
17:43:33 SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`oc_autotest28`.`oc_share`, CONSTRAINT `FK_24BEB63E79D59CC` FOREIGN KEY (`file_source`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)" at
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:59
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1012
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/Connection.php:211
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:208
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/QueryBuilder/QueryBuilder.php:141
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/tests/lib/Share20/DefaultShareProviderTest.php:2068
17:43:33 .
17:43:33 
17:43:33 4) Test\Share20\DefaultShareProviderTest::testDeleteFromSelfLink
17:43:33 Failed asserting that exception of type "Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException" matches expected exception "\OC\Share20\Exception\ProviderException". Message was: "An exception occurred while executing 'INSERT INTO `oc_share` (`share_type`, `uid_owner`, `uid_initiator`, `item_type`, `file_source`, `file_target`, `permissions`, `token`) VALUES('3', 'user1', 'user1', 'file', '1', 'myTarget1', '2', 'token')':
17:43:33 
17:43:33 SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`oc_autotest28`.`oc_share`, CONSTRAINT `FK_24BEB63E79D59CC` FOREIGN KEY (`file_source`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)" at
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:59
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1012
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/Connection.php:211
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:208
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/lib/private/DB/QueryBuilder/QueryBuilder.php:141
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/tests/lib/Share20/DefaultShareProviderTest.php:2109
17:43:33 .
17:43:33 
17:43:33 5) Test\User\ManagerTest::testCountUsersOnlySeen
17:43:33 Failed asserting that 5 matches expected 4.
17:43:33 
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/tests/lib/User/ManagerTest.php:223
17:43:33 
17:43:33 6) Test\User\ManagerTest::testCallForSeenUsers
17:43:33 useruMLrfsb82X00d, testseencount1, testseencount2, testseencount3, testseencount4 !== useruMLrfsb82X00d, testseencount1, testseencount2, testseencount3, testseencount4, testseen1, testseen2, testseen3, testseen4
17:43:33 Failed asserting that 9 matches expected 8.
17:43:33 
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/tests/lib/User/ManagerTest.php:269
17:43:33 
17:43:33 7) OCA\FederatedFileSharing\Tests\FederatedShareProviderTest::testCreateCouldNotFindServer
17:43:33 Failed asserting that two strings are equal.
17:43:33 --- Expected
17:43:33 +++ Actual
17:43:33 @@ @@
17:43:33 -'Sharing myFile failed, could not find [email protected], maybe the server is currently unreachable.'
17:43:33 +'An exception occurred while executing 'INSERT INTO `oc_share` (`share_type`, `item_type`, `item_source`, `file_source`, `share_with`, `uid_owner`, `uid_initiator`, `permissions`, `token`, `stime`, `file_target`) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [6, "file", 42, 42, "[email protected]", "shareOwner", "sharedBy", 19, "token", 1501256600, ""]:
17:43:33 +
17:43:33 +SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`oc_autotest28`.`oc_share`, CONSTRAINT `FK_24BEB63E79D59CC` FOREIGN KEY (`file_source`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)'
17:43:33 
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/apps/federatedfilesharing/tests/FederatedShareProviderTest.php:230
17:43:33 
17:43:33 8) OCA\FederatedFileSharing\Tests\FederatedShareProviderTest::testCreateException
17:43:33 Failed asserting that two strings are equal.
17:43:33 --- Expected
17:43:33 +++ Actual
17:43:33 @@ @@
17:43:33 -'dummy'
17:43:33 +'An exception occurred while executing 'INSERT INTO `oc_share` (`share_type`, `item_type`, `item_source`, `file_source`, `share_with`, `uid_owner`, `uid_initiator`, `permissions`, `token`, `stime`, `file_target`) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [6, "file", 42, 42, "[email protected]", "shareOwner", "sharedBy", 19, "token", 1501256600, ""]:
17:43:33 +
17:43:33 +SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`oc_autotest28`.`oc_share`, CONSTRAINT `FK_24BEB63E79D59CC` FOREIGN KEY (`file_source`) REFERENCES `oc_filecache` (`fileid`) ON DELETE CASCADE)'
17:43:33 
17:43:33 /var/lib/jenkins/workspace/owncloud-core_core_PR-28529-ZKJAAJMYNOVNLWLAEFN7W7LYVM7ZRH7UGQWGEXIMDUVOOTEXTF4A@2/apps/federatedfilesharing/tests/FederatedShareProviderTest.php:286

@PVince81
Copy link
Contributor Author

to be redone step by step: not all tables at once

@PVince81 PVince81 closed this Apr 27, 2018
@PVince81 PVince81 deleted the add-foreign-keys branch April 27, 2018 08:46
@lock
Copy link

lock bot commented Jul 31, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Jul 31, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Introduce foreign keys / cascading delete
3 participants