Add populateHashField.php

[mwjames]

This PR is made in reference to: #

This PR addresses or contains:

• During an upgrade some elements of a table may require to be updated and pending its importance has to happen before any access to that table occurs but due to the sheer size of the table it could hog resources making the update.php run for a considerable time
• This PR provides a mechanism to postpone such tasks and trigger a message block about outstanding tasks
• The PR includes an example implementation for the smw_hash conversion task. The initial update of smw_hash (Add IDs warmUpCache #3080) only happens once when switching to 3.0 and with a set execution threshold of 200000 (meaning missing converted rows) the conversion will be part of the normal update.php / setupStore.php routine, hereafter the process is postponed (the status is saved in .smw.json) to allow administrators to quickly run the upgrade while a message will start to appear to remind users and administrators that the upgrade hasn't finished yet and requires immediate action.
• The message cannot be turned off unless actions follow and remove the necessary obstacles. In case of smw_hash, the populateHashField.php script is provided to finalize the conversion and set the "populate.smw_hash_field_complete": true.
• As previously noted (Add smwgConfigFileDir, refs 3506, 3563 #3596 (comment)), .smw.json will contain more than the upgrade key allowing the system to store state information hereby providing a method to check its software, upgrade, and components.

This PR includes:

• Tests (unit/integration)
• CI build passed

Example

{
    "mw-31-00": {
        "upgrade_key": "a10c426c369eddc1bb5bffda68e565c430d2d96a",
        "populate.smw_hash_field_complete": true
    },
    "mw-31-00-sunittest_": {
        "upgrade_key": "01bae0a6f5c40524001a7fa4912de253b5a2f6da",
        "populate.smw_hash_field_complete": true
    }
}

[mwjames]

execution threshold of 200000

200000 (defined by PopulateHashField::COUNT_SCRIPT_EXECUTION_THRESHOLD) is more a less a house number I picked so that by normal server hardware standards it should finish the task within 2 minutes tops and hereafter switches to the populateHashField.php script.

This is mainly of interested for users who have a huge smw_object_ids table (with millions of rows) and would make update.php (the SMW part) to monopolize the time required to finish the update.

We may trust people to run populateHashField.php when we tell them to do so but we cannot be sure and reminding them in a persistent way (== annoy users) is the only guarantee we have to ensure administrators actually do follow the task required.

[mwjames]

@kghbln On some thread (can't find the ref) you mentioned that it may take up the 15 min to run update.php, this PR should minimize the impact yet provide the software with a way to inform users about missing tasks and continue its functioning.

Crucial part is the information we store in .smw.json therefore the file is no longer just a simple key file, it provides us with the method to store the state and compare its components in a performant way (no DB connection or table access, just a simple file read like the one that happens on other .json message files or when PHP loads a .php file).

[kghbln]

I will "rename" the file from "update key file" to "setup key file" in the docu since this appears to be more appropriate in the light of this enhancement.

[kghbln]

"setup key file"

Second thoughts, now "setup information file".

[mwjames]

I will "rename" the file from "update key file" to "setup key file"

Fine by me, I hadn't any name in mind so I generally refer to it as .smw.json. For technical reasons that matches the $GLOBALS['smw.json'] key that holds the content of the file once loaded by Installer:loadSchema.

[kghbln]

Fine by me ...

Fair enough. I just figured something a bit more descriptive and more intuitive is good for the end users "marketing wise".

[mwjames]

docu since this appears to be more appropriate in the light of this enhancement.

At first it just contained the upgrade key but I needed a place to store certain information without relying on a DB connection (also we cannot manipulate LocalSettings.php to set some variables) and since .smw.json was already in place it seemed a natural choice to extent its application.

[mwjames]

the main page of the wiki which will be accessible:

To be clear, the message will be shown an every page to really "annoy" users so that administrator gets his act together and runs the job. It is also a warning to the end-user that something hasn't been finished and they should avoid data intensive work.

[kghbln]

Back-ported with 8f1177a

[kghbln]

Documented

[kghbln]

Just upgraded a rather big instance:

Checking 'smw_hash' field consistency ...
   ... missing 344933 rows ...
   ... updating document no.       14077114 (100%)
   ... writing the status to the setup information file ... 
   ... done.

For 4 Cores, 10 GB RAM with PHP 7.0.x it took 7 minutes + another 5 minutes for the first 200,000 rows.

[kghbln]

@mwjames Not sure if related to the changes connected with this feature, but the size of the database connected to the wiki mentioned in my previous post dropped from 3.3 GB to 1.9 GB [!sic]. Talking about more efficient data storage. :)

[mwjames]

@mwjames Not sure if related to the changes connected with this feature, but the size of the database connected to the wiki mentioned in my previous post dropped from 3.3 GB to 2.9 GB. Talking about more efficient data storage. :)

There could be several reasons including that outdated entities have been removed from the DB hereby no longer occupying any table space. Also, one has to remember that the "raw" data alone doesn't count towards the size, indices count as well and we made some changes in 3.0 to the table index structure. Orthogonal to the mentioned size changes here, the 3985 issue scratches a different topic but would have a similar impact on the size especially for users that use text properties frequently where unique term values are at the lower end of the distribution [0] (see `unique_terms_occurrence_in_percent`). [0] https://sandbox.semantic-mediawiki.org/w/index.php?title=Sp%C3%A9cial:SemanticMediaWiki&action=stats%2Ftable

…

On 6/3/19, Karsten Hoffmeyer ***@***.***> wrote: @mwjames Not sure if related to the changes connected with this feature, but the size of the database connected to the wiki mentioned in my previous post dropped from 3.3 GB to 2.9 GB. Talking about more efficient data storage. :) -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #3611 (comment)