extremely large database size when sync attempts are made but don’t complete #7270
Comments
|
Additional information: Sync is not working with the database shared with @lyw07 on Kolibri 0.13.3 or later despite several attempts. I had cleared But still the sync process never progresses when another server is trying to pull for over 8 hours. It always shows that connection has been successfully established and the sync has started. Next reverted the database to the state where only |
redis-cli FLUSHALL Running this will get it going forward |
|
@intelliant01 The sync process can take a significant amount of time, as the message says. With changes coming in 0.14, we've added progress tracking that will provide better visibility on the sync. Using a smaller chunk size may help speed up the sync, depending on the machines and networks involved. You can adjust it with the |
|
Using beta11. I was able to complete the sync only after clearing out morango_buffer as well as morango_transfersession, if morango_transfersession was not cleared then sync failed after downloading some data with error about already existing morango_transfersess_sync_session_id_0455b5bd_fk_morango_s(don't remember exact error). After the successful sync and multiple reruns of sync, the morango_buffer is still full of data. "kolibri manage flushsyncsessions" from #7313 clears out morango_buffer. Not sure if this is a bug or something wrong in our database, https://pastebin.com/jXcSyNTx lot of: WARNING Error deserializing instance of XXXXXX with id XXXXX ['XXXXXXXX matching query does not exist.'] Want me to open another issue? |
|
@cyberorg I don't see it in the pastebin, but was there a log entry at the very beginning of the warnings that said something like: If so, I believe the next beta may resolve that issue for you, as some fixes from #7251 will be in it. In that case, the Morango buffers still had data in it because Morango was unable to merge those records into your facility data, as shown by the many log messages. Re-running a sync after that will always attempt to merge buffered changes if possible. On the next beta, if the data is still in the buffers, it should be merged in then leaving the buffers empty. |
No there is no such error, here is the beginning and ending: morango_buffer is now clean. morango_recordmaxcounter and morango_store has lot of rows though. |
It seems kolibri/core/auth/management/commands/deletefacility.py and kolibri/core/auth/management/utils.py changes are missing in beta 12. |
|
Ran kolibri manage flushsyncsessions and redis-cli FLUSHALL then sync Here is the full error: After Followed by long list of deserializing errors, ending with this: RE-running sync after above: Morango_buffer on client is clean, server still has lot of rows. Sync also does not seem to be complete |
Beta 12 does not have the Regarding the other errors, I suggested only clearing the |
jonboiser
added
the
APP: Device
|
Morango and syncing enhancements have been made to address this on a targeted basis. To finalize this specifically, I propose we add a recurring scheduled task that's configurable, but is disabled by default. |
rtibbles
modified the milestones:
Kolibri 0.16 Release: Existing Projects/General/Maintenance,
0.16 Future Patches triage
marcellamaki
modified the milestones:
Kolibri 0.17: Planned Patch 1,
Kolibri 0.17: Planned Patch 2
rtibbles
modified the milestones:
Kolibri 0.17: Planned Patch 2,
Kolibri 0.18: General maintenance
Observed behavior
Reported by a user:
When sync attempts are made but don’t complete, the table
morango_bufferin the database took 8.2 GB.Expected behavior
According to Blaine, the table
morango_buffershould be automatically cleared at the end of the syncUser-facing consequences
Users have an extremely large database
Errors and logs
…
Steps to reproduce
…
Context
Kolibri version: 0.14.0b8
The text was updated successfully, but these errors were encountered: