Store editable mappings in multiple fossildb columns+keys #6903
Conversation
|
Great stuff! Testing of proofreading went mostly well 👍 However, during merging/splitting in proofreading mode, I occassionally ran into failed network requests: Unfortunately, I cannot really figure out what's going wrong. Since there is no status code, I'm not even sure the request arrived at the server? The failed requests point towards Next, I'll test the migration. |
|
The migration worked flawlessly 👍 |
|
Hmm, I’m not sure about that. The chunk data request did not change in this PR. However, the chunk list request may have changed (since it has to list all chunks of the segments currently mapped to that particular agglomerate id) so there may be invalid byte offsets there? However, I don’t see how that would not lead in a 4xx or 5xx but this strange behavior of your screenshot 🤔 I could not reproduce this behavior. |
|
I think I found an unrelated problem, where I cannot perform a particular split 🤔 I’ll have a look at that |
|
What would be the rollback strategy for this PR? I just had some trouble working on other branches, since fossilDB complains that I'm not opening the new column families (which don't exist on the other branches). How did you solve this while branch hopping? I deleted |
|
This was also my strategy locally. In production, we would have to continue opening the full list of column families. This already works for the dev deployment fossildb (which is shared by all dev deployments) |
It turned out to be caused by mag1 not being present. The agglomerate file had a node very close to the edge of the agglomerate. The voxel there was eaten by downsampling (the other segment id won at that position). That meant that looking up the segment id by the position from the agglomerate file lead to the wrong segment id, essentially making the split a no-op since the edge was not actually present in the agglomerate. I’d say that this is acceptable since it only happens in case of a mismatch between the present segmentation data and the agglomerate file. I fear there is no good way to guard against this. I added some warning logging in case an edge is not present and a split becomes a no-op. |
|
@philippotto do you think we can go ahead with this? Can you still reproduce the issue with mesh loading? Maybe you could show me in person? |
…t split/merge no-op
…nds/webknossos into editable-mappings-distributed
As discussed in person, I think we can go ahead with this. I also fixed the incorrect fallback fetching case. |
…ove_wkconnect * 'master' of github.com:scalableminds/webknossos: Update docker compose commands + dev install readme (#7002) Add segment groups (#6966) Add screenshot nightly test for wkorg (#7030) Workaround for WebGL crash for datasets with many segmentation layers (#6995) Fix download of public annotation, include access ctx in user cache key (#7025) Fix that changing a segment color could lead to a crash (#7000) Add more error chaining to annotation download (#7023) Guard against NaNs in shader (#7018) Store editable mappings in multiple fossildb columns+keys (#6903) Context action to move tree to group (#7005) Release 23.05.0 (#7014) Remove vault cache when reloading dataset (#7007) Fix viewing of public datasets (#7010) Update screenshots scalebar positioning (#7003) Update team members (#6999)
…ty-list-drawings * 'master' of github.com:scalableminds/webknossos: (25 commits) Fix issues with styling in dark mode on login page (#7052) Fix nightly by setting missing token (#7048) Release 23.05.1 (#7042) DRY types in update_actions.ts (#7036) Remove some spammy logging from backend (#7039) Use zarr string fill values (#7017) Fix voxel offset for Neuroglancer Precomputed datasets (#7019) Log when user is activated (#7027) Fix exception in applying UpdateTreeGroupVisibility skeleton action (#7037) Fix organization storage layouting (#7034) Update docker compose commands + dev install readme (#7002) Add segment groups (#6966) Add screenshot nightly test for wkorg (#7030) Workaround for WebGL crash for datasets with many segmentation layers (#6995) Fix download of public annotation, include access ctx in user cache key (#7025) Fix that changing a segment color could lead to a crash (#7000) Add more error chaining to annotation download (#7023) Guard against NaNs in shader (#7018) Store editable mappings in multiple fossildb columns+keys (#6903) Context action to move tree to group (#7005) ...
URL of deployed dev instance (used for testing):
TODO
Steps to test:
l4dense_motta_et_al_dev_v2)python tools/migrate-editable-mappings/migrate-editable-mappings.py -vw localhost:7155, then reload the same annotation and see that everything is there and still works as expectedIssues: