Add check_pixels.py script #55
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
14 tasks
|
Testing last commit on idr-testing:omeroreadwrite: EDIT: oops - those are Screen IDs! - Need to run against Screens... |
|
Going again... with typos fixed!... |
|
After the weekend, checked logs... Looks like the loop above ran for a couple of days... Returning to screen... Looking at individual logs... Looks like all images were failing pixels check. Logged-in and re-ran - Failure to getPlane() causes script to crash - need to fix error handling |
|
Started loop above at Failures found so far:
|
|
All the following Screens ran to completion, with the number of Images representing the number of Plates in each Screen (1 Image per Plate). Others that didn't complete:
|
|
Progress... Only 1 of the 4 Screens completed: The error is coming from an Image in idr0016 Plate named Returning to the terminal screen where this was running, I see ConnectionLostException |
|
Test idr0090 Screen: Hangs at this point. Checking that image in webclient - also fails to load. |
|
Tried checking the rest of the idr0090 plates one at a time, but the first one just hangs... Decided to test the remainder manually in webclient. Here,
Summary - 9 Plates from idr0090 that are not viewable EDIT - these plates eventually did work OK. All passed for check_pixels below (Screen:2851) |
|
Updated |
|
Checking idr0004... Only 2 plates failing on idr0004...
|
|
All idr0013-ScreenB Plates are OK! Checked again idr0015 (plates were hanging above) and all looks good now So, to summarise:
|
|
Try testing more than 1 image from each Plate... 5 images per plate from idr0004 Wow, last Image failed! Check all images from that Plate: Every Well after
|
|
Better check more Images from each Plate (50 images) in case there are others from idr0004 with this issue... Looks like all these are from the plate Also see errors like "Error: Image:693514 unpack requires a buffer of 688128 bytes" These images look corrupted the same as for |
|
Try a bunch of images from each plate of idr0010... Ran to completion: All these Errors come from the same Plate (we checked 50 images from each Plate) which is broken in IDR: https://idr.openmicroscopy.org/webclient/?show=plate-5894 Summary: no unexpected errors for idr0010. |
|
Let's run for 20 Images per Fileset for all the others... (5 images per Plate was enough to pick up idr0004 issues above) idr0011-screenA has 20 images from each of the 4 known failing plates. None for idr0011-screenB or C/D/E, 13:57: idr0013 screenA - 16:00 - 2 of the 3 plates parsed so far... |
|
Run completed - found 1 plate from idr0036 where all Images failed (20 checked). All other plates and Screens contain no new Errors. |
|
Failing Plate for idr0036 - rendered images don't match Thumbnails: EDIT: Ran this psql to update... Tried to view an Image, which should trigger memo file regeneration... Still not viewable a day later after several attempts to view. Searching logs doesn't reveal much. |
|
As discussed in IDR meeting today, we need to scale up We want to check ALL images in each Fileset. |
|
Start with idr0004... On idr-testing:omeroreadwrite... EDIT: took 2hours 15 mins. These are the 35 Images (2 channels each) from plate There are 38 images from P115 and P124 with the "unpack" issue: 70 + 38 accounts for all Errors for idr0004: |
|
For most studies, we can run a whole Screen at at time on one of the proxy servers. E.g. lets get all the Plates for idr0013 and idr0016... Edited both those files to remove the first line. Total of 923 Plates... Copy this file from omeroreadwrite onto parent idr-testing server: Instead of EDIT 20:12: But file is empty on EDIT: 28th.. 9:52... - we have log on Currently processing the 2nd Plate on -3... 28th: 9:46 But then, the log disappeared (file is now empty). Probably the issue is that the On idr-testing server, I do Let's kill that screen and re-run (see below...) |
|
Running idr0010 on idr-testing: omeroreadonly-1, as wmoore EDIT: 20:27... 28th 9:50... 29th 9:25... 30th 12:14... About a quarter of the way through idr0010 after 3 days (~ETA 9th December) 4th Dec. Still no Errors 6th Dec - Done! Only Error is Plate |
|
Check idr0033 on 1st Dec: 20.5 hours, -> predict total time to be 30 days! (ETA ~30th December) 4th Dec 6th Dec Looks like process got faster after parallel running of idr0013 and idr0016 completed. Done now with no Errors: |
|
On omeroreadonly-1, check idr0035... 11th Dec - all done, no Errors |
|
Remaining studies...
Plates for idr0015 and idr0090, on idr-testing omeroreadwrite Edited both files to remove first row. Then... idr0090 first... Copy the ids.txt onto parent idr-testing server.. Run... 10:31... 20th Dec check logs... Errors... omeroreadwrite... only 2 Images NOT from know broken plate: http://localhost:1080/webclient/?show=plate-4653 TODO: This image http://localhost:1080/webclient/?show=image-2013055 looks OK in webclient (idr-testing) - Need to run check_pixels on just this image. Fails on idr-testing but OK on idr-next... Other failure is OK on idr-next. This was a temp failure on idr-testing, but worked on idr0125-pilot. omeroreadonly-1 Lots of ConnectionLostErrors. and a bunch of Image:1977981 (idr0015) looks OK in webclient - No error Check these again, an Image at a time... No errors found! omeroreadonly-2 Too many to check manually! Just check_pixels... omeroreadonly-3 - Similar pattern as above... Checked all these failures with omeroreadwrite-4 All OK with: |
|
Checking idr0091 on idr-testing:omeroreadwrite... 13th December Noticed pixeltype bug - see ome/omero-cli-zarr#157 - re-exported data etc at IDR/idr-metadata#650 (comment) EDIT 3rd Jan... All of the |
Same for datasets.txt >> objects.txt Copied onto the parent host node, then... |
|
check_pixels Error: - Ignoring ApiUsageException (from getPlane() on Big Images):
Need to focus on No From idr0009, on Plates where first Well is blank. Also more below. No other mismatches. |
|
The last 2 commits are tested at IDR/idr-metadata#685 (comment) |
|
We want to use On idr-next proxy... |
|
Logs files are not being created as expected: We have 2 Used Ctrl-D, then I see the 2nd screen... Last line is repeated when I hit Enter, but doesn't update. Eventually quit with... |
|
Test again with smaller number of jobs - 2! With a smaller Plate (from idr0011): Got the same as above: Let's go back to running as above, with an Input file of IDs... With Plate IDs just from idr0013... Exactly as above (except for log file)... Then Don't know if this was output under This took a while but then started showing output... screen -r |
will-moore
force-pushed
the
check_pixels
branch
from
e7e54a3 to
1f4c0ba
Compare
|
Seb: "At leat 1 of these 5 servers is the omeroreadwrite that end-users should never access So, reducing the Also, want to run above with Killed the Screen running above... deleted logs. reduced servers in nodes: Ran again... Seems like we're not able to connect to Let's try idr-testing... Killed the screen, Deleted logs again. But |
|
Switch to using #62 screen -r See lots of threads starting the But we've not reached and |
|
Still running above... issues with logging-in: screen -r Edit - looking OK... |
|
After running for ~18 hours now... Seems to be running happily - need to check logs... On Checking both logs... EDIT: 12:25 Time to cancel this and start new run (see below)... |
|
Updated script at #62 to add on idr-next:omeroreadonly on proxy idr-next |
|
screen -r |
|
Checking logs... We see errors pretty quickly - probably corresponds to See almost as many |
|
At 13:24, updated to d6e0b1d while still running... |
|
No Only 93 Plates processed - 413 in total, would expect half to be processed here... Checking webclient at http://localhost:1080/webclient/ now gives But no Errors in Blitz log Let's restart the server, and run again - hopefully with better stack traces this time... |
|
Use |
|
screen -r EDIT - next day Something went wrong?! |
|
Trying again,
Seems we've run out of space on This is due to logs, including 71G of Which contains lots of... Going to delete that, restart the server and run parallel again, with lower settings... |
|
Seems that deleting that file doesn't free up space... even though we only have However, it looks like Edit: 13:13 (2 hours) |
|
Moving |
Closed
15 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This script allows validation of pixel values using
pixels.getPlanes()and comparing with the same image between the current server (that you are logged-in to) and IDR, using the same Image IDs.To be used as the final step in the Fileset replacement workflow, to validate that Images on test or "idr-next" server have the same pixel values as on IDR itself.
Can be used to validate all Images within a container. Supported Object types are "Screen", "Plate", "Project", "Dataset", "Image".
Option
--max-planesallows you to only check a subset of all planes for an Image. Otherwise, some images could take a very long time to check. By default we check ALL planes of each image.Output is written to a log file.