Porting hazen to python 3.9

[tomaroberts]

See #142.

Currently, for python3.9, hazen requires conda package management as not all wheels working in pip. hazen can be run in 3.9, but some tools failing because of changes to old libraries, and possibly other issues.

TODO:

• Fix cv2 related error. OpenCV has changed syntax. Now recommends working on binary image prior to findContours. Might affect our results? See here: https://docs.opencv.org/4.x/d4/d73/tutorial_py_contours_begin.html
• others...

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
hazenlib
init.py	129	62	52%	135, 139, 149, 154, 190–193, 198–203, 214, 217–224, 244–246, 264–266, 285–287, 296, 300–303, 307, 314–364
ghosting.py	156	50	68%	24, 89–90, 94, 104–105, 132–133, 142–143, 198–226, 233–251, 255
relaxometry.py	317	90	72%	238–256, 630, 689–691, 745, 793–815, 833–848, 1173–1176, 1185–1188, 1200–1213, 1216–1221, 1233–1263
slice_position.py	110	20	82%	73–74, 99, 183, 190–203, 211, 219–220, 231
slice_width.py	289	45	84%	24, 27, 31, 35–40, 106, 277, 282–283, 289, 294, 357–358, 593–632, 668–671, 674–677
snr.py	165	48	71%	42–47, 75, 209–219, 262–266, 304–315, 321–329, 357–368, 392, 400–402, 405
spatial_resolution.py	294	99	66%	87–145, 159, 251, 291–298, 326, 457–490, 517–534
tools.py	75	6	92%	43–50, 89
uniformity.py	79	22	72%	69–70, 77, 111–123, 135–137, 140, 144–147, 155
TOTAL	1650	442	73%

Tests	Skipped	Failures	Errors	Time
170	0 💤	0 ❌	0 🔥	124.706s ⏱️

[tomaroberts]

Reminder: OpenCV findContours() changed the number of output arguments between versions... Unhelpful -_-

https://stackoverflow.com/a/58244568

[tomaroberts]

tests/resolution/RESOLUTION passes
tests/resolution/eastkent fails
tests/resolution/philips fails

eastkent and philips failing here:

These 5 all have rescale_to_byte(), thresh() and find_square() in common. I'm pretty sure find_square is working, otherwise the RESOLUTION test set wouldn't pass.

In the eastkent/philips test_data, I think something might be happening with rescale_to_byte() or thresh() which is causing find_square() to output coordinates which don't match the test values. e.g.:

I've also noticed that there is a subtle different in the images output between hazen3.6 and hazen3.9:

Left = old hazen (3.6)
Right = new hazen (3.9)

The 3.9 images are smoother when loaded in – are they supposed to look like left or right?! Is the 'shimmer' in the 3.6 images an MR artefact or python artefact?

[tomaroberts]

Should add, current spatial_resolution results:

RESOLUTION
perfect match between py3.6/py3.9

eastkent
py3.6: FE = 0.98 / PE = 0.99
py3.9: FE = 0.99 / PE = 0.99

philips:
py3.6: FE = 0.49 / PE = 0.54
py3.9: FE = 0.49 / PE = 0.49

So pretty similar, seemingly indicating numerical/subtle threshold or contrast issue.

[hshuaib90]

What does this file do?

[tomaroberts]

Sorry, shouldn't have spread between two places. Bit more context in #142.

Basically, installing some of our libraries (mainly scikit-image) doesnt work on M1 silicon via pip. So instead I made the venv in miniforge (just a smaller version of conda). The environment.yml is the equivalent to our normal requirements.txt.

I specifically made most of it install via pip anyway, so it was easy to see which ones work via pip and which only work via conda.

[hshuaib90]

Sorry for being a bit slow here. But doesn't the dockerisation of Hazen make this issue a moot point?

[hshuaib90]

Sorry I should have made this comment on the issue when you first raised it.

[hshuaib90]

tests/resolution/RESOLUTION passes tests/resolution/eastkent fails tests/resolution/philips fails

eastkent and philips failing here:

These 5 all have rescale_to_byte(), thresh() and find_square() in common. I'm pretty sure find_square is working, otherwise the RESOLUTION test set wouldn't pass.

I just checked and all the resolution tests are failing with your recent commit #866dc22 . find_square is the only code you've changed from develop branch according to the GitHub PR.

The 3.9 images are smoother when loaded in – are they supposed to look like left or right?! Is the 'shimmer' in the 3.6 images an MR artefact or python artefact?

I'm not sure why the appearance of the images changes between Python versions, but it is worrying. Need to see if anyone else can reproduce locally.

[tomaroberts]

I just checked and all the resolution tests are failing with your recent commit #866dc22 . find_square is the only code you've changed from develop branch according to the GitHub PR.

These are the latest results I'm getting from running pytest tests/

[tomaroberts]

I've just noticed the Git Actions pytest section where it's failing all the spatial_resolution tests, as you say.

I think the output is different to the tests I'm running locally on my laptop because the Git Actions test is building the code from requirements.txt with py3.6 and using the old OpenCV version, meaning that my alteration to findContours() is causing it to fail.

[tomaroberts]

@hshuaib90

I'm not sure why the appearance of the images changes between Python versions, but it is worrying. Need to see if anyone else can reproduce locally.

Have discovered that the matplotlib version makes a significant different on how the images are displayed...!

See images below, generated using identical python3.9 venvs, only difference is matplotlib versions:

Top: matplotlib==3.1.3 (which is used in main hazen version)
Bottom: matplotlib==3.4.3

Matplotlib shouldn't be affecting the values in the underlying dicom pixel_arrays however as it's just for plotting, so the discrepancies in hazen output values is something else.

At the moment, I'm pretty sure that the hazen code is behaving consistently between the 3.6 build and my 3.9-miniforge build, but a subtle change in one of the libraries is giving us slightly different results, hence the tests are failing.

My feeling is that we just need to update the tests to reflect these slight changes in values.

environment-mplib3.1.3.yml.txt

[tomaroberts]

Updated requirements.txt to specify mistune version as m2r install was causing error. See: miyakogi/m2r#66

[tomaroberts]

December release of scikit-image added arm64 support so hazen install via pip now works on M1 Macs \o/

Thanks to @laurencejackson for additional assistance in OpenSSL installation, required for psycopg2-binary package and killing conda.