Daskify NIR reflectance calculations #59
Conversation
This will fix saving of the LUT for Py2.7
| if np.isscalar(tb_near_ir): | ||
| tb_nir = np.array([tb_near_ir, ]) | ||
| else: | ||
| tb_nir = np.array(tb_near_ir) | ||
| tb_nir = np.asanyarray(tb_near_ir) |
There was a problem hiding this comment.
Does np.asanyarray convert dask arrays to numpy arrays?
There was a problem hiding this comment.
It does:
In [1]: import dask.array as da
In [2]: a = da.arange(5., chunks=2)
In [3]: import numpy as np
In [4]: type(np.asanyarray(a))
Out[4]: numpy.ndarray
There was a problem hiding this comment.
As mentioned on slack, I'd prefer if everything could be kept as dask arrays for as long as possible and if things have to be computed they should be done in a map_blocks or delayed function. As @pnuu mentioned on slack converting to numpy here made the code faster and use less memory in his test cases. I'm fairly certain this is because the array is used later on in a non-dask-friendly way and the dask array is actually being computed multiple times. I think I pointed them out on slack during the PCW.
There was a problem hiding this comment.
I'm fine with merging. Then we could also merge pytroll/satpy#529 so this PR would be usable via SatpY.
There was a problem hiding this comment.
I think there hasn't been anything that didn't work since I last touched this, just that it might not be quite optimal in performance and how things should be done with dask.
There was a problem hiding this comment.
Fine, I will just try do some system testing off line first to be sure. Any suggested recipes/tests @pnuu ?
|
I tested four RGBs: |
This PR makes the internals of near-infrared calculations to use dask. All the existing code unit tests pass, so hopefully this won't break anything for existing non-SatPy users.
git diff origin/master **/*py | flake8 --diff