Implement Dask collection interface

[rpmanser]

This pull request implements the Dask collection interface for the Quantity class and adds convenience methods compute(), persist(), and visualize(). As is, only the convenience methods are wrapped to check that the magnitude of the Quantity is a dask array, which should cover current use cases. Ping @jthielen since we've been discussing this.

I have not tested persist() on a HPC cluster, which will have different behavior than when it is called on a single machine (e.g., desktop). Is there a way to integrate tests for distributed cases? If not, I can test this independently if needed.

• Closes Integration with Dask (add tests; implement the Dask collection interface on Quantity) #883
• Executed black -t py36 . && isort -rc . && flake8 with no errors
• The change is fully covered by automated unit tests
• Documented in docs/ as appropriate
• Added an entry to the CHANGES file

[keewis]

xarray's distributed tests might be useful as an example? They use the loop fixture and the Cluster and Client context managers to set up a fake distributed cluster.

[rpmanser]

I'll take a look and see if I can setup something similar. Thanks!

[rpmanser]

I made all of the suggested changes and added tests for visualize and explicitly checking that compute() and persist() return the same results for the single machine case. I will be working on distributed tests next.

[hgrecco]

This is looking really good, please feel free to ping me when you want me to review it and merge it.

I would like to take the opportunity to ask a question: When are we going to provide an external package (e.g. pint-pandas) and when are we going to include it in pint. I am not suggesting to create pint-dask but rather to start building a rationale about pint extensions.

[dopplershift]

My concern with extension packages is that they won't necessarily receive the same level of maintenance as the core pint library. As a maintainer of a downstream library, I'm always concerned about adding more dependencies.

What do people see as the benefits of having external packages?

[jthielen]

@rpmanser As far as implementing the collection interface and testing it, this looks great! One small suggestion I would have is referring to dask/dask#4583 in the NumPy doc example, since it is Dask holding us back from the usual "multiply on the right" construction. Also, I think this gets enough of #883 taken care of to consider it closed, but I think a follow-up issue should be raised as a reminder to finish up tests once dask/dask#4583 is resolved (masked arrays and numpy/numpy#15200 are in the same position).

One bigger-picture question/discussion point: in pydata/xarray#525 and #883 (and in other discussions), I was thinking of Pint Quantities "acting like a Dask Array" or "being a duck Dask Array" would occur if and only if the Quantity is wrapping a Dask Array. That is, a library like xarray could recognize a suitable Dask Array-like by a test like is_duck_array(data) and hasattr(data, "__dask_graph__") or similar, and this test could be True if the magnitude of the Quantity was a Dask Array and False if not. However, with the implementation here, a Pint Quantity will always be a duck Dask Array since the Dask interface methods will always be available (though of course not always usable). Also, in seeing the implementation here, I've realized that a hacky implementation of these methods through attachment in __new__ or routing through __getattr__ would be needed to achieve my initial idea (rather than a simple decorator check).

So, with that in mind, is it okay for xarray (and any other libraries depending on the duck Dask Array interfacing of Quantity) to have Quantity always be a duck Dask Array, or do these interface methods need to be "hidden" and only available when the magnitude of a Quantity is a Dask Array? While there ostensibly is another option (a "DaskQuantity" subclass that has these methods available), I think that would make the situation worse, since too much magic/special casing may be required to get a DaskQuantity as the result of arbitrary operations between Dask Arrays and regular Quantities.

[rpmanser]

This is ready for your review @hgrecco. Looks like the travis CI has been failing due to the -rc flags given to isort, but otherwise black -t py36 . && isort -rc . && flake8 runs without error.

[dopplershift]

@jthielen In the absence of documentation on some kind of protocol, this might be one of those things that has to be worked out in practice. Having said that, personally I don't consider __getattr__ a hack for this use case.

[jthielen]

@jthielen In the absence of documentation on some kind of protocol, this might be one of those things that has to be worked out in practice. Having said that, personally I don't consider __getattr__ a hack for this use case.

Sounds good. I've also asked over in xarray (pydata/xarray#4208), so we'll see if we get any feedback there.

[hgrecco]

bors r+

[hgrecco]

Congratulations @rpmanser for the awesome work!

[bors]

Build succeeded:

• continuous-integration/travis-ci/push

[rpmanser]

Congratulations @rpmanser for the awesome work!

Thanks, happy I could help!