Fix optimize for chunked DataArray

[TomAugspurger]

Previously we generated in invalidate Dask task graph, becuase the lines
removed here dropped keys that were referenced elsewhere in the task
graph. The original implementation had a
comment indicating that this was to cull:

xarray/xarray/core/variable.py

Line 384 in 502a988

results = {k: v for k, v in results.items() if k[0] == name} # cull

Just spot-checking things, I think we're OK here though. Something like
dask.visualize(arr[[0]], optimize_graph=True) indicates that we're OK.

• Closes dask.optimize on xarray objects #3698
• Tests added
• Passes isort . && black . && mypy . && flake8
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[max-sixty]

Thanks @TomAugspurger !
Merging so we can get this into 0.16.1

[keewis]

with the merge we have a test failure:

_______________________ test_persist_Dataset[<lambda>1] ________________________

persist = <function <lambda> at 0x7fe8d1646048>

    @pytest.mark.parametrize(
        "persist", [lambda x: x.persist(), lambda x: dask.persist(x)[0]]
    )
    def test_persist_Dataset(persist):
        ds = Dataset({"foo": ("x", range(5)), "bar": ("x", range(5))}).chunk()
        ds = ds + 1
        n = len(ds.foo.data.dask)
    
        ds2 = persist(ds)
    
>       assert len(ds2.foo.data.dask) == 1
E       AssertionError: assert 2 == 1
E        +  where 2 = len(<dask.highlevelgraph.HighLevelGraph object at 0x7fe8c45f4518>)
E        +    where <dask.highlevelgraph.HighLevelGraph object at 0x7fe8c45f4518> = dask.array<add, shape=(5,), dtype=int64, chunksize=(5,), chunktype=numpy.ndarray>.dask
E        +      where dask.array<add, shape=(5,), dtype=int64, chunksize=(5,), chunktype=numpy.ndarray> = <xarray.DataArray 'foo' (x: 5)>\ndask.array<add, shape=(5,), dtype=int64, chunksize=(5,), chunktype=numpy.ndarray>\nDimensions without coordinates: x.data
E        +        where <xarray.DataArray 'foo' (x: 5)>\ndask.array<add, shape=(5,), dtype=int64, chunksize=(5,), chunktype=numpy.ndarray>\nDimensions without coordinates: x = <xarray.Dataset>\nDimensions:  (x: 5)\nDimensions without coordinates: x\nData variables:\n    foo      (x) int64 dask.array<chunksize=(5,), meta=np.ndarray>\n    bar      (x) int64 dask.array<chunksize=(5,), meta=np.ndarray>.foo

/home/vsts/work/1/s/xarray/tests/test_dask.py:943: AssertionError

does anyone know why that happens?

[max-sixty]

Hmmm. It did pass before the merge from master:

[TomAugspurger]

Huh, I'm able to reproduce locally. Looking into it now.

[TomAugspurger]

Might be best to proceed with #4434 for now. I'll need to give this a bit of thought.

[max-sixty]

Might be best to proceed with #4434 for now. I'll need to give this a bit of thought.

OK, as you wish, I'll merge if that passes.

But your change did pass before the merge. Could it be a conflict (in functionality, not git) with recent changes on master?

[TomAugspurger]

I'm not sure, but I don't think so. It's strange that it didn't fail on the pull request.

…

On Thu, Sep 17, 2020 at 8:51 PM Maximilian Roos ***@***.***> wrote: Might be best to proceed with #4434 <#4434> for now. I'll need to give this a bit of thought. OK, as you wish, I'll merge if that passes. But your change did pass before the merge. Could it be a conflict (in functionality, not git) with recent changes on master? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4432 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOISNY5A5N2A44YR2ZMLSGK4JTANCNFSM4RQ6OP2Q> .

[keewis]

it did, the first failing commit is 381aaf8

Edit: sorry, you're right, the first commit should have also failed. Not sure why that happened, and we can't really check because the build logs were already deleted.