Stop loading tutorial data by default

[jhamman]

• Tests added
• Fully documented, including whats-new.rst for all changes and api.rst for new API

In working on an xarray/dask tutorial, I've come to realize we eagerly load the tutorial datasets in xarray.tutorial.load_dataset. I'm going to just say that I don't think we should do that but I could be missing some rational. I didn't open an issue so please feel free to share thoughts here.

One option would be to create a new function (xr.tutorial.open_dataset) that does what I'm suggesting and then slowly deprecate tutorial.load_dataset. Thoughts?

xref: dask/dask-examples#51

[pep8speaks]

Hello @jhamman! Thanks for updating the PR.

• There are no PEP8 issues in the file xarray/tests/test_tutorial.py !

• There are no PEP8 issues in the file xarray/tutorial.py !

Comment last updated on November 05, 2018 at 14:17 Hours UTC

[shoyer]

Our current tutorial datasets are 8MB and 17MB, which is pretty small. You'll definitely get better performance loading datasets of this size into NumPy arrays.

[jhamman]

@shoyer - absolutely we'll get better performance with numpy arrays in this case. So I'm trying to use our tutorial datasets for some examples with dask (dask/dask-examples#51). The docstring for the load_dataset function states that we can pass kwargs on to the open_dataset function but if we pass chunks to the load_dataset call currently, we still get data back as numpy arrays. We have some other options here:

1. if chunks is a kwargs, return a dataset with data as persisted dask arrays
2. provide a second function to handle returning datasets using the same logic as open_dataset (caching, dask arrays, lazy loading, etc.)
3. tell people (like me) to rechunk the dataset after the fact

(3) won't require any changes but makes it a little harder to connect the typical use pattern of open_dataset with tutorial.load_dataset.

[shoyer]

OK, that seems reasonable. The default behavior should cache the arrays loaded with NumPy anyways. I would not be opposed to renaming this to open_dataset, either.

…

On Sun, Nov 4, 2018 at 9:19 AM Joe Hamman ***@***.***> wrote: @shoyer <https://github.com/shoyer> - absolutely we'll get better performance with numpy arrays in this case. So I'm trying to use our tutorial datasets for some examples with dask (dask/dask-examples#51 <dask/dask-examples#51>). The docstring for the load_dataset function states that we can pass kwargs on to the open_dataset function but if we pass chunks to the load_dataset call currently, we still get data back as numpy arrays. We have some other options here: 1. if chunks is a kwargs, return a dataset with data as persisted dask arrays 2. provide a second function to handle returning datasets using the same logic as open_dataset (caching, dask arrays, lazy loading, etc.) 3. tell people (like me) to rechunk the dataset after the fact (3) won't require any changes but makes it a little harder to connect the typical use pattern of open_dataset with tutorial.load_dataset. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2538 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKS1mBjbk7l2qXi4EqFtMGdvDDoPJHaks5uryGUgaJpZM4YM5-d> .

[shoyer]

The default behavior should cache the arrays loaded with NumPy anyways.

Sorry, to be clear what I meant here is that by default arrays loaded with NumPy get cached after the first/access/operation. Not that we need to preserve the existing behavior of load_dataset().

[jhamman]

@shoyer - I think I was tracking with you. I've gone ahead and deprecated the current load_dataset in favor of the open_dataset name. The switch is accompanied by a change in behavior as well.