Merge remote-tracking branch 'upstream/master' into yohai-ds_scatter

* upstream/master: Feature: N-dimensional auto_combine (pydata#2553) Support HighLevelGraphs (pydata#2603) Bump cftime version in doc environment (pydata#2604) use keep_attrs in binary operations II (pydata#2590) Temporarily mark dask-dev build as an allowed failure (pydata#2602) Fix wrong error message in interp() (pydata#2598) Add dayofyear and dayofweek accessors (pydata#2599) Fix h5netcdf saving scalars with filters or chunks (pydata#2591) Minor update to PR template (pydata#2596) Zarr consolidated (pydata#2559) fix examples (pydata#2581) Fix typo (pydata#2578) Concat docstring typo (pydata#2577) DOC: remove example using Dataset.T (pydata#2572) python setup.py test now works by default (pydata#2573) Return slices when possible from CFTimeIndex.get_loc() (pydata#2569) DOC: fix computation.rst (pydata#2567)
yohai · Dec 16, 2018 · 6975b9e · 6975b9e
2 parents 6440365 + 9e8707d
commit 6975b9e
Show file tree

Hide file tree

Showing 29 changed files with 1,182 additions and 310 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,3 +1,5 @@
- - [ ] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
- - [ ] Tests added (for all bug fixes or enhancements)
- - [ ] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)
+<!-- Feel free to remove check-list items aren't relevant to your change -->
+
+ - [ ] Closes #xxxx
+ - [ ] Tests added
+ - [ ] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API
diff --git a/ci/requirements-py36-zarr-dev.yml b/ci/requirements-py36-zarr-dev.yml
@@ -18,4 +18,4 @@ dependencies:
   - pip:
     - coveralls
     - pytest-cov
-    - git+https://github.com/alimanfoo/zarr.git
+    - git+https://github.com/zarr-developers/zarr.git
diff --git a/doc/computation.rst b/doc/computation.rst
@@ -164,9 +164,8 @@ Note that rolling window aggregations are faster when bottleneck_ is installed.
 
 We can also manually iterate through ``Rolling`` objects:
 
-.. ipython:: python
+.. code:: python
 
-   @verbatim
    for label, arr_window in r:
       # arr_window is a view of x
 
@@ -298,9 +297,9 @@ operations. The default result of a binary operation is by the *intersection*
 If coordinate values for a dimension are missing on either argument, all
 matching dimensions must have the same size:
 
-.. ipython:: python
+.. ipython::
+    :verbatim:
 
-    @verbatim
     In [1]: arr + xr.DataArray([1, 2], dims='x')
     ValueError: arguments without labels along dimension 'x' cannot be aligned because they have different dimension size(s) {2} than the size of the aligned dimension labels: 3
 

diff --git a/doc/dask.rst b/doc/dask.rst
diff --git a/doc/environment.yml b/doc/environment.yml
@@ -16,7 +16,7 @@ dependencies:
   - zarr=2.2.0
   - iris=2.1.0
   - flake8=3.5.0
-  - cftime=1.0.0
+  - cftime=1.0.3.4
   - bottleneck=1.2
   - sphinx=1.7.6
   - numpydoc=0.8.0

diff --git a/doc/io.rst b/doc/io.rst
@@ -635,6 +635,35 @@ For example:
     Not all native zarr compression and filtering options have been tested with
     xarray.
 
+Consolidated Metadata
+~~~~~~~~~~~~~~~~~~~~~
+
+Xarray needs to read all of the zarr metadata when it opens a dataset.
+In some storage mediums, such as with cloud object storage (e.g. amazon S3),
+this can introduce significant overhead, because two separate HTTP calls to the
+object store must be made for each variable in the dataset.
+With version 2.3, zarr will support a feature called *consolidated metadata*,
+which allows all metadata for the entire dataset to be stored with a single
+key (by default called ``.zmetadata``). This can drastically speed up
+opening the store. (For more information on this feature, consult the
+`zarr docs <https://zarr.readthedocs.io/en/latest/tutorial.html#consolidating-metadata>`_.)
+
+If you have zarr version 2.3 or greater, xarray can write and read stores
+with consolidated metadata. To write consolidated metadata, pass the
+``consolidated=True`` option to the
+:py:attr:`Dataset.to_zarr <xarray.Dataset.to_zarr>` method::
+
+    ds.to_zarr('foo.zarr', consolidated=True)
+
+To read a consolidated store, pass the ``consolidated=True`` option to
+:py:func:`~xarray.open_zarr`::
+
+    ds = xr.open_zarr('foo.zarr', consolidated=True)
+    
+Xarray can't perform consolidation on pre-existing zarr datasets. This should
+be done directly from zarr, as described in the
+`zarr docs <https://zarr.readthedocs.io/en/latest/tutorial.html#consolidating-metadata>`_.
+
 .. _io.cfgrib:
 
 GRIB format via cfgrib
@@ -678,7 +707,7 @@ Formats supported by PseudoNetCDF
 ---------------------------------
 
 xarray can also read CAMx, BPCH, ARL PACKED BIT, and many other file
-formats supported by PseudoNetCDF_, if PseudoNetCDF is installed. 
+formats supported by PseudoNetCDF_, if PseudoNetCDF is installed.
 PseudoNetCDF can also provide Climate Forecasting Conventions to
 CMAQ files. In addition, PseudoNetCDF can automatically register custom
 readers that subclass PseudoNetCDF.PseudoNetCDFFile. PseudoNetCDF can

diff --git a/doc/reshaping.rst b/doc/reshaping.rst
@@ -18,14 +18,13 @@ Reordering dimensions
 ---------------------
 
 To reorder dimensions on a :py:class:`~xarray.DataArray` or across all variables
-on a :py:class:`~xarray.Dataset`, use :py:meth:`~xarray.DataArray.transpose` or the
-``.T`` property:
+on a :py:class:`~xarray.Dataset`, use :py:meth:`~xarray.DataArray.transpose`:
 
 .. ipython:: python
 
     ds = xr.Dataset({'foo': (('x', 'y', 'z'), [[[42]]]), 'bar': (('y', 'z'), [[24]])})
     ds.transpose('y', 'z', 'x')
-    ds.T
+    ds.transpose()  # reverses all dimensions
 
 Expand and squeeze dimensions
 -----------------------------

diff --git a/doc/time-series.rst b/doc/time-series.rst
@@ -263,14 +263,16 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
    da.sel(time=slice('0001-05', '0002-02'))
 
 - Access of basic datetime components via the ``dt`` accessor (in this case
-  just "year", "month", "day", "hour", "minute", "second", "microsecond", and
-  "season"): 
+  just "year", "month", "day", "hour", "minute", "second", "microsecond",
+  "season", "dayofyear", and "dayofweek"): 
 
 .. ipython:: python
 
    da.time.dt.year
    da.time.dt.month
    da.time.dt.season
+   da.time.dt.dayofyear
+   da.time.dt.dayofweek
 
 - Group-by operations based on datetime accessor attributes (e.g. by month of
   the year):

diff --git a/doc/whats-new.rst b/doc/whats-new.rst
@@ -21,7 +21,7 @@ What's New
     always be available to python 2.7 users. For more information see the
     following references
 
-  - `Xarray Github issue discussing dropping Python 2 <https://github.com/pydata/xarray/issues/1829>`__
+     - `Xarray Github issue discussing dropping Python 2 <https://github.com/pydata/xarray/issues/1829>`__
   - `Python 3 Statement <http://www.python3statement.org/>`__
   - `Tips on porting to Python 3 <https://docs.python.org/3/howto/pyporting.html>`__
 
@@ -36,9 +36,23 @@ Breaking changes
 Enhancements
 ~~~~~~~~~~~~
 
+- Ability to read and write consolidated metadata in zarr stores (:issue:`2558`).
+  By `Ryan Abernathey <https://github.com/rabernat>`_.
+- :py:class:`CFTimeIndex` uses slicing for string indexing when possible (like
+  :py:class:`pandas.DatetimeIndex`), which avoids unnecessary copies.
+  By `Stephan Hoyer <https://github.com/shoyer>`_
+- Like :py:class:`pandas.DatetimeIndex`, :py:class:`CFTimeIndex` now supports
+  "dayofyear" and "dayofweek" accessors (:issue:`2597`).  By `Spencer Clark
+  <https://github.com/spencerkclark>`_.
+- Support Dask ``HighLevelGraphs`` by `Matthew Rocklin <https://matthewrocklin.com>`_.
+
+
 Bug fixes
 ~~~~~~~~~
 
+- Fix h5netcdf saving scalars with filters or chunks (:issue:`2563`).
+  By `Martin Raspaud <https://github.com/mraspaud>`_.
+
 .. _whats-new.0.11.0:
 
 v0.11.0 (7 November 2018)
@@ -52,15 +66,15 @@ Breaking changes
   - ``Dataset.T`` has been removed as a shortcut for :py:meth:`Dataset.transpose`.
     Call :py:meth:`Dataset.transpose` directly instead.
   - Iterating over a ``Dataset`` now includes only data variables, not coordinates.
-    Similarily, calling ``len`` and ``bool`` on a ``Dataset`` now  
+    Similarily, calling ``len`` and ``bool`` on a ``Dataset`` now
     includes only data variables.
   - ``DataArray.__contains__`` (used by Python's ``in`` operator) now checks
-    array data, not coordinates. 
+    array data, not coordinates.
   - The old resample syntax from before xarray 0.10, e.g.,
     ``data.resample('1D', dim='time', how='mean')``, is no longer supported will
     raise an error in most cases. You need to use the new resample syntax
     instead, e.g., ``data.resample(time='1D').mean()`` or
-    ``data.resample({'time': '1D'}).mean()``. 
+    ``data.resample({'time': '1D'}).mean()``.
 
 
 - New deprecations (behavior will be changed in xarray 0.12):
@@ -97,13 +111,13 @@ Breaking changes
     than by default trying to coerce them into ``np.datetime64[ns]`` objects.
     A :py:class:`~xarray.CFTimeIndex` will be used for indexing along time
     coordinates in these cases.
-  - A new method :py:meth:`~xarray.CFTimeIndex.to_datetimeindex` has been added 
+  - A new method :py:meth:`~xarray.CFTimeIndex.to_datetimeindex` has been added
     to aid in converting from a  :py:class:`~xarray.CFTimeIndex` to a
     :py:class:`pandas.DatetimeIndex` for the remaining use-cases where
     using a :py:class:`~xarray.CFTimeIndex` is still a limitation (e.g. for
     resample or plotting).
   - Setting the ``enable_cftimeindex`` option is now a no-op and emits a
-    ``FutureWarning``. 
+    ``FutureWarning``.
 
 Enhancements
 ~~~~~~~~~~~~
@@ -146,6 +160,8 @@ Enhancements
   to returning (and is now deprecated). This was changed in order to facilitate
   using tutorial datasets with dask.
   By `Joe Hamman <https://github.com/jhamman>`_.
+- ``DataArray`` can now use ``xr.set_option(keep_attrs=True)`` and retain attributes in binary operations,
+  such as (``+, -, * ,/``). Default behaviour is unchanged (*Attributes will be dismissed*). By `Michael Blaschek <https://github.com/MBlaschek>`_
 
 Bug fixes
 ~~~~~~~~~
@@ -190,7 +206,7 @@ Bug fixes
   the dates must be encoded using cftime rather than NumPy (:issue:`2272`).
   By `Spencer Clark <https://github.com/spencerkclark>`_.
 
-- Chunked datasets can now roundtrip to Zarr storage continually 
+- Chunked datasets can now roundtrip to Zarr storage continually
   with `to_zarr` and ``open_zarr`` (:issue:`2300`).
   By `Lily Wang <https://github.com/lilyminium>`_.
 

diff --git a/examples/xarray_multidimensional_coords.ipynb b/examples/xarray_multidimensional_coords.ipynb