Docs/more fixes #2934

dcherian · 2019-05-02T02:43:29Z

partially addresses docs questions from new user #2909 , closes Link to dask documentation on chunks #2901, closes More efficient rolling with large dask arrays #2908

dcherian · 2019-05-02T02:59:18Z

doc/dask.rst

 Currently, Dask is an entirely optional feature for xarray. However, the
 benefits of using Dask are sufficiently strong that Dask may become a required
 dependency in a future version of xarray.

 For a full example of how to use xarray's Dask integration, read the
-`blog post introducing xarray and Dask`_.
+`blog post introducing xarray and Dask`_. More up-to-date examples
+may be found at the `Pangeo project's use-cases <http://pangeo.io/use_cases/index.html>`_.


Are there other examples?

We can link to the dask-examples repo: https://examples.dask.org/xarray.html

doc/dask.rst

dcherian · 2019-05-02T02:59:57Z

doc/dask.rst

@@ -145,8 +149,12 @@ explicit conversion step. One notable exception is indexing operations: to
 enable label based indexing, xarray will automatically load coordinate labels
 into memory.

+.. note::
+
+   Something about schedulers and running locally vs on a cluster. Do I start a cluster first? etc...


Need help here.

dcherian · 2019-05-02T03:00:25Z

doc/dask.rst

@@ -355,30 +397,6 @@ application.
    structure of a problem, unlike the generic speedups offered by
    ``dask='parallelized'``.

-Chunking and performance


combined with earlier chunk stuff

dcherian · 2019-05-02T03:00:44Z

doc/extending.rst

@@ -0,0 +1,90 @@
+.. _extending:
+


Moved "extending xarray" one level up

dcherian · 2019-05-02T03:01:08Z

doc/io.rst

@@ -15,82 +15,6 @@ format (recommended).
    import xarray as xr
    np.random.seed(123456)

-.. _io.pickle:


moved netCDF up for emphasis.

doc/quick-overview.rst

doc/why-xarray.rst

dcherian · 2019-05-23T21:56:32Z

Edited to reduce the number of changes so that it's easier to review.

doc/dask.rst

doc/index.rst

doc/why-xarray.rst

doc/dask.rst

dcherian

ready for review and merge.

doc/dask.rst

jhamman · 2019-10-04T18:34:16Z

Thanks @dcherian!

max-sixty · 2019-10-04T19:43:44Z

Thanks @dcherian !

* upstream/master: Fix concat bug when concatenating unlabeled dimensions. (pydata#3362) Rolling minimum dependency versions policy (pydata#3358) added geocube and rioxarray to related projects (pydata#3383) Lint (pydata#3373) Docs/more fixes (pydata#2934) Explicitly keep track of indexes with merging (pydata#3234)

* origin/fix/groupby: Fix concat bug when concatenating unlabeled dimensions. (pydata#3362) Rolling minimum dependency versions policy (pydata#3358) added geocube and rioxarray to related projects (pydata#3383) Lint (pydata#3373) Docs/more fixes (pydata#2934) Explicitly keep track of indexes with merging (pydata#3234)

@shoyer

commit b0c336f Author: Maximilian Roos <[email protected]> Date: Mon Oct 21 04:52:36 2019 -0400 Whatsnew for #3419 (#3422) * pyupgrade --py36-plus * Update xarray/core/nputils.py Co-Authored-By: crusaderky <[email protected]> * Update xarray/core/parallel.py Co-Authored-By: crusaderky <[email protected]> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <[email protected]> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <[email protected]> * whatsnew commit 2984415 Author: Rhys Doyle <[email protected]> Date: Mon Oct 21 01:17:47 2019 +0100 Revert changes made in #3358 (#3411) * Revert #3358 * Revision * Code review * Merge from master * Obsolescence note commit 3c462b9 Author: Maximilian Roos <[email protected]> Date: Sun Oct 20 20:16:58 2019 -0400 Python3.6 idioms (#3419) * pyupgrade --py36-plus * Update xarray/core/nputils.py Co-Authored-By: crusaderky <[email protected]> * Update xarray/core/parallel.py Co-Authored-By: crusaderky <[email protected]> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <[email protected]> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <[email protected]> commit 9886e3c Author: crusaderky <[email protected]> Date: Sun Oct 20 23:42:36 2019 +0100 Temporarily mark pseudonetcdf-3.1 as incompatible (#3420) commit 0f7ab0e Author: Dan Nowacki <[email protected]> Date: Thu Oct 17 14:13:44 2019 -0700 Fix and add test for groupby_bins() isnan TypeError. (#3405) * Fix and add test for groupby_bins() isnan TypeError. * Better testing * black commit 6cd50cc Author: pmallas <[email protected]> Date: Thu Oct 17 10:15:47 2019 -0400 Update where docstring to make return value type more clear (#3408) * Update where docstring to make return value type more clear The where docstring was a little confusing to me. I misunderstood "Same type as caller' to mean the values in the xarray not the xarray itself. I think this small change will clean it up for most users. Thanks. * Update xarray/core/common.py Co-Authored-By: Maximilian Roos <[email protected]> commit 55b1ac0 Author: keewis <[email protected]> Date: Thu Oct 17 05:13:39 2019 +0200 tests for arrays with units (#3238) * create the empty test file * add tests for data array aggregation functions * include pint in the ci * ignore missing type annotations for pint * really skip the tests if pint is not available * remove the reason from the importorskip call * test that the dataarray constructor does not strip the unit * convert every unit stripped warning to an error * work around pint not implementing np.allclose yet * remove the now unnecessary filterwarnings decorator * xfail all tests that depend on pint having a __array_function__ * treat nans as equal * implement tests for simple arithmetic operations * use param's id argument to assign readable names * add tests for sel() and isel() * add more readable names for the unary arithmetics * xfail every test that is not yet xfailing These don't pass because the constructor raises a unit stripped warning - fixed in pint#764. * only xfail if pint is not the current development version This is test is not really reliable, but sufficient for now. * always use lists instead of tuples for indexing * add tests for loc and squeeze * black * add names and xfail marks to the parameters * add tests for interp and interp_like * implement tests for reindex * remove the xfail marks where it is not clear yet that pint is to blame * add tests for reindex_like * don't pass the new DataArray to a kwarg * xfail if not on pint dev * refactor the tests * add tests for univariate and bivariate ufuncs * black * xfail aggregation only if pint does not implement __array_function__ yet * remove the global filterwarnings mark apparently, this caused the tests to change behavior, resulting in different errors, or causing tests to pass that should actually fail. * add a test case for the repr * create a pytest mark that explicitly requires pint's __array_function__ * also check the string representation in addition to the repr * add helpers for creating method tests * check that simple aggregation methods work * use format() instead of format strings * make sure the repr of method calls is different from functions * merge the two aggregation tests * explicitly check whether pint supports __array_function__ relying on versions is somewhat fragile. * provide a fallback for the new base quantity * check that no warning is raised for both with and without coords * also check that the repr works both with and without coords * wrap all aggregation function calls * xfail every call that fails because of something outside xarray * xfail tests related to dimension coordinates and indexes * use the dimensions from the original array * allow passing arguments to the method on call * add tests for comparisons * add tests for detecting, filling and dropping missing values * mark the missing value tests as requiring pint to support duck arrays * add tests for isin, where and interpolate_na * reformat unit ids and add a test parameter for compatible units * remove an unnecessary xfail * add tests for the top-level replication functions (*_like) * check for whatever pint does with *_like functions * add tests for combine_first * xfail the bivariate ufunc tests * xfail the call to np.median * move the top-level function tests out of the DataArray namespace class * add cumsum and cumprod to the list of aggregation functions * add tests for the numpy methods * check for equal units directly after checking the magnitude * add tests for content manipulation methods * add tests for comparing DataArrays (equals, indentical) * add a test for broadcast_equals * refactor the comparison operation tests * rewrite the strip, attach and assert_equal functions and add extract * preserve multiindex in strip and attach * attach the unit from element "data" as fallback * fix some small typos * compare QuantityScalar and QuantitySequence based on their values * make the isel test more robust * add tests for reshaping and reordering * unify the structure of the tests * mark the remaining tests as requiring a recent pint version, too * explicitly handle quantities as parameters * change the repr of the function / method wrappers * check whether __init__ and repr / str handle units in data and coords * generalize array_attach_units * move the redefinition of DimensionalityError * identify quantities using isinstance * typo * skip tests with a pint version without __array_function__ * compare DataArrays where possible * mark only the compatible unit as xfailing * preserve the name of data arrays * also attach units to x_mm * Test in more CI environments; documentation * What's New * remove a stale function * use Quantity directly for instance tests * explicitly set roll_coords to silence a deprecation warning * skip the whole module if pint does not implement __array_function__ the advantage is that now forgetting to decorate a test case is not possible. * allow to attach units using the mapping from extract_units * add tests for computation methods resampling fails until I figure out how to use it with non-datetime coords. * add tests for grouped operations * add a test for rolling_exp * add a todo note for the module level skip on __array_function__ * add a test for dot * use attach_units instead of manually attaching * modify the resample test to actually work * add a test for to_unstacked_dataset * update whats-new.rst and installing.rst * reformat the whats-new.rst entry * What's New commit 1f81338 Author: keewis <[email protected]> Date: Wed Oct 16 20:54:27 2019 +0200 Fixes to the resample docs (#3400) * add a missing newline to make sphinx detect the code block * update the link to the pandas documentation * explicitly state that this only works with datetime dimensions * also put the datetime dim requirement into the function description * add Series.resample and DataFrame.resample as reference * add the changes to whats-new.rst * move references to the bottom of the docstring commit 3f9069b Author: Joseph Hamman <[email protected]> Date: Mon Oct 14 14:22:08 2019 -0700 Revert to dev version commit 62943e2 Author: Joseph Hamman <[email protected]> Date: Mon Oct 14 12:51:46 2019 -0700 Release v0.14.0 commit 30472ec Author: Joseph Hamman <[email protected]> Date: Mon Oct 14 12:24:05 2019 -0700 updates for 0.14 release [black only] commit 4519843 Author: Joseph Hamman <[email protected]> Date: Mon Oct 14 12:18:54 2019 -0700 updates for 0.14 release commit 4f5ca73 Author: Deepak Cherian <[email protected]> Date: Mon Oct 14 18:06:53 2019 +0000 Make concat more forgiving with variables that are being merged. (#3364) * Make concat more forgiving with variables that are being merged. * rename test. * simplify test. * make diff smaller. commit ae1d8c7 Author: Crypto Jerônimo <[email protected]> Date: Sun Oct 13 15:38:36 2019 +0100 Fix documentation typos (#3396) commit 863e490 Author: Joe Hamman <[email protected]> Date: Sat Oct 12 17:33:33 2019 -0400 OrderedDict --> dict, some python3.5 cleanup too (#3389) * OrderedDict --> dict, some python3.5 cleanup too * respond to part of @shoyer's review * fix set attr syntax on netcdf4 vars * fix typing errors * update whats new and todo comments * Typing annotations * Typing annotations * Fix regression * More substantial changes * More polish * Typing annotations * Rerun notebooks commit 6851e3e Author: crusaderky <[email protected]> Date: Sat Oct 12 21:05:32 2019 +0100 Annotate LRUCache (#3395) commit 4c05d38 Author: Stephan Hoyer <[email protected]> Date: Fri Oct 11 08:47:57 2019 -0700 BUG: overrides to a dimension coordinate do not get aligned (#3393) Fixes GH3377 commit 3f29551 Author: Deepak Cherian <[email protected]> Date: Thu Oct 10 23:44:19 2019 +0000 map_blocks (#3276) * map_block attempt 2 * Address reviews: errors, args + kwargs support. * Works with datasets! * remove wrong comment. * Support chunks. * infer template. * cleanup * cleanup2 * api.rst * simple shape change error check. * Make test more complicated. * Fix for when user function doesn't set DataArray.name * Now _to_temp_dataset works. * Add whats-new * chunks kwarg makes no sense right now. * review feedback: 1. skip index graph nodes. 2. var → name 3. quicker dataarray creation. 4. Add restrictions to docstring. 5. rename chunk construction task. 6. error when non-xarray object is returned. 7. restore non-coord dims. review * Support nondim coords in make_meta. * Add Dataset.unify_chunks * doc updates. * minor. * update comment. * More complicated test dataset. Tests fail :X * Don't know why compute is needed. * work with DataArray nondim coords. * fastpath unify_chunks * comment. * much improved tests. * Change args, kwargs syntax. * Add dataset, dataarray methods. * api.rst * docstrings. * Fix unify_chunks. * Move assert_chunks_equal to xarray.testing. * minor changes. * Better error handling when inferring returned object * wip * wip * better to_array * Docstrings + nicer error message. * remove unify_chunks in map_blocks + better tests. * typing for unify_chunks * address more review comments. * more unify_chunks tests. * Just use dask.core.utils.meta_from_array * get tests working. assert_equal needs a lot of fixing. * more unify_chunks test. * assert_chunks_equal fixes. * copy over meta_from_array. * minor fixes. * raise chunks error earlier and test for map_blocks raising chunk error * fix. * Type annotations * py35 compat * make sure unify_chunks does not compute. * Make tests functional by call compute before assert_equal * Update whats-new * Work with attributes. * Support attrs and name changes. * more assert_equal * test changing coord attribute * fix whats new * rework tests to use fixtures (kind of) * more review changes. * cleanup * more review feedback. * fix unify_chunks. * read dask_array_compat :) * Dask 1.2.0 compat. * documentation polish * make_meta reflow * cosmetic * polish * Fix tests * isort * isort * Add func call to docstrings. commit 291cb80 Author: Deepak Cherian <[email protected]> Date: Thu Oct 10 18:23:20 2019 +0000 Add groupby.dims & Fix groupby reduce for DataArray (#3338) * Fix groupby reduce for DataArray * bugfix. * another bugfix. * bugfix unique_and_monotonic for object indexes (uniqueness is enough) * Add groupby.dims property. * update reduce docstring to point to xarray.ALL_DIMS * test for object index dims. * test reduce dimensions error. * Add whats-new * fix docs build * sq whats-new * one more test. * fix test. * undo monotonic change. * Add dimensions to repr. * Raise error if no bins. * Raise nice error if no groups were formed. * Some more error raising and testing. * Add dataset tests. * update whats-new. * fix tests. * make dims a cached lazy property. * fix whats-new. * whitespace * fix whats-new commit 3f0049f Author: crusaderky <[email protected]> Date: Wed Oct 9 19:01:29 2019 +0100 Speed up isel and __getitem__ (#3375) * Variable.isel cleanup/speedup * Dataset.isel code cleanup * Speed up isel * What's New * Better error checks * Speedup * type annotations * Update doc/whats-new.rst Co-Authored-By: Maximilian Roos <[email protected]> * What's New * What's New * Always shallow-copy variables commit 132733a Author: Deepak Cherian <[email protected]> Date: Tue Oct 8 22:13:47 2019 +0000 Fix concat bug when concatenating unlabeled dimensions. (#3362) * Fix concat bug when concatenating unlabeled dimensions. * Add whats-new * Add back older test. * fix test * Revert "fix test" This reverts commit c33ca34. * better fix commit 6fb272c Author: crusaderky <[email protected]> Date: Tue Oct 8 22:23:46 2019 +0100 Rolling minimum dependency versions policy (#3358) * - Downgrade numpy to 1.14, pandas to 0.20, scipy to 0.19 (24 months old) - Downgrade dask to 1.1 (6 months old) - Don't pin patch versions * Apply rolling policy (see #3222) * Automated tool to verify the minimum versions * Drop Python 3.5 * lint * Trivial cosmetic * Cosmetic * (temp) debug CI failure * Parallelize versions check script * Remove hacks for legacy dask * Documentation * Assorted cleanup * Assorted cleanup * Fix regression * Cleanup * type annotations upgraded to Python 3.6 * count_not_none backport * pd.Index.equals on legacy pandas returned False when comparing vs. a ndarray * Documentation * pathlib cleanup * Slide deprecations from 0.14 to 0.15 * More cleanups * More cleanups * Fix min_deps_check * Fix min_deps_check * Set policy of 12 months for pandas and scipy * Cleanup * Cleanup * Sphinx fix * Overhaul readthedocs environment * Fix test crash * Fix test crash * Prune readthedocs environment * Cleanup * Hack around versioneer bug on readthedocs CI * Code review * Prevent random timeouts in the readthedocs CI * What's New polish * Merge from Master * Trivial cosmetic * Reimplement pandas.core.common.count_not_none commit 3e2a754 Author: Alan D. Snow <[email protected]> Date: Tue Oct 8 09:36:52 2019 -0500 added geocube and rioxarray to related projects (#3383) commit 4254b4a Author: crusaderky <[email protected]> Date: Fri Oct 4 23:17:56 2019 +0100 Lint (#3373) * raise exception instance, not class * isort * isort * Bump mypy version commit 283b4fe Author: Deepak Cherian <[email protected]> Date: Fri Oct 4 17:04:36 2019 +0000 Docs/more fixes (#2934) * Move netcdf to beginning of io.rst * Better indexing example. * Start de-emphasizing pandas * misc. * compute, load, persist docstrings + text. * split-apply-combine. * np.newaxis. * misc. * some dask stuff. * Little more dask. * undo index.rst changes. * link to dask docs on chunks * Fix io.rst. * small changes. * rollingupdate. * joe's review commit dfdeef7 Author: Stephan Hoyer <[email protected]> Date: Thu Oct 3 21:42:50 2019 -0700 Explicitly keep track of indexes with merging (#3234) * Explicitly keep track of indexes in merge.py * Typing fixes * More tying fixes * more typing fixes * fixup commit 86fb71d Author: Deepak Cherian <[email protected]> Date: Thu Oct 3 15:41:50 2019 +0000 groupby repr (#3344) * groupby repr * add test. * test datetime and nondim coord * rename test_da → repr_da * Add whats-new * Update doc/whats-new.rst commit dd2b803 Author: Ryan May <[email protected]> Date: Wed Oct 2 15:43:44 2019 -0600 Remove setting of universal wheels (#3367) Universal wheels indicate that one wheel supports Python 2 and 3. This is no longer the case for xarray. This causes builds to generate files with names like xarray-0.13.0-py2.py3-none-any.whl, which can cause pip to incorrectly install the wheel when installing from a list of wheel files. commit 21705e6 Author: crusaderky <[email protected]> Date: Tue Oct 1 19:13:55 2019 +0100 Revisit # noqa annotations (#3359) commit fb575eb Author: crusaderky <[email protected]> Date: Tue Oct 1 15:11:21 2019 +0100 Fix codecov.io upload on Windows (#3360) commit 1ab2279 Author: Deepak Cherian <[email protected]> Date: Mon Sep 30 21:12:22 2019 +0000 Add how do I ... section (#3357) * Add how do I ... section * Bbugfix. * Update doc/howdoi.rst Co-Authored-By: Maximilian Roos <[email protected]> * Update doc/howdoi.rst Co-Authored-By: Maximilian Roos <[email protected]> * small updates. * Add more. commit bd1069b Author: Gregory Gundersen <[email protected]> Date: Sun Sep 29 19:39:53 2019 -0400 Add glossary to documentation (#3352) * First draft at terminology glossary. * Made name matching rules more explicit and hopefully clearer. * Amended what's new. * Changes based on feedback. * More changed based on feedback. commit b51683f Author: Anderson Banihirwe <[email protected]> Date: Sun Sep 29 07:50:21 2019 -0600 Documentation improvements (#3328) * Add examples for full_like, zeros_like, ones_like * Add examples for xr.align * Add examples for xr.merge * Update xr.where docstring * Update xr.dot docstring * Update xarray/core/common.py Co-Authored-By: Deepak Cherian <[email protected]> * Update xarray/core/common.py Co-Authored-By: Deepak Cherian <[email protected]> * Update xr.combine_by_coords docstring * Apply black formatting only * More black formatting * Remove unnecessary pandas bits * Fix indentation issues * Update assign and pipe * Update `Dataset.reindex` with examples * Update `Dataset.fillna` with examples * Address styling issues * Update docstring Co-Authored-By: Deepak Cherian <[email protected]> commit f3c7da6 Author: Gregory Gundersen <[email protected]> Date: Sat Sep 28 15:57:36 2019 -0400 Remove `complex.nc` from built docs (#3353) * Rolling back to prevent a different issue from leaking into this one. * Amended what's new. commit 6ece6a1 Author: Tony Tung <[email protected]> Date: Thu Sep 26 22:45:26 2019 -0700 Fix DataArray.to_netcdf type annotation (#3325) It calls DataSet.to_netcdf, which returns Union[bytes, "Delayed", None]. So this should as well. commit 16fdac9 Author: crusaderky <[email protected]> Date: Thu Sep 26 10:38:46 2019 +0100 CI test suites with pinned minimum dependencies (#3346) * CI test suites with pinned minimum dependencies * code review * Clarity re lxml commit ea101f5 Author: Tom Nicholas <[email protected]> Date: Thu Sep 26 10:51:58 2019 +0200 Bugfix/plot accept coord dim (#3345) Bug in plot.line fixed by ensuring 1D coords are cast down to their associated dims. There was previously two particular cases where this would not happen. commit 85c9a40 Author: crusaderky <[email protected]> Date: Wed Sep 25 02:40:54 2019 +0100 CI environments overhaul (#3340) * Rationalize and align CI environments. Add many optional dependencies to individual CI suites. * pynio and cdms2 are not available on Windows * cfgrib causes Python interpreter crash on Windows * dtype of np.arange defaults to int64 on Linux and int32 on Windows * Suppress failure to delete file on Windows * Mark hypotesis tests as @slow

dcherian commented May 2, 2019

View reviewed changes

dcherian force-pushed the docs/more-fixes branch 4 times, most recently from 3bce6a8 to 1bbf8bf Compare May 23, 2019 21:37

shoyer reviewed Jun 15, 2019

View reviewed changes

doc/dask.rst Outdated Show resolved Hide resolved

doc/index.rst Outdated Show resolved Hide resolved

doc/why-xarray.rst Show resolved Hide resolved

doc/dask.rst Outdated Show resolved Hide resolved

dcherian added 14 commits September 28, 2019 14:13

Move netcdf to beginning of io.rst

cb5a348

Better indexing example.

ef11ef0

Start de-emphasizing pandas

84028dd

misc.

c45f7f6

compute, load, persist docstrings + text.

d9173df

split-apply-combine.

d77455d

np.newaxis.

056288d

misc.

6851c8a

some dask stuff.

9d52931

Little more dask.

f7fc375

undo index.rst changes.

959f186

link to dask docs on chunks

b1dfc66

Fix io.rst.

8df0ca4

small changes.

0b6045b

dcherian force-pushed the docs/more-fixes branch from 047063b to 0b6045b Compare September 28, 2019 20:13

dcherian commented Sep 28, 2019

View reviewed changes

rollingupdate.

fadcb04

jhamman reviewed Sep 29, 2019

View reviewed changes

doc/dask.rst Outdated Show resolved Hide resolved

joe's review

b6de7f9

dcherian merged commit 283b4fe into pydata:master Oct 4, 2019

dcherian deleted the docs/more-fixes branch October 4, 2019 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs/more fixes #2934

Docs/more fixes #2934

dcherian commented May 2, 2019 •

edited

Loading

dcherian May 2, 2019

jhamman Sep 28, 2019

dcherian May 2, 2019

dcherian May 2, 2019

dcherian May 2, 2019

dcherian May 2, 2019

dcherian commented May 23, 2019

dcherian left a comment

jhamman commented Oct 4, 2019

max-sixty commented Oct 4, 2019

Docs/more fixes #2934

Docs/more fixes #2934

Conversation

dcherian commented May 2, 2019 • edited Loading

dcherian May 2, 2019

Choose a reason for hiding this comment

jhamman Sep 28, 2019

Choose a reason for hiding this comment

dcherian May 2, 2019

Choose a reason for hiding this comment

dcherian May 2, 2019

Choose a reason for hiding this comment

dcherian May 2, 2019

Choose a reason for hiding this comment

dcherian May 2, 2019

Choose a reason for hiding this comment

dcherian commented May 23, 2019

dcherian left a comment

Choose a reason for hiding this comment

jhamman commented Oct 4, 2019

max-sixty commented Oct 4, 2019

dcherian commented May 2, 2019 •

edited

Loading