Dask compatibility #76

…to documentation

Documentation (WiP)

README: Finish switch to README.md from README.rst.

resource: Bugfix in retrieving local turbines.

utlis.py add class arrowdict

…documentation

…tation

Accidentally uploaded.

…documentation

…erwise

basics.rst distribute among other chapters

…documentation

…dencies.

…mber.

convert.py small docstring corrections

…tation

further writing in doc/user-guide

…ll works)

…#33)

Copyright and licensing

To avoid circular dependencies.

wind.py remove unnecessary imports resource.py fix missing imports, use _update_resource_dict in download_windturbineconfig ensure replacing '-' in wind_config strings

…r packages as required

…tation

Tab-completion for panels and turbines

Fixes #44 changse in documentation.

environment_doc test pypi installations due to memory efficiency doc/* small continuations

…ation

…ion)

* MD does not allow for RST include directives making the switch necessary * By using the includes, we remove duplications in author listings in the documentation and README.rst and have the chance to have a separate AUTHORS.rst file * The release notes are also no longer duplicated and have a separate file in the root directory. An include-link brings them into the documentation.

Examples which where missing from the commit before #72c0eda .

Workaround for execution errors (do not execute noetbooks. Any output wanted has to be generated locally and uploaded).

Windows - Linux problem: Now solved.

…documentation

… (try 2).

… (try 3).

… (last try).

* Autoformatting (autoformat pep8) * Spell checks * Axis labels * Add note on descartes * Add our basic logging recommendation

sarah: Update for data retrieval update sarah: Fix imports sarah: Fix _get_filenames sarah: Fix get_coords sarah: Add static_features sarah: Fix imports sarah: Fix get_data_era5 sarah: Add debug messages sarah: Fix get_data sarah: Fix _get_filenames sarah: Fixup implementation

The finalizer deletes files too early in a distributed setting, where the file handles are pickled and restored.

rio.warp.reproject fails when using 'latlong', so we use the epsg number instead

Leading to a memory use of about 2.2GB for a cutout the size of Europe.

…ile.

toolz is required by dask.

* Unset unused directories for gebco, ncep, sarah, cordex. * Provide error message for unsuccessfull gdalwarp subprocess calls.

* Reverse default config. * Rather than checking for a set gebco_path, we check if the path specified exists.

…aths.

Refer to pydata/xarray#2535, rasterio/rasterio-wheels#12

The default array ordering traverses now from small latitudes to large latitudes, since this is how ERA-5 organises its data by default and it lead to non-intuitive confusion several times. The most important changes are: - One now specifies cutout bound slices always from small to large: atlite.Cutout("bla", x=slice(-10, 10), y=slice(40, 45), time="2012-01") - Plotting using imshow has to be inverted explicitly from now on capacity_factor = cutout.wind(turbine="Vestas...", capacity_factor=True) plt.imshow(capacity_factor.transpose('y', 'x').values[::-1], extent=cutout.extent) We encourage users to use xarrays plotting facilities, instead, capacity_factor.plot()

Caching is still possible with `cutout.data = cutout.data.load()`.

CORDEX is available from CDS.

With azimuth=180, it is angled to the south on the northern hemisphere and to the north on the southern hemisphere (due to the negative values for slope)

One needs to use: x=slice(None), y=slice(None)

This reverts commit 3e7b434.

Absolute paths will be deprecated in pkg_resources.resource_filename. Including a leading (back-)slash is recognised as such and raises a warning, this change fixes this behaviour.

* Now including the figures for the example to be displayed in the doc. * Update the plotting commands to xarray's plot() to match the new v0.2 reverse indexing order * Misc: Show warnings and use the logger.

Example is based on the old "openmod-atlite-de.ipynb" example.

* On trying to automatically migrate an existing cutout, display the correct (new) order (min,max) for coordinates in the cutout recreation command. * Use the logging.error(...) facilities to correctly categorise the MergeError incl. the stack trace.

utils: Correct error message for cutout migration.

Turns out the spatial index gets distorted by pickling, so we turn to_file and from_file to no-ops, until we find a work-around.

* fix #64 * ensure min/max assingments in coordinates

* ensure correct ordering of slices * small fix

Add test scripts

use shapely's strtree instead of rtree

Co-authored-by: FabianHofmann <[email protected]>

Expand test scripts

…n to cutout.data.attr, slices are directly processes in coordinates creation - re-enable tmp_dir in cutout prepare - to_netcdf has different mode 'a'/'w' depending on whether file exists gebco.py: fix output of get_data sarah.py: set interpolate always to true

- modify input variables of get_coords function.

- update docstrings for cutout class - remove support for cutou_dir, add warning and pointer to migration function - remove support for data argument as this requires further TODOs and can worked around very easily - remove default for module, this argument must be given - abolish is_view cases - add assertions for argument requirements 'x', 'y', 'time', 'module' when building new cutout

- Improve argument exception - Make cutout representation better - Ensure projection in cutout building

tests: run tests for era5 and mixed ['sarah', 'era5'] cutouts cutout.py: reenable data as an optional argument

fix typo in migration function

…on function without windows

- clean imports - fully intergrate chunks as an cutout parameter and property - set chunking as standard loading of cutout data.py - use cutout.chunks property era5.py - use cutout.chunks property sarah.py - use cutout.chunks property

- make module dask friendly, this commit removes all .values call which cause dask to be unable to chunk. - direct import of numpy functions which are often used convert.py: - restructure convert_and_aggregate function, this makes the function faster if only a layout is given. - change show_progress to bool only - change layout to be xr.DataArray only

From xarray version 0.15.1, .values cannot be assigned. You should use the .assign_coords() method instead. See release notes for xarray 0.15.1 "breaking changes": http://xarray.pydata.org/en/stable/whats-new.html#v0-15-1-23-mar-2020

…erent modules is tested when initializing the cutout. The property 'projection' will then only look at the projection of first module.

convert.py: - try out dense operation for indicator matrix multiplication - replace aggregate_matrix function with tensor dot (still figuring out performance) gis.py: - argument shapes can now also be geopandas frame

…ndex

Commits on Mar 18, 2020

Fix unmatched braket.

euronion committed Mar 18, 2020

Configuration menu

View commit details

Copy full SHA for d1c8fa0

Browse repository at this point

Copy the full SHA

d1c8fa0 View commit details

Browse the repository at this point in the history

Dask compatibility #76

Dask compatibility #76

Commits on Aug 7, 2019

Commits on Aug 8, 2019

Commits on Aug 9, 2019

Commits on Aug 12, 2019

Commits on Aug 13, 2019

Commits on Aug 14, 2019

Commits on Aug 15, 2019

Commits on Aug 16, 2019

Commits on Aug 19, 2019

Commits on Aug 20, 2019

Commits on Aug 23, 2019

Commits on Aug 29, 2019

Commits on Sep 17, 2019

Commits on Sep 18, 2019

Commits on Sep 19, 2019

Commits on Sep 23, 2019

Commits on Sep 24, 2019

Commits on Sep 25, 2019

Commits on Nov 10, 2019

Commits on Feb 26, 2020

Commits on Mar 6, 2020

Commits on Mar 18, 2020

Commits on Mar 19, 2020

Commits on Apr 20, 2020

Commits on Apr 24, 2020

Commits on Apr 30, 2020

Commits on May 6, 2020

Commits on May 8, 2020

Commits on May 10, 2020

Commits on May 11, 2020

Commits on May 12, 2020

Commits on May 13, 2020

Commits on May 14, 2020

Commits on May 26, 2020

Commits on May 27, 2020

Commits on May 28, 2020

Commits on May 29, 2020

Commits on Jun 1, 2020

Commits on Jun 2, 2020

Commits on Jun 3, 2020