Preparation of v0.2 #20
Merged
This pull request is big! We’re only showing the most recent 250 commits.
Commits on Aug 8, 2019
-
-
-
-
resource.py add for accessing solarpanels and windturbines
utlis.py add class arrowdict
-
-
-
-
-
-
-
-
Commits on Aug 9, 2019
Commits on Aug 12, 2019
-
-
instructions.rst add images and 'how it works' description
basics.rst distribute among other chapters
-
doc/cutout.rst update text and add commands
convert.py small docstring corrections
-
-
-
-
-
-
-
-
Merge pull request #40 from euronion/documentation
Copyright and licensing
-
Move download_turbineconf to resource module
To avoid circular dependencies.
-
Commits on Aug 13, 2019
-
__init__.py load turbines and panel from resource.py
wind.py remove unnecessary imports resource.py fix missing imports, use _update_resource_dict in download_windturbineconfig ensure replacing '-' in wind_config strings
-
-
environment_docs.yaml: try again with nbshinx extension, include othe…
…r packages as required
-
Merge pull request #43 from PyPSA/pr-39
Tab-completion for panels and turbines
-
-
-
environment_doc test pypi installations due to memory efficiency doc/* small continuations
-
-
-
-
-
-
-
-
-
-
-
-
-
Commits on Aug 14, 2019
-
Switch back to RST and remove duplications by softlinking.
* MD does not allow for RST include directives making the switch necessary * By using the includes, we remove duplications in author listings in the documentation and README.rst and have the chance to have a separate AUTHORS.rst file * The release notes are also no longer duplicated and have a separate file in the root directory. An include-link brings them into the documentation.
-
-
-
Provide examples in documentation subfolder via link to repo root. (2)
Examples which where missing from the commit before #72c0eda .
-
Disable nbsphinx cell execution in SPHINX for empty notebooks.
Workaround for execution errors (do not execute noetbooks. Any output wanted has to be generated locally and uploaded).
-
-
-
-
Rename Logfiles_and_messages.ipynb to logfiles_and_messages.ipynb
Windows - Linux problem: Now solved.
-
-
-
-
Commits on Aug 15, 2019
Commits on Aug 16, 2019
Commits on Aug 19, 2019
-
Update plotting_with_atlite.ipynb
* Autoformatting (autoformat pep8) * Spell checks * Axis labels * Add note on descartes * Add our basic logging recommendation
-
Commits on Aug 20, 2019
-
-
sarah: Port atlite.datasets.sarah to new framework
sarah: Update for data retrieval update sarah: Fix imports sarah: Fix _get_filenames sarah: Fix get_coords sarah: Add static_features sarah: Fix imports sarah: Fix get_data_era5 sarah: Add debug messages sarah: Fix get_data sarah: Fix _get_filenames sarah: Fixup implementation
-
-
era5: Create and clean up separate tmpdir for intermediate downloads
The finalizer deletes files too early in a distributed setting, where the file handles are pickled and restored.
-
-
rio.warp.reproject fails when using 'latlong', so we use the epsg number instead
-
-
-
Leading to a memory use of about 2.2GB for a cutout the size of Europe.
-
-
Commits on Aug 23, 2019
Commits on Aug 29, 2019
Commits on Sep 17, 2019
-
* Unset unused directories for gebco, ncep, sarah, cordex. * Provide error message for unsuccessfull gdalwarp subprocess calls.
-
Extends and partially reverses euronion/atlite@7a225ca .
* Reverse default config. * Rather than checking for a set gebco_path, we check if the path specified exists.
Commits on Sep 18, 2019
Commits on Sep 19, 2019
-
Invert default ordering in latitude dimension
The default array ordering traverses now from small latitudes to large latitudes, since this is how ERA-5 organises its data by default and it lead to non-intuitive confusion several times. The most important changes are: - One now specifies cutout bound slices always from small to large: atlite.Cutout("bla", x=slice(-10, 10), y=slice(40, 45), time="2012-01") - Plotting using imshow has to be inverted explicitly from now on capacity_factor = cutout.wind(turbine="Vestas...", capacity_factor=True) plt.imshow(capacity_factor.transpose('y', 'x').values[::-1], extent=cutout.extent) We encourage users to use xarrays plotting facilities, instead, capacity_factor.plot() -
Open cutout files with cache=False for saving memory
Caching is still possible with `cutout.data = cutout.data.load()`.
-
-
-
-
pv: Add simple latitude orientation
With azimuth=180, it is angled to the south on the northern hemisphere and to the north on the southern hemisphere (due to the negative values for slope)
-
era5: Enable creating global cutouts
One needs to use: x=slice(None), y=slice(None)
-
Revert "era5: Enable creating global cutouts"
This reverts commit 3e7b434.
Commits on Sep 23, 2019
-
utils: Prevent deprecation warning from pkg_resources.
Absolute paths will be deprecated in pkg_resources.resource_filename. Including a leading (back-)slash is recognised as such and raises a warning, this change fixes this behaviour.
-
example plotting: Include figures, update to match v0.2 and misc.
* Now including the figures for the example to be displayed in the doc. * Update the plotting commands to xarray's plot() to match the new v0.2 reverse indexing order * Misc: Show warnings and use the logger.
Commits on Sep 24, 2019
-
Add new example: Historic comparison for Germany (PV, wind in 2012).
Example is based on the old "openmod-atlite-de.ipynb" example.
-
Commits on Sep 25, 2019
-
utils: Correct error message for cutout migration.
* On trying to automatically migrate an existing cutout, display the correct (new) order (min,max) for coordinates in the cutout recreation command. * Use the logging.error(...) facilities to correctly categorise the MergeError incl. the stack trace.
Commits on Nov 10, 2019
-
-
Merge pull request #49 from PyPSA/misc/v0.2-misc-002
utils: Correct error message for cutout migration.
Commits on Mar 6, 2020
Commits on Mar 18, 2020
Commits on Mar 19, 2020
-
-
gis: Do not pickle the spatial index
Turns out the spatial index gets distorted by pickling, so we turn to_file and from_file to no-ops, until we find a work-around.
-
-
Commits on Apr 20, 2020
Commits on Apr 24, 2020
Commits on Apr 30, 2020
Commits on May 8, 2020
-
-
-
ensure correct ordering of slices (#68)
* ensure correct ordering of slices * small fix
-
-
Commits on May 11, 2020
Commits on May 12, 2020
Commits on May 13, 2020
-
Co-authored-by: FabianHofmann <[email protected]>
-
-
-
Commits on May 14, 2020
Commits on Jun 4, 2020
-
* make interpolation optional * replace hourly_mean by resampling function * sarah module: apply autopep8 * fix typo * autopep8; move get_coords to high level * revise data.py, sarah.py * sarah use again as_slice * spread common.py among dataset modules * fix gebco as module * adjust test * change cutout representation * fix era5 static * fix features and allows multiple modules in cutout * fix feature preparation * sanitize prepare function, unify output of get_data * fix logging for requests * enable optional parallel loading * data.py: - remove literal_eval -> cutout_parameters are directly given to cutout.data.attr, slices are directly processes in coordinates creation - re-enable tmp_dir in cutout prepare - to_netcdf has different mode 'a'/'w' depending on whether file exists gebco.py: fix output of get_data sarah.py: set interpolate always to true * sarah.py re-enable optional interpolation * - add docstrings for all functions in data.py/sarah.py/era5.py - modify input variables of get_coords function. * add docstrings for all of cutout.py and gebco.py * cutout.py: - update docstrings for cutout class - remove support for cutou_dir, add warning and pointer to migration function - remove support for data argument as this requires further TODOs and can worked around very easily - remove default for module, this argument must be given - abolish is_view cases - add assertions for argument requirements 'x', 'y', 'time', 'module' when building new cutout * cutout.py: - Improve argument exception - Make cutout representation better - Ensure projection in cutout building * data.py make window class working with new feature handling tests: run tests for era5 and mixed ['sarah', 'era5'] cutouts cutout.py: reenable data as an optional argument * autopep8 in pv/*.py fix typo in migration function * first take for benchmarking: load data with chunks and apply conversion function without windows * cutout.py - clean imports - fully intergrate chunks as an cutout parameter and property - set chunking as standard loading of cutout data.py - use cutout.chunks property era5.py - use cutout.chunks property sarah.py - use cutout.chunks property * utils.py: fix import and swap_dimensions for old style cutout * pv module: - make module dask friendly, this commit removes all .values call which cause dask to be unable to chunk. - direct import of numpy functions which are often used convert.py: - restructure convert_and_aggregate function, this makes the function faster if only a layout is given. - change show_progress to bool only - change layout to be xr.DataArray only * convert: Fix heat demand hourshift for xarray 0.15.1 (#63) From xarray version 0.15.1, .values cannot be assigned. You should use the .assign_coords() method instead. See release notes for xarray 0.15.1 "breaking changes": http://xarray.pydata.org/en/stable/whats-new.html#v0-15-1-23-mar-2020 * convert.py - rename heatdemand array * cutout.py: - raise Error if old style * cutout.py: restructure handling of projection. The projection of different modules is tested when initializing the cutout. The property 'projection' will then only look at the projection of first module. * aggregate.py: remove aggregate_sum function convert.py: - try out dense operation for indicator matrix multiplication - replace aggregate_matrix function with tensor dot (still figuring out performance) gis.py: - argument shapes can now also be geopandas frame * convert.py: reinstatiate aggregate_matrix function, but fix name of index * convert.py: - make index valid for geopandas series and frame * Allow saving of pseudo-boolean cutout attributes to netCDF. * Fix interpolation option and defaults for SARAH cutouts. * convert.py fix division for capacity factor calculation cutout.py fix represention of features * Rename cutout parameters for interpolation of SARAH data. * review examples: - clean and run create_cutout and plotting notebook - delete tiny create_cutout.py script - add comparison script for verions 1 and 2 * Apply suggestions from code review Co-authored-by: Jonas Hörsch <[email protected]> * add suggestions: - aggregate.py ensure index name - convert.py updat docstring - convert.py remove index.name defaults as done by aggregate_matrix function - convert.py reenable progrss bar for hydro - cutout.py fix import structure - data.py remove unneeded code - sarah.py add assertion for time resolution * data.py store booleans as int for to_netcdf * prepare cutout: move prepared_feature assignment directly before storing * update authors list fix suggestions in migration function * revise all imports * * change chunk size attribute from 'chunk_{dim}' to 'chunksize_{}' * fix cutout.prepared_features for one feature only Co-authored-by: Tom Brown <[email protected]> Co-authored-by: euronion <[email protected]> Co-authored-by: Jonas Hörsch <[email protected]>
4 people authoredJun 4, 2020
Commits on Jun 5, 2020
-
-
doc: - remove User Guide section in favour of notebooks
- add notebook for gebco heightmap cutout.py & data.py: small fix for cutout with spatial dimension only
Commits on Jun 7, 2020
Commits on Jun 8, 2020
-
data: Re-enable parallelized queueing (#87)
* era5: Use lock to support download progress bars * era5: Fix retrieval_times * era5: Re-enable warnings * era5: Fix int64 is not serializable to JSON exception * data: Re-enable parallelized queueing Since queueing times are significantly longer than downloading, combine dask.delayed with a lock to queue in parallel but download in series.
Commits on Jun 10, 2020
-
fix updating features & sanitize static variables in era5 (#91)
* era5.py - set output of get_data to xr.Dataset - sanitize static features, whereas the variable should not have a time dimension, the returned dataset of get_data should have data.py - parallize feature loading -> delayed call of get_data * era5.py - set delete file message to logger.debug * era5.py fix typo * era5.py - bug fix time keys * data.py: - store data file without appending to netcdf as it is unsecure, instead create a new one era5.py - revert time assigning to static feature as not needed test - add test for updating a variable * data.py - fix name of temporary file era5.py - stream info log of client through logger.debug * Update atlite/data.py Co-authored-by: euronion <[email protected]> * reenable progress bar * era5: Logging differentiates between request and download (#94) * cutou.py: - selection must have a separate name data.py: - make tempfile creation safe - logging for tmp dir test - adjust test for selection era5.py - fix division by zero * test - use fixture for updatable cutout * era5.py remove sleep time again cutout.py set chunks to 100 as it is faster Co-authored-by: euronion <[email protected]> Co-authored-by: Jonas Hörsch <[email protected]>
3 people authoredJun 10, 2020
Commits on Jun 12, 2020
-
[v0.2] review suggestions (#95)
* cutout: Make new `path` argument optional * cutout: Make dx/dy calculation more robust * data: Do not leak the temporary directory
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Commits on Jun 15, 2020
-
era5.py disable warning for ds.drop()
test: temporarly disable windows machines
Commits on Jan 11, 2021
Commits on Jan 12, 2021
Commits on Jan 13, 2021
-
* Rename projection to crs Follows pyproj in nomenclature. See https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1 . * environment: Remove channel pinning Channel pinning has been superseed by strict channel_priority as proposed at https://conda-forge.org/docs/user/tipsandtricks.html. * gis: Add grid_cell_areas function to compute areas of grid cells * cutout: Fix forgotten conversion * gis: Improve grid_cell_areas * remove area calculation due to geopandas implementation * update release notes * gis.py: revise imports Co-authored-by: Fabian <[email protected]>
-
gebco: Extract and resample data from GEBCO using rasterio [DNMY] (#93)
* gebco: Extract and resample data from GEBCO using rasterio * tiny fixup of inversed y-axis and data array accessing * fix numeric tags Co-authored-by: Fabian <[email protected]>
Commits on Jan 15, 2021
Commits on Jan 16, 2021
Commits on Jan 18, 2021
-
irradiation.py: add comment to clipping, use other approach
solar_position.py: saver/cleaner approach for chunking
Commits on Jan 19, 2021
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.