Skip to content

Commit

Permalink
Gap refactor v0.3.0 (#467)
Browse files Browse the repository at this point in the history 
* WIP: refactoring to only gap --> major implementation is done

* get IO test running

* fixed analysis class

* fix breaking test

* get gapfill and test running

* version bump and template show in IO test

* get ploting example running

* mulit inheriance of Dataset and fix of the add and irregular test

* fix modeldata test

* get qc check working

* cleanup and docstring updates

* cleanup

* logging and exmple + kwargs for interp gap

* upgrade the interpolation method to include higher order interpolation techniques

* update examples

* fixed the analysis example

* bugfix in sync + irregular demo in docs

* fix io test and cleanup doc examples/topics

* logginmodule fix + wroking on  docstrigs

* updating docstrings + print info bug fix

* fix doctest for dataset core + update lcz map

* fix docstrings and doctest in the Analysis class

* fix doctest in visuals, qc and gaps

* add see also for visual module

* cleanup files

* cleanup files

* cleanup files

* Static and Dynamic modeldata classes for GEE interaction (#469)

* functionality done, ready for cleanup

* savepoint in code clieanup

* savepoint docstring

* savepoint: text for docstrings

* docstring examples for GeeStaticModelData

* added examples in all docstrings

* docstrings in dataset_modeldata

* get tests running + direct gee plot methods on Dataset

* add Gee dataset classes and modeldata in the documentation

* fix tests

* cleanup

* test workflow windows runner

* work on the documentation

* minor changes + paper notebook compatible and tracking with new versions

* savepoint: writing docstrings in the Gap class

* Get docsring examples running

* fix identations in docstrings

* embeded plots in the documentation

* version fix

* fix gap fill test

* avoid tracking of py copies of notebooks

* fix docstring tests

* fix running devpipeline

* rm chmod

* minor code review

* savepoint codereview

* test windows runner

* rename update settings

* fix setting updater methods

* add docstrings logging and fix docsrings for settings updater

* code review fixes (deleting unused lines + sys.path inserts for all the test

* import dataset not a method but function

* fix path bug + add matplotlib in the requirements for rtd

* fix bug that creates the "=" file

* try rtd with poetry install

* fix bugs in paper ipynb

* test with rtd yaml

* new rtd yaml test

* new test rtd yaml

* sphinx build in poetry env

* test RTD yaml

* fix rtd yaml

* test yaml build doc

* sytax error in rtd yaml

* testing

* test rtd yml

* test rtd build with sphinx to requirements.txt build

* test in pre install

* requirements txt with group

* new attempt

* test rtd build

* without sudo rtd build

* without pandoc rtd build test

* without mocking dependencies

* add a pip list

* now with dependecy install

* export req in pre_install

* new test

* explicit groups

* cleanup after all testing

* spell checking

* spelling check on all files

* ready for last tests !!

* fix (last?) review points and fixes

* add ipython to documentation group for fixing error

* yaml workflow fix

* Update main_workflow.yml

* version bump + add pip install metobs to rtd builder

* pip install of self for testing

* add data in the included testdata folder for the irr and __add__ tests

* make sure that notebooks are not executed when (online) building docs

* testing

* try new fix for workflows

* update the output of the ipynb examples, so they are rendered

* do not build documentation on github

* cleanup + fix last doc issues. Ready for MERGE !!!!
  • Loading branch information
@vergauwenthomas
vergauwenthomas authored Sep 10, 2024
1 parent e9463db commit 1de19d0
Show file tree
Hide file tree
Showing 148 changed files with 52,786 additions and 23,036 deletions.
353 changes: 214 additions & 139 deletions .github/workflows/main_workflow.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,12 @@ jobs:
poetry install --all-extras
- name: export documentation requirement group
run: |
poetry export -f requirements.txt --without-hashes --only documentation -o only_doc_req.txt
poetry export -f requirements.txt --without-hashes --with=documentation,dev,titan --format=requirements.txt -o doc_build_req.txt
- name: convert doc requirments to artifact
uses: actions/upload-artifact@v3
with:
name: documentation_requirements
path: only_doc_req.txt
path: doc_build_req.txt
- name: export titan requirement group
run: |
poetry export -f requirements.txt --without-hashes --only titan -o only_titan_req.txt
Expand Down Expand Up @@ -109,6 +109,7 @@ jobs:
python tests/push_test/breaking_test.py
#python tests/push_test/gui_launch_test.py
python tests/push_test/analysis_test.py
python tests/push_test/irregular_and_add_test.py
#--- Package os installation ---
mac_install_testing:
Expand Down Expand Up @@ -138,11 +139,11 @@ jobs:
needs: packageing
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Set up Python39
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.9"
python-version: "3.11"
- name: Download the package build
uses: actions/download-artifact@v3
with:
Expand Down Expand Up @@ -177,151 +178,225 @@ jobs:
echo "version tags are not aligned!"
exit 1
#---- Documentation build -----#
doctests:
name: documentation-test
needs: packageing
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python39
uses: actions/setup-python@v4
with:
python-version: '3.9'
#cache: 'poetry'
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: install pandoc (system wide)
run: |
sudo apt-get -y install pandoc
- name: Download the package build
uses: actions/download-artifact@v3
with:
name: package_build
- name: Install the package
run: |
python3 -m pip install ./metobs_toolkit-*.tar.gz
- name: get documentation requirements
uses: actions/download-artifact@v3
with:
name: documentation_requirements
- name: install doc depending packages
run: |
pip install -r only_doc_req.txt
- name: Build documentation
run: |
sphinx-build -a -E docs _build
- name: deploy documentation on dev
if: |
github.ref == 'refs/heads/dev' ||
github.ref == 'refs/head/main' ||
contains(github.event.head_commit.message, 'build doc')
uses: peaceiris/actions-gh-pages@v3
with:
publish_branch: gh-pages-dev
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: _build/
force_orphan: true
# #---- Documentation build -----#
# docsbuild:
# name: documentation-test
# needs: packageing
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v3
# - name: Set up Python310
# uses: actions/setup-python@v4
# with:
# python-version: '3.10'
# #cache: 'poetry'
# # You can test your matrix by printing the current Python version
# - name: Display Python version
# run: python -c "import sys; print(sys.version)"
# # - name: install pandoc (system wide)
# # run: |
# # sudo apt-get -y install pandoc
# - name: Download the package build
# uses: actions/download-artifact@v3
# with:
# name: package_build
# - name: Install the package
# run: |
# python -m pip install ./metobs_toolkit-*.tar.gz
# - name: get documentation requirements
# uses: actions/download-artifact@v3
# with:
# name: documentation_requirements
# - name: install doc depending packages
# run: |
# pip install -r doc_build_req.txt
# - name: Build documentation
# run: |
# sphinx-build -a -E docs _build
# - name: deploy documentation on dev
# if: |
# github.ref == 'refs/heads/dev' ||
# github.ref == 'refs/head/main' ||
# contains(github.event.head_commit.message, 'build doc')
# uses: peaceiris/actions-gh-pages@v3
# with:
# publish_branch: gh-pages-dev
# github_token: ${{ secrets.GITHUB_TOKEN }}
# publish_dir: _build/
# force_orphan: true


#---- Deploy documentation MAIN-----#
deploy_doc_main:
name: Deploy main documentation
needs: [doctests,run-tests,versiontest,mac_install_testing, windows_install_testing]
runs-on: ubuntu-latest
if: |
github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- name: Set up Python39
uses: actions/setup-python@v4
with:
python-version: '3.9'
#cache: 'poetry'
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: install pandoc (system wide)
run: |
sudo apt-get -y install pandoc
- name: Download the package build
uses: actions/download-artifact@v3
with:
name: package_build
- name: Install the package
run: |
python3 -m pip install ./metobs_toolkit-*.tar.gz
- name: get documentation requirements
uses: actions/download-artifact@v3
with:
name: documentation_requirements
- name: install doc depending packages
run: |
pip install -r only_doc_req.txt
- name: Build documentation
run: |
sphinx-build -a -E docs _build
- name: deploy documentation
uses: peaceiris/actions-gh-pages@v3
with:
publish_branch: gh-pages
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: _build/
force_orphan: true
#---- Documentation build -----#
# docsbuild:
# name: documentation-building
# needs: packageing
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v3
# - name: Set up Python39
# uses: actions/setup-python@v4
# with:
# python-version: '3.10'
# #cache: 'poetry'
# # You can test your matrix by printing the current Python version
# - name: Display Python version
# run: python -c "import sys; print(sys.version)"
# - name: remove previous builds
# run: |
# rm -f dist/metobs_toolkit-*.whl
# rm -f dist/metobs_toolkit-*.tar.gz
# - name: Install poetry
# run: |
# pip install poetry
# - name: make build of the package
# run: |
# ./deploiment/build_package.sh
# - name: install the package for doc building
# run: |
# poetry install --with=documentation,dev
# poetry add .
# - name: Build documentation
# run: |
# poetry run sphinx-build -a -E docs _build
# steps:
# - uses: actions/checkout@v3
# - name: Set up Python310
# uses: actions/setup-python@v4
# with:
# python-version: '3.10'
# #cache: 'poetry'
# # You can test your matrix by printing the current Python version
# - name: Display Python version
# run: python -c "import sys; print(sys.version)"
# # - name: install pandoc (system wide)
# # run: |
# # sudo apt-get -y install pandoc
# - name: Download the package build
# uses: actions/download-artifact@v3
# with:
# name: package_build
# - name: Install the package
# run: |
# python -m pip install ./metobs_toolkit-*.tar.gz
# - name: get documentation requirements
# uses: actions/download-artifact@v3
# with:
# name: documentation_requirements
# - name: install doc depending packages
# run: |
# pip install -r doc_build_req.txt
# - name: Build documentation
# run: |
# sphinx-build -a -E docs _build
# - name: deploy documentation on dev
# if: |
# github.ref == 'refs/heads/dev' ||
# github.ref == 'refs/head/main' ||
# contains(github.event.head_commit.message, 'build doc')
# uses: peaceiris/actions-gh-pages@v3
# with:
# publish_branch: gh-pages-dev
# github_token: ${{ secrets.GITHUB_TOKEN }}
# publish_dir: _build/
# force_orphan: true

# #---- Deploy documentation MAIN-----#
# deploy_doc_main:
# name: Deploy main documentation
# needs: [docsbuild,run-tests,versiontest,mac_install_testing, windows_install_testing]
# runs-on: ubuntu-latest
# if: |
# github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main'
# steps:
# - uses: actions/checkout@v3
# - name: Set up Python39
# uses: actions/setup-python@v4
# with:
# python-version: '3.9'
# #cache: 'poetry'
# # You can test your matrix by printing the current Python version
# - name: Display Python version
# run: python -c "import sys; print(sys.version)"
# - name: install pandoc (system wide)
# run: |
# sudo apt-get -y install pandoc
# - name: Download the package build
# uses: actions/download-artifact@v3
# with:
# name: package_build
# - name: Install the package
# run: |
# python3 -m pip install ./metobs_toolkit-*.tar.gz
# - name: get documentation requirements
# uses: actions/download-artifact@v3
# with:
# name: documentation_requirements
# - name: install doc depending packages
# run: |
# pip install -r doc_build_req.txt
# - name: Build documentation
# run: |
# sphinx-build -a -E docs _build
# - name: deploy documentation
# uses: peaceiris/actions-gh-pages@v3
# with:
# publish_branch: gh-pages
# github_token: ${{ secrets.GITHUB_TOKEN }}
# publish_dir: _build/
# force_orphan: true

#---- Deploy documentation dev -----#
deploy_doc_dev:
name: Deploy Dev documentation
needs: [doctests,run-tests,versiontest,mac_install_testing, windows_install_testing]
runs-on: ubuntu-latest
if: |
github.ref == 'refs/heads/dev'
steps:
- uses: actions/checkout@v3
- name: Set up Python39
uses: actions/setup-python@v4
with:
python-version: '3.9'
#cache: 'poetry'
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: install pandoc (system wide)
run: |
sudo apt-get -y install pandoc
- name: Download the package build
uses: actions/download-artifact@v3
with:
name: package_build
- name: Install the package
run: |
python3 -m pip install ./metobs_toolkit-*.tar.gz
- name: get documentation requirements
uses: actions/download-artifact@v3
with:
name: documentation_requirements
- name: install doc depending packages
run: |
pip install -r only_doc_req.txt
- name: Build documentation
run: |
sphinx-build -a -E docs _build
- name: deploy documentation
uses: peaceiris/actions-gh-pages@v3
with:
publish_branch: gh-pages-dev
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: _build/
force_orphan: true
# deploy_doc_dev:
# name: Deploy Dev documentation
# needs: [docsbuild,run-tests,versiontest,mac_install_testing, windows_install_testing]
# runs-on: ubuntu-latest
# if: |
# github.ref == 'refs/heads/dev'
# steps:
# - uses: actions/checkout@v3
# - name: Set up Python39
# uses: actions/setup-python@v4
# with:
# python-version: '3.9'
# #cache: 'poetry'
# # You can test your matrix by printing the current Python version
# - name: Display Python version
# run: python -c "import sys; print(sys.version)"
# - name: install pandoc (system wide)
# run: |
# sudo apt-get -y install pandoc
# - name: Download the package build
# uses: actions/download-artifact@v3
# with:
# name: package_build
# - name: Install the package
# run: |
# python3 -m pip install ./metobs_toolkit-*.tar.gz
# - name: get documentation requirements
# uses: actions/download-artifact@v3
# with:
# name: documentation_requirements
# - name: install doc depending packages
# run: |
# pip install -r doc_build_req.txt
# - name: Build documentation
# run: |
# sphinx-build -a -E docs _build
# - name: deploy documentation
# uses: peaceiris/actions-gh-pages@v3
# with:
# publish_branch: gh-pages-dev
# github_token: ${{ secrets.GITHUB_TOKEN }}
# publish_dir: _build/
# force_orphan: true




# ---- delete artifacts that are not for storage -----
cleanup_artifacts:
name: delete artifacts
needs: [doctests,run-tests,versiontest,mac_install_testing, windows_install_testing, deploy_doc_main]
needs: [run-tests,versiontest,mac_install_testing, windows_install_testing]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
Expand Down
Loading

0 comments on commit 1de19d0

Please sign in to comment.