Documentation improvement - tutorial

[pz-max]

From Discord. I feel our documentation needs some restructuring:

• installation https://pypsa-earth.readthedocs.io/en/latest/installation.html
• getting started https://pypsa-earth.readthedocs.io/en/latest/short_tutorial.html
• tutorial https://pypsa-earth.readthedocs.io/en/latest/tutorial.html

Problem: The installation section & tutorial sections motivates people to create a config.yaml from the config.default.yaml that often leads to issues. Sometimes we refer people to the tutorial but what we mean is the "getting started" section. Getting started and tutorial should be put together.

[davide-f]

Completely agree. I add below some preliminary comments for revision:

• Installation: Need to first clarify that we need miniconda and git installation, as well as an IDE (e.g. VSCode)
• Getting started may be revised into "Tutorial", it is very confusing to have Getting started and tutorial, where the getting started is also a tutorial. On top of that the quick start may be improved as well. Could be good to add a video where we go through the steps. The old tutorial shall be revised as discussed in the following point
• The Data used by the model may be placed after the tutorial
• The old Tutorial section may renamed in Use cases/Full model/etc. to highlight that this section will detail the full model execution, beyond the tutorial. In that section, we should significnatly revise the build_cutout section to make it even more clear
• The "Examples" section may be revised in "Hands-on documentation pkg" where we refer to the documentation package

Feel free all to add and revise :)

[davide-f]

Moreover, we shall improve the contributing section by revising the pre-commit part and clarify how to use that.

[ekatef]

Moreover, we shall improve the contributing section by revising the pre-commit part and clarify how to use that.

Added a point to pypsa-meets-earth/documentation#17 to keep track of the hints we have :)

[davide-f]

@yerbol-akhmetov and @Netotse , thanks for being interested in this interesting issue, with lots of opportunities to learn and contribute :D

I'll draft here few proposals to break down the problem above and start with interesting and catchy tasks :)

So, I think it is very important to:

1. Improve the installation and the now-called getting started section, so to ease the barrier for the first user.
   @Netotse, I think this task may be very interesting for you, but feel free to disagree, and I think you can give some good improvements.
   • First, we should revise the installation section to improve its usage for the first user. In particular, we should revise the subsection on the "Set configuration file" as it currently focuses on the default configuration. We want the first user to run the tutorial model. So that shall be revised accordingly, best adding references to the dedicated section.
   • Second, the current section getting started details the use of the tutorial model. I think this section may be renamed "Short tutorial" and improved, as discussed above.
     In particular, this section shall be enriched where we say users to start with the tutorial configuration, but highlighting that there is also a default configuration. This to avoid having misunderstanding what is tutorial and what is not. Moreover, to investigate the networks and improve the documentation, you can get inspired by the documentation package.
     I think it is good to have also some feedback on this from you and understand in your opinion what you would like to find here. In the above, there is much content to start with, also with the reference by @ekatef
2. A second task is to revised the section now-called "Tutorial" that details more advanced topics that exploit the default configuration, not the tutorial configuration. I personally think this is highly not nice. A new naming like "Advanced topics and FAQ" or alike. This aims to clarify further comments and answer some typical questions as mentioned in documentation#17 suggested by katia.
3. A third task can be improving the description of the rules and in particular the (solving part](https://pypsa-earth.readthedocs.io/en/latest/solving.html). There we could add the description of run_scenario and run_all_scenarios that are currently missing, but partially described in the introduction of this notebook

To actually propose a new version of any of the documentation file, you shall create a PR to this repository where you change the corresponding .rst file available in the doc folder. In the above, references to the files are provided.
To do so, you shal "clone" the repository of github, do:

1. "git clone https://github.com/{your github name}/pypsa-earth"
2. create a new branch (guide with vscode and github or directly with git commands)
3. modify the files locally
4. create commits and push to your github (see guides above)
5. open a PR on pypsa-earth

Happy to add details :D
Moreover @ekatef expressed interests, feel free to support anytime :) likewise @pz-max

[Netotse]

Thanks @davide-f ,

I agree with your comments and have started implementing but I think to be friendly to the absolute noob, the begining should be the software hits portion, we can put a link to it in the installation page if the team feel most of the users will be more advanced but it would be good to start by:

1. Identifying hardware requirements(it's at the bottom of the page, i'd like to move it to the top)
2. List out the different software packages required to run pypsa meets earth
3. Since I've seen questions about setting up vs code, it would be good to also include the link to the video @pz-max posted in the support channel.

what do you think?

[TomFer97]

As a new user I think that this improvement could contribute to reduce first steps for participate in pypsa-earth!

[davide-f]

I add to the discussion that it is important to add a description on how to install the pre-commit:

• in vscode, view-Command palette->Python: Select Intepreter-> pypsa-earth environment
• once that is done, in the prompt of commands, digit "pre-commit install"
• before commiting, may be good to execute "pre-commit run --all" to make sure the code is compliant to pre-commit

[davide-f]

Thanks @davide-f ,

I agree with your comments and have started implementing but I think to be friendly to the absolute noob, the begining should be the software hits portion, we can put a link to it in the installation page if the team feel most of the users will be more advanced but it would be good to start by:

1. Identifying hardware requirements(it's at the bottom of the page, i'd like to move it to the top)
2. List out the different software packages required to run pypsa meets earth
3. Since I've seen questions about setting up vs code, it would be good to also include the link to the video @pz-max posted in the support channel.

what do you think?

Feel free to revise as you best think, I highly like the proposals :D as also confirmed by @TomFer97 :)
Sorry but I've been on holiday recently...

[Netotse]

I add to the discussion that it is important to add a description on how to install the pre-commit:

* in vscode, view-Command palette->Python: Select Intepreter-> pypsa-earth environment

* once that is done, in the prompt of commands, digit "pre-commit install"

* before commiting, may be good to execute "pre-commit run --all" to make sure the code is compliant to pre-commit

Hi David,

I will reach out to you on discord for more info on this, when I open the command palette, I do not see the pypsa-earth environment in the select interpreter section

[davide-f]

I add to the discussion that it is important to add a description on how to install the pre-commit:

* in vscode, view-Command palette->Python: Select Intepreter-> pypsa-earth environment

* once that is done, in the prompt of commands, digit "pre-commit install"

* before commiting, may be good to execute "pre-commit run --all" to make sure the code is compliant to pre-commit

Hi David,

I will reach out to you on discord for more info on this, when I open the command palette, I do not see the pypsa-earth environment in the select interpreter section

Hello!
Just to share, during the meeting, we discussed about this and now the problem is solved, right?

[ekatef]

Adding to the points above, there is also a need to update documentation on data management. Currently, the docs suggest to build a custom cutout, while currently we still have a number of pre-built cutouts which can be the first choice to avoid all the issue with Copernicus API registration.

[ekatef]

Hello @yerbol-akhmetov, hello @Netotse, as discussed yesterday, documentation update is getting crucial. Thank you for pushing this issue!

@Netotse your ideas on documentation are very much appreciated. Do you have any updates? Let me know please how can we support your work.

@yerbol-akhmetov thanks a lot for suggesting help. Please feel free to open a PR. Happy to assist, if needed.

[yerbol-akhmetov]

Hi, I have created a draft PR (#918). Currently working on installation section. I am using the reStructuredText plugin to preview the changes in .rst file. Although it works fine, in some cases it does not show the final version as it will be in documentation website. Could you please recommend any other plugins?

[ekatef]

Hi, I have created a draft PR (#918). Currently working on installation section. I am using the reStructuredText plugin to preview the changes in .rst file. Although it works fine, in some cases it does not show the final version as it will be in documentation website. Could you please recommend any other plugins?

Hey @yerbol-akhmetov, awesome! Thanks a lot for pushing this.

Have you tried to use shinx itself to build the documentation locally? To do so, you would need to install the dedicated docs environment which contains shpinx, and run sphinx with make html. More detailed instructions are in How to docs: https://pypsa-earth.readthedocs.io/en/latest/tutorial.html#compile-the-documentation-locally

[ekatef]

Copying here the ideas generated during the developers meeting and a discussion on Discord:

• Full model section needs more structure
• More clear instructions are needed to retrieve a cutout, an inventory of the available cutouts is needed
• A tutorial on work with time customisation can be considered. A cross-link to the config page may be a good idea.
• Advanced tutorial section can be used to explain the details about some advanced use cases. FAQ section may be considered. We can consider pandapower, Calliope, PyPSA for inspiration
• Split the “Full model” section on smaller parts like mini-lessons. Keep in mind a need to obtain a working model as fast as possible. Step-by-step approach can be used to show how to achieve particular goals relevant for use-cases of starting model usage. Links to the configs can be added to provide more technical details.
• Links to pypsa-earth videos can be added to supplement PyPSA-Eur video.

Thanks a lot @pz-max and @yerbol-akhmetov for the amazing discussion. Obviously, feel free to correct me and add points to the list.

[davide-f]

In the documentation, better in the full model representation, it would be important to add a section with a procedure to validate the model.
Many users need to validate their model and having a guidance on how to do that would be quite good.
FYI @yerbol-akhmetov and @ekatef

[ekatef]

In the documentation, better in the full model representation, it would be important to add a section with a procedure to validate the model. Many users need to validate their model and having a guidance on how to do that would be quite good. FYI @yerbol-akhmetov and @ekatef

Absolutely agree. Validation is definitely a crucial part right now, and would be great to facilitate it as much as we can. Currently, we have some cumulative experience of validation for a number of use cases, although it could be great to harmonise that. Mainly, the notebooks have been used for that. A list of the relevant notebooks may look as follows:

1. installed generation capacity;
2. electricity demand;
3. topology of the power transmission system for different voltages.;
4. power dispatch -- the final plot may be handy in sample_network_analysis notebook.

There is also an example of "holistic" validation approaches which relate to all the model components for Nigeria and Namibia.

[ekatef]

An update regarding a validation methodology: that is absolutely needed to include an approach to calculate the overall energy demand. That is one of the key steps of the validation, while that is not obvious at all how to account for the time step.

As a reminder, the needed magic looks like follows:

overall_load = (
    n.loads_t.p_set
    .multiply(n.snapshot_weightings.generators, axis=0)
    .sum()
    .sum()
)

[swaechchha]

Just want to add a small comment: The current link to IRENA's data in the "Full Model: Where to look for reference data" section leads to an error page. It's possible to navigate back to the main page from there. To enhance user experience, it would be better to update the link to the IRENA Statistics Main Page https://pxweb.irena.org/pxweb/en/.

[ekatef]

Just want to add a small comment: The current link to IRENA's data in the "Full Model: Where to look for reference data" section leads to an error page. It's possible to navigate back to the main page from there. To enhance user experience, it would be better to update the link to the IRENA Statistics Main Page https://pxweb.irena.org/pxweb/en/.

Hey @swaechchha, great, thanks a lot for the suggested fix 🙂 I can't suggest you to open a PR as there is still a big PR is under review to improve Full Model section... But we are happy to count you as a contributor, anyway!

[ekatef]

My feeling is that this issue is completely covered with #918 and #959. Can we probably close it?
@pz-max

[ekatef]

Have closed as it feels like completed with #918 and #959. Feel free to reopen if needed