Edit/synthesize introductory materials - Manual into MD for RTD

[sbenthall]

There are several different, but overlapping, introductory materials to HARK.
Identify these materials and their differences/similarities.

Edit these for more streamlined onboarding.

Use other scientific software toolkits (astropy, etc.) as a model for how to write effective onboarding documentation.

Assigned to @sbenthall -- task defined in today's roadmapping meeting.

[sbenthall]

See related tickets: #407 #362 #223

[sbenthall]

I've partially mapped out the resources that people are directed to when beginning to use HARK.

I've rooted the search in the HARK README.

Main takeaways:

• The web documentation (readthedocs) is buried deep within the README and easy to miss, and not pointed to directly from the Econ-ARK website. If we want these docs to be the go-to location for tutorial information, they need to be featured much more prominently.
• Most things point to the Gentle Introduction to HARK notebook one way or another. However, there are multiple versions of this notebook distributed across different repositories. The DemARK version is linked via the website and the README. But the documentation currently embeds a copy of the Gentle Introduction notebook from the HARK repository.
• There are multiple legacy resources still floating around. The website and (indirectly) the README both point to the PDF manual (which, by the way, I thought was a very nice read). There is also an old Quickstart guide in the HARK/Documentation/readme.md

Based on this survey, I think the next steps are:

• Decide if readthedocs is going to become the main place where new users are going to be directed first. If so, make this much more prominent in the README and on the Econ-ARK website.
• Decide which Gentle Introduction to HARK notebook is the canonical one, direct users consistently to it, and clean out the other versions.
• Remove links to deprecated documentation versions.

[sbenthall]

It looks like there are many automatically generated files checked into the repository here:
https://github.com/econ-ark/HARK/tree/master/Documentation/HARKmanual

Generally, it is best practice to not check autogenerated content into the repository.

Is there a positive reason why this is there, @llorracc, or may I clean these files out?

[sbenthall]

There is yet another file with a different version of introductory doc materials:

https://github.com/econ-ark/HARK/blob/f4684aae27bf7e246b396149da57f03e51ce40aa/Documentation/readme.md

This one is the primary introduction used in ReadTheDocs.

After PR #411, this readme.md is not getting into the generated readthedocs index. This may be due to the removal of recommonmark. In a sense, it's my fault; though I'm not sure how it was working last time the docs were updated, which was 3 months ago.

• I need to get this readme.md rendering into the readthedocs working again.

[llorracc]

It looks like there are many automatically generated files checked into the repository here:

Is there a positive reason why this is there, @llorracc, or may I clean these files out?

There is a reason, but one I am willing to consider abandoning. The original version of the manual was a LaTeX document, because that let us copy and paste the math from previous work. At various times there were quick demands for:

1. A markdown version of the manual;
2. A markdown excerpt of just the part of the manual about contributing;
3. An html version of the manual.

From the beginning I'd been storing the generator version of everything in the HARK-make repo. Scripts in that repo would run to generate the content you found. You will find four reasonably self-explanatory scripts in HARK-make:

makeWeb-HARKmanual-And-Contributing-To-HARK.sh
makeWeb-Simple-HARKmanual.sh
makeWeb-Simple-HARKmanual-ContributingToHARKOnly.sh
makeWeb-Simple.sh

Your goal, in a sense, is to make that process obsolete. If we have one unified system of documentation with everything properly interlinked, then there will be no need for generated material and we can remove this content from HARK-make. (And maybe even eliminate HARK-make altogether!).

[llorracc]

Two other examples of introductory material are at:

#380 (comment)

• This was constructed last summer by a student of Pablo's who is now in the PhD program at the University of British Columbia. He was completely new to HARK, but is very bright, so what he wrote is probably a good readout of what a person in the target audience of "first year PhD student" might understand after a month or so of trying to write an intro guide

Another effort, specifically to trace out the logic of our solution method for the basic model, was made by Karsten Chipeniuk [email protected]

https://github.com/econ-ark/DemARK/tree/b009a0922a540559274e4e5718de27a5b3d8c6fb/notebooks/Quickstart_tutorial

At some point, I asked Patrick to try to incorporate Karsten's notebook in the documentation but I don't think that ever happened.

[sbenthall]

Another wrinkle, RE: this point: I have checked to see if the current master branch compiles the docs properly, with readme.md, in ReadTheDocs, which uses a differently configured documentation building environment than the one I'm using.

And it works!

Which means that locally building the docs is sensitive to some (undocumented) environmental configuration, but also that the issue I flagged earlier about the readme.md not compile was a non-issue.

[sbenthall]

@llorracc So is it indeed the case that goal here is to have ReadTheDocs be the canonical, go-to documentation, and everything else will be deprecate when ReadTheDocs is adequate?

[llorracc]

My sense is that there are several different modes in which people will be interacting with the toolkit, and that the most useful way to help them will be different for each mode.

1. Somebody is trying to figure out how some existing code is working or what it is doing
   • For this, making it easy to quickly get to the relevant part of the ReadTheDocs autogenerated material is likely the best approach -- especially if, when they go to the relevant part of ReadTheDocs, they can discover other examples of how the tool in question works. (This is something that already happens, partly, with the inclusion of our DemARKs in the ReadTheDocs build)
2. The user is a student trying to learn substantive material about a particular topic in economics
   • This will likely be in the context of using one of our tutorial or DemARK notebooks
   • In this scenario, having access to the docstrings of the functions may not be all that helpful. "Best practice" here is to include, in the relevant notebooks, references to fuller treatments of the substance (like links to relevant papers or Wikipedia articles or lecture notes).
   • One resource we have created to make it easier for us to construct such references is our Zotero bibliography of papers and other resources we cite. This has been integrated with the cite2c tool that allows us to embed citations in notebooks
3. The user is a student in a "computational methods" course and is trying systematically to learn the elements and principles of building and solving models like those in the toolkit
   • This is where we are the weakest at present. The TITLARK repo contains syllabi for some mini-courses I have taught over time, using the toolkit, but we do not have anything like a structured or orderly guide to the toolkit itself
   • There must be good examples out there of structured course material for learning to use other toolkits
   • We are not at the point of being able to build much of this kind of thing yet, but that's partly because we have not identified a model of how to do it that we want to emulate.
   • This is where it would be particularly valuable to have you do a fairly extensive (and, admittedly, time-consuming) deep dive into what new things other toolkit projects are doing that look worth emulating
     • My sense is that this is the area where the greatest innovation is happening, in particular in the use of Jupyter notebooks. Older toolkits may have some nice static tutorials that were created 5 or 10 years ago, but that's probably not what we want to be doing nor what anybody else will be doing 5 or 10 years from now.

In the short run, probably the right strategy is indeed just to consolidate the material that we already have into a combination of ReadTheDocs and DemARK notebooks that are auto-integrated with ReadTheDocs. But it would be nice to have a vision for how we want to go about building a structure to guide people in a systematic and step-by-step way into the toolkit.

One resource I'd particularly like you to dive into is this symposium from the PASC conference this summer. There was a lot of good advice from all of the presentations, and we would profit from adopting many of the ideas mentioned in those presentations.

[llorracc]

One other comment: A while ago in some of the discussions between @sbrice and @shaunagm about restructuring and rationalizing everything, @sbrice suggested that the notational conventions defined in the NARK were really also a form of documentation, and should be integrated somehow (so that people could figure out what the various variables' cryptic names mean).

[sbenthall]

Ah, ok.

So, if I can try to rephrase:

• DemARK is for the basic economics students. It's meant for the classroom use case.
• ReadTheDocs is for software users/contributors. See Wrangle contributor guidelines #410. These should follow the idioms of open source software documentation, in general.
• There is a third use case that is the "computational methods" course. There are open questions here about pedagogy, research design, and software design, which lines up with broader problems of the ways scientific software is changing higher education institutions, etc.

The third point is a broad one--so broad it probably doesn't fit into a software issue tracker ticket :) Looking forward to discussing it with you some time.

I'll consider this ticket scoped to the software users/contributors use case.

[sbenthall]

HARK-make

Digging a bit deeper (because of #420), I'm looking at the HARK-make repository.

I understand why these scripts are valuable.
I wonder if it would make more sense to:

• move the shell scripts that make the different versions of the documentation from the source files (latex) into HARK
• remove the files generated by the scripts

The logic here is that if somebody changes the manual, they should change only the source files.
Having to update all the files downstream of the source is wasteful.

(The generated files are also making the Documentation directory more confusing--not just for me, but apparently also for the sphinx documentation compiler)

[llorracc]

The logic here is that if somebody changes the manual, they should change only the source files.
Having to update all the files downstream of the source is wasteful.

Well, if the downstream files are all autogenerated from a bash script, the degree of waste (from the perspective of the only person editing the files, which was me) is small (when there is a Single Source of Truth which is the tex file) compared with trying to edit several different (LaTeX; markdown; html; RST) versions of the doc.

There's nothing I like more than an SST. So, if I understand your point, it is that we should move toward an .rst version of HARKmanual that is our SST? (Esp given that the markdown generated by pandoc seems to have messed up Sphinx).

Sounds like that's probably a good idea. But also sounds like it would require at least a little work to translate HARKmanual.tex to HARKmanual.rst? At least, if running pandoc on HARKmanual.tex yields an RST file that is as deficient as the pandoc-generated markdown file was ...

[sbenthall]

So, if I understand your point, it is that we should move toward an .rst version of HARKmanual that is our SST?

My point was more of a principle:
It's best if a repository has: (a) data in a usable raw form (SST), and (b) scripts for transforming it into other forms, as needed.
Rather than several versions of the same content in slightly different forms.

(In my view, if the scripts have only a single function, it's best to keep them in the same repository as the data they ingest.)

If, say, the SST is LaTeX, and the script generates Markdown that is unusable, then I think it's best to consider that a bug in the script, rather than a problem with the Markdown to be edited by hand.

I don't understand the ecosystem of documentation, the toolchains around them, or the directions of the project well enough yet to have an opinion of whether the manual SST should be in LaTeX, RST or MD.

[llorracc]

If the “HARKmanual is a LaTeX document that gets automatically converted to md, html, and RST using pandoc“ procedure worked flawlessly, I don’t know that there would be a reason to change things. But you indicated that the markdown it generated was flawed, and I think there are problems with the RST and html as well. (Basically, pandoc does as good a job as it can in converting, but that’s still not such a great job). The case for moving to an RST version of HARKmanual would probably be strengthened if any of several other conditions were true: 1. We decided that it would be best to split the content currently in HARKmanual up into more digestible chunks, which can be linked to each other 2. Converting it to RST-native form would allow useful integrations with ReadTheDocs 3. RST is essentially becoming the universal norm for documentation and we would increasingly be an outlier in having a PDF manual Could you take a look at how QuantEcon does their documentation? Generically I am an admirer of everything they have done, and if they are an all-RST shop that would be pretty persuasive. (In my view, if the scripts have only a single function, it’s best to keep them in the same repository as the data they ingest.) The problem with that is that LaTeX adds a lot of confusing clutter; this is a reason to move toward RST because it renders natively without any need for compiling. Let’s discuss this tonight or over the weekend. On Fri, Nov 1, 2019 at 10:03 AM Sebastian Benthall [email protected] <http://mailto:[email protected]> wrote: So, if I understand your point, it is that we should move toward an .rst

version of HARKmanual that is our SST? My point was more of a principle: It's best if a repository has: (a) data in a usable raw form (SST), and (b) scripts for transforming it into other forms, as needed. Rather than several versions of the same content in slightly different forms. (In my view, if the scripts have only a single function, it's best to keep them in the same repository as the data they ingest.) If, say, the SST is LaTeX, and the script generates Markdown that is unusable, then I think it's best to consider that a bug in the script, rather than a problem with the Markdown to be edited by hand. I don't understand the ecosystem of documentation, the toolchains around them, or the directions of the project well enough yet to have an opinion of whether the manual SST should be in LaTeX, RST or MD. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#409?email_source=notifications&email_token=AAKCK75PYLUNQI6JN4DDSUTQRQZLZA5CNFSM4JDEV432YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC3AE7A#issuecomment-548799100>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK72DMDHUHY4HS7WALQ3QRQZLZANCNFSM4JDEV43Q> .

-- - Chris Carroll

[sbenthall]

• autosummarize only top 2 levels (i.e. HARK.core) in the API docs

[sbenthall]

Could you take a look at how QuantEcon does their documentation? Generically I am an admirer of everything they have done, and if they are an all-RST shop that would be pretty persuasive.

I've looked into it. Indeed, they seem to use RST for everything.
They do have some mathematical notation in their docs, but this seems to be rendered from RST through a LaTeX renderer.

For example:
https://quanteconpy.readthedocs.io/en/latest/markov/core.html#definitions-and-some-basic-facts-about-markov-chains
https://github.com/QuantEcon/QuantEcon.py/blob/master/quantecon/markov/core.py#L5

[llorracc]

Thanks. We should think about whether we should follow them or not. We don't have a huge amount of documentation in LaTeX, so maybe it would not be too painful (esp since the pandoc converter would produce a first draft). PS. I really like their hierarchical organization of the material.

…

On Thu, Nov 7, 2019 at 2:44 PM Sebastian Benthall ***@***.***> wrote: Could you take a look at how QuantEcon does their documentation? Generically I am an admirer of everything they have done, and if they are an all-RST shop that would be pretty persuasive. I've looked into it. Indeed, they seem to use RST for everything. They do have some mathematical notation in their docs, but this seems to be rendered from RST through a LaTeX renderer. For example: https://quanteconpy.readthedocs.io/en/latest/markov/core.html#definitions-and-some-basic-facts-about-markov-chains https://github.com/QuantEcon/QuantEcon.py/blob/master/quantecon/markov/core.py#L5 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#409?email_source=notifications&email_token=AAKCK7ZJ3T6L35WD55UDT7LQSRV2ZA5CNFSM4JDEV432YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDNSUQY#issuecomment-551234115>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK756X6SHVAEVELK5YYTQSRV2ZANCNFSM4JDEV43Q> .

-- - Chris Carroll

[sbenthall]

Ok, then I'll add a task:

• Render the manual into working Markdown and include this material in readthedocs, then remove links to manual from the new user/contributor onboarding path.

[sbenthall]

Working towards this, I'm comparing what is in the Manual with what is already in the ReadTheDocs.

There are a number of areas where the content overlaps. It would not make sense to copy the entire manual into ReadTheDocs.

These are sections of the Manual that have roughly equivalent sections in the docs already:

1.1 Getting Started
2 General Purpose Tools. Corresponds to https://github.com/econ-ark/HARK/blob/master/Documentation/list-of-files.md
5. Contributing to HARK . This also has an issue devoted to it #410

Sections that have no equivalent in the current ReadTheDocs are:

1.2 Structure of HARK
1.3 Other Resources
3. Microeconomics: the AgentType Class
4. Macroeconomics: the Market Class
6. Future of HARK

[llorracc]

I don't think we had settled on a solution to the "manual problem." The thing that needs to be immediately fixed is just that the page should not break. Somehow our readthedocs software had incorporated the content of the html file, and for some reason it is no longer there. That does not necessarily imply that the solution is convert HARKmanual to rst". Actually, having discussed with DrDrij a few days ago, I'm more inclined to move toward markdown rather than rst, if readthedocs can index markdown. In any case, there should be an immediate short-term fix (relying on either the .html or the .md versions of the manual) that will work until we agree on a permanent plan for HARKmanual.

…

On Mon, Nov 25, 2019 at 5:08 PM Sebastian Benthall ***@***.***> wrote: Working towards this, I'm comparing what is in the Manual with what is already in the ReadTheDocs. There are a number of areas where the content overlaps. It would not make sense to copy the entire manual into ReadTheDocs. These are sections of the Manual that have roughly equivalent sections in the docs already: 1.1 Getting Started 2 General Purpose Tools. Corresponds to https://github.com/econ-ark/HARK/blob/master/Documentation/list-of-files.md 5. Contributing to HARK . This also has an issue devoted to it #410 <#410> Sections that have no equivalent in the current ReadTheDocs are: 1.2 Structure of HARK 1.3 Other Resources 3. Microeconomics: the AgentType Class 4. Macroeconomics: the Market Class 6. Future of HARK — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#409?email_source=notifications&email_token=AAKCK72YRWHQ37ESXLSAUATQVREERA5CNFSM4JDEV432YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFD62II#issuecomment-558361889>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK75Y5LENYXBSBLZXGOLQVREERANCNFSM4JDEV43Q> .

-- - Chris Carroll

[sbenthall]

By 'break', do you mean getting a 404/"Sorry, This page does not exist yet" when following a link from the Quick Start guide?

[sbenthall]

The short term fix I've tried to implement in PR #436 is that this link would point to the PDF version of the manual.

But now I'm trying to rebuild the readthedocs documentation and the build fails. (It works locally on my machine, but this isn't the first time local behavior and readthedocs.io have been different).
I'll need to work on fixing that next.

[sbenthall]

This is the current error. It appears to be a problem with the nbsphinx plugin:

python /home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/bin/sphinx-build -T -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html
Running Sphinx v1.6.7
making output directory...
Inserting /home/docs/checkouts/readthedocs.org/user_builds/hark/checkouts/latest
loading translations [en]... done

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/lib/python3.7/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/lib/python3.7/site-packages/sphinx/application.py", line 196, in __init__
    self.setup_extension(extension)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/lib/python3.7/site-packages/sphinx/application.py", line 456, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/lib/python3.7/site-packages/sphinx/registry.py", line 207, in load_extension
    metadata = mod.setup(app)
  File "/home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/lib/python3.7/site-packages/nbsphinx.py", line 1766, in setup
    app.add_source_suffix('.ipynb', 'jupyter_notebook')
AttributeError: 'Sphinx' object has no attribute 'add_source_suffix'

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/hark/envs/latest/lib/python3.7/site-packages/nbsphinx.py", line 1766, in setup
    app.add_source_suffix('.ipynb', 'jupyter_notebook')
AttributeError: 'Sphinx' object has no attribute 'add_source_suffix'
The full traceback has been saved in /tmp/sphinx-err-42nbiwkf.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

[sbenthall]

Six days ago there was an nbsphinx release, 0.5.0. RTD is giving the following error when it installs this package

ERROR: nbsphinx 0.5.0 has requirement sphinx>=1.8, but you'll have sphinx 1.6.7 which is incompatible.

I'll try pegging the version of nbsphinx to the earlier version.

[sbenthall]

See #380

[sbenthall]

Reviewing this ticket---the task that remains on it is converting the manual to Markdown and including in ReadTheDocs.

[llorracc]

@sbenthall, @MridulS I've used my newly-acquired knowledge of how to use gh-pages for rendering html, to rerun and post the html version of the HARKmanual.

The natural place to put this would be in the gh-pages branch of HARK. But, per my message earlier today, that seems to contain an ancient and defunct version of the documentation. Since both of you are now much more expert than I am at using sphinx etc, I defer to you to clean that up and remove the old docs. In the meantime, I've put the recompiled version of HARKmanual.html at

https://econ-ark.github.io/HARK-make/Documentation/HARKmanual/

At a minimum, this may be a better starting point for converting into Markdown than the original LaTeX document. In fact, given that it does not have the glaring deficiencies of the HARKmanual.md (bad links, etc), it seems to me not a high priority to convert into markdown at all right now. @sbenthall said recently that he had started to do the conversion by hand but did not get far before he stopped. I'd say to wait on that until some more comprehensive planning has been done. For example, we may well just want to chop the HARKmanual up into bits and pieces and put them in the relevant parts of ReadTheDocs. In that case there would be no reason to do a hand-conversion to a HARKmanual.md file.

So my short-term proposal is that, when the old bad gh-pages/sphinx documentation is gone, to post the new HARKmanual.html at HARK/Documentation/HARKmanual/, update links in whatever are the appropriate place, and leave it be until we have time to return to it more systematically.

https://econ-ark.github.io/HARK-make/Documentation/HARKmanual/

[sbenthall]

Ok, I see the manual is now up
http://econ-ark.github.io/HARK/Documentation/HARKmanual/

I'll change the links in RTD to point to this manual.

[sbenthall]

The manual appears to have found a happy place to be: in the github.io pages.

Future documentation work is going to focus on ReadTheDocs.
So I'm going to close this ticket.