New Documentation Structure

[cgbotta]

As discussed with @yebai, we are porting all the md files in the "Using Turing" section of the public-facing site to jmd. This entails porting everything from https://github.com/TuringLang/Turing.jl/tree/master/docs/src/using-turing to TuringTutorials/docs. We are going to mimic the structure of the TuringTutorials/tutorials directory.

I began this process by adding a source_folder argument to the weave functions so that they can be used for weaving both tutorials and the new documentation (docs) folder. I also ported the first lesson.

Open questions I have are:

• I have not opened any PRs for the Turing.jl repo - what changes need to be made on that end to recognize and render the new documentation structure on the site?
• I will also update the README instructions for weaving once this pattern is finalized

[yebai]

@devmotion how do we add a new folder (e.g. TuringTutorials/docs/) to the build workflow?

[yebai]

Would these fields of permalink and redirect_from still work with jmd?

[cgbotta]

I copied this structure from all the files in TuringTutorials/tutorials, but I am assuming this is read by the build process. Perhaps @devmotion could also advise here?

[devmotion]

That's not relevant for weaving at all IIRC, it's just copied to the Markdown output (e.g., https://github.com/TuringLang/TuringTutorialsOutput/blob/main/markdown/00-introduction/00_introduction.md). This is just used by the Jekyll setup in turing.ml AFAIK.

[yebai]

Thanks @cgbotta it looks good overall. I think some additional work is needed to fix the weave workflow. By default, if we call weave it should process all jmd files inside docs and tutorials.

[devmotion]

@devmotion how do we add a new folder (e.g. TuringTutorials/docs/) to the build workflow?

To the buildkite CI? One has to change the configuration in https://github.com/TuringLang/TuringTutorials/tree/master/.buildkite, the structure in https://github.com/TuringLang/TuringTutorialsOutput/ (everything has to be moved to subfolders /docs/ and /tutorials/ I assume), and therefore also the integration in https://github.com/TuringLang/turing.ml.

I wonder if it would be simpler, and docs and tutorials more cleanly separated, if one would use a separate repo instead of multiple subfolders. Or if one should just move everything into /docs (or /tutorials, whatever seems more reasonable).

[yebai]

Or if one should just move everything into /docs (or /tutorials, whatever seems more reasonable)

I also think it is simpler to setup and maintain to have both docs and tutorials under one new folder (e.g. docs). We can distinguish between them by setting up the navigation bar.

[cgbotta]

Thank you both for the input - I agree with one main docs folder

Is there a preference for me to port all of the md docs to jmd in this PR, or a separate PR for each one?

[yebai]

Is there a preference for me to port all of the md docs to jmd in this PR, or a separate PR for each one?

Let's setup the workflow first. We can have a few more additional PRs to transfer all other doc md files.

[devmotion]

I think it would be good to change the structure (ie move tutorials to docs) first before adding any new pages (ie even without the one in this PR), since that also requires downstream changes.

[cgbotta]

After looking through the build process, there is a fair bit of refactoring involved across many files to change the name of the folder from tutorials to docs. Of course, I am more than happy to do that if we decide it is best. However, I wanted to discuss the changes I made in the latest commit because it seems like a simpler solution.

By leaving the folder named tutorials, I could just append "tutorial-" to the existing files, then the documentation files can begin with "docs-".

All that should be needed after this is to update the navigation bar. We could test it by only changing the first item https://turing.ml/v0.21/docs/using-turing/get-started

[devmotion]

If we keep the /tutorials folder, couldn't we just not rename the existing tutorials? tutorials/tutorials-00-... seems a bit unnecessarily complicated.

[cgbotta]

Good point - I am all for simplicity here 👍 Perhaps tutorials/docs-00-... is still worthwhile, but I agree that writing tutorials again is not necessary. Or if it really doesn't matter, I will just keep incrementing and start with tutorials/15-getting-started and not worry about distinguishing between old and new anywhere other than the nav bar

[yebai]

Perhaps tutorials/docs-00-... is still worthwhile

That sounds like a good tradeoff between minimal disruptions and being descriptive for now. Let us be incremental here, we can rename the tutorials to reflect a new naming scheme when things settle down a bit.

[devmotion]

Buildkite reports an error:

Unable to find project for tutorials/docs-00-getting-started/00_getting-started.jmd

Could you add Project.toml and Manifest.toml (with Julia 1.6) and add the folder to the CompatHelper Github action?

[yebai]

@cgbotta it seems the build is broken. https://buildkite.com/julialang/turingtutorials/builds/185#018414e3-f154-4902-a5b3-f01bacda6dd8

[cgbotta]

@cgbotta it seems the build is broken. https://buildkite.com/julialang/turingtutorials/builds/185#018414e3-f154-4902-a5b3-f01bacda6dd8

I was just having a look at that. I see this:

/etc/buildkite-agent/plugins/github-com-staticfloat-coppermind-buildkite-plugin/lib/common.sh:
 line 11: /etc/buildkite-agent/plugins/github-com-staticfloat-coppermind-buildkite-plugin/lib/bash-tools/lib/buildkite.sh: No such file or directory

Not quite sure what that is implying is wrong, but I am digging through some of the build kite pipelines. Could it be an issue with the commit a week ago that added coppermind v1?

[yebai]

I think that's correct. Updating this PR with changes from the master branch seems to fix it.

[yebai]

@cgbotta Thanks for the updates. Let's replace the moment-matched Normal approximation to the posterior with a KDE estimation -- see the density function from MCMCChains.

[cgbotta]

[cgbotta]

@cgbotta Thanks for the updates. Let's replace the moment-matched Normal approximation to the posterior with a KDE estimation -- see the density function from MCMCChains.

Thanks Hong - forgive me if this is obvious, but that be show anything different from the posterior densities given when running plot(chn) a bit earlier in the tutorial?

[yebai]

Thanks Hong - forgive me if this is obvious, but that be show anything different from the posterior densities given when running plot(chn) a bit earlier in the tutorial?

I think plot will show multiple things, including the density plot. So, yes, the plot function is the same. But, notice that the samples from the analytical posterior are exact -- we know how to sample from a Normal-Inverse-Gamma distribution in closed form. So we are using the KDE desity plot to compare the exact samples against MCMC samples. If the MCMC samples converge well, they should match the exact samples. Does that answer your question? If not, we can chat a bit more when we meet.

[yebai]

Thanks, @cgbotta!