docs: refactor and move to Quarto

[lostmygithubaccount]

subsumes #6512
related #3701
related #6511
related #6618

---

The purpose of this issue is to propose and track the implementation of Ibis documentation in Quarto.

There are several motivations for this work. The first is the existing documentation using mkdocs is slow to build and has a lot of minor issues. The most important reason for moving to Quarto is to improve the user experience of using Ibis documentation and key features like more native interactive Python support (in markdown files) offers both a better user and documentation developer experience. Additionally, numerous minor issues are fixed.

Timeline

We should aim to have fully moved our documentation over by the release of v7, but as soon as possible.

Priorities

P0

We must have these to consider switching to Quarto as main site.

• CI/CD for new docs process (in progress chore: quarto docs follow-ups #6937)
• home page finalized
• fix broken links (so many) (in progress chore: quarto docs follow-ups #6937)
• "Why use Ibis" article
• "What is Ibis" article
• installation page fixed
• backend pages fixed
• dplyr tutorial Quarto-fied
• how-to guides structure finalized
• how-to guides all Quarto-fied
• reference documentation finished
• release notes process updated (in progress chore: quarto docs follow-ups #6937)
• community documentation fixed
• AI tab removed or updated (try it out!)

P1

We should have these before merging.

• add metadata to all pages for SEO
• preview for docs changes on PR setup
• theme and font finalized
• redesigned logo (for new theme)
• quartodoc reference tool upstreamed (cc: @machow)
• community section updates (e.g. new async tool? documentation?)

P2

Nice to have before merge; likely follow ups.

• backlog of tips to post
• new concepts
• pyspark tutorial
• visualization tutorial (plotting libraries -- one or multiple)
• new how-to guides
• old posts Quarto-fied

Questionable items:

• automate unique image generation for posts?
• automate posts to social media?

Organization, structure, opinions

Open to feedback.

I have started porting the docs to Quarto here: https://github.com/lostmygithubaccount/ibis/tree/quarto-docs.

See the README in the docs2 directory for how to setup and preview the docs locally. The website has 5 primary sections mostly aligning with the https://diataxis.fr/ framework. Backends are important enough for their own section. Then, we follow the standard getting-started/concepts/how-to/reference split for the rest of the documentation.

The concepts are very light right now. It would be helpful to decide on the top-level structure here before writing all of the documents. I will take on drafting the "Why use Ibis" document based on recent discussions.

The how-to guides are also light and again I think we should decide on a top-level structure. These should be search engine optimized and contain useful code snippets a user is likely to want. We should section them by task, e.g. "file input and output", "eda", "ml", "timeseries". I don't love the current layout though.

We should limit backend-specific documentation to that section, i.e. disallow backend-specific how-to guides, to avoid sprawl.

All documentation should be written in .qmd files; we will convert old documents and if a user insists on PR'ing a notebook or otherwise we'll convert it over (this is easy and automatable). This format allows for significantly easier CI/CD to test examples (something both GCP and Azure have custom solutions in) by embedding the code directly in markdown. Modern IDE tooling (vim and VSCode, at least) make working with these files in what I would argue is a better experience than traditional notebooks. So, we will embrace this format for documentation to both improve the end user experience (less broken examples, better look) and developer experience (easier PRs).

More thoughts on AI in a separate issue. Try out the app though, it seems decent especially for getting started.

[machow]

Hey! I'm happy to PR in a reference section from this prototype of porting the ibis docs to python: https://github.com/machow/ibis-docs-demo.

Do you want me to open a PR against lostmygithubaccount:quarto-docs?

I'm also happy to pair, in case you have questions about translating anything to quarto / funkiness. I think an interesting migration piece will be this impala backend page, which embeds documentation on some classes / methods.

[lostmygithubaccount]

going to defer to @cpcloud, we are getting ready to merge into master soon (as a separate docs2/ subdirectory). not sure what would be best (PR to my branch or wait a bit for it to be merged, then iterate there)

the impala backend page was the only one that broke, I believe I changed it to a .txt file for now to avoid the issue

[cpcloud]

@machow Let's get #6911 in first, that way there's less stuff to review 😅

[lostmygithubaccount]

I'm going to reopen this and add a few items instead of opening new issues

[machow]

Glad to see this trucking along! Whenever the timing is right / useful for API docs, lemme know and I'm happy to PR in reference docs (or if you want to take a swing yourself at quartodoc, I can diligently fix anything that breaks 😅)

[lostmygithubaccount]

@machow I think the time is now! the docs2/ folder on master branch contains our new docs website. Phillip has one major update PR open (#6937)

[gforsyth]

To add one more TODO:

• - point release note title links to release page and not to a github compare tags

[cpcloud]

Bunch of notes from a doc review of https://ibis-quarto.netlify.app:

• link to backend docs instead of empty ibis.${backend}.connect() fixed in chore: render do_connect arguments and file support backend APIs using quartodoc #7007
• jim hates bouncing docs things, it's jarring
• code format the other python libs docs: a bunch of small fixes #6992
• docstring links @machow (ignore this for now, I will make a quartodoc issue about it) fixed in Quarto api interlinks #7013
• extra line at the top of internals docs: a bunch of small fixes #6992
• compilation section is very shouty docs: a bunch of small fixes #6992
• revisit internals for accuracy
• raw_support_matrix.csv needs to run in the render pipeline fixed in docs: a bunch of small fixes #6992
• product links fixed in chore: add product links to the various backends #7006
• prose line below badges at the top of backend qmds
• connection parameters docstring fixed in chore: render do_connect arguments and file support backend APIs using quartodoc #7007
• bigquery google cloud link notworking docs: a bunch of small fixes #6992
• sqlite tooltip extravanganza fixed in docs: a bunch of small fixes #6992
• remove http://localhost:8000/how-to/input-output/basics.html#backends-supported is not a valid link fixed in docs: a bunch of small fixes #6992
• round description is not rendered correctly (quartodoc issue)
• variable width font for types is ugly (quartodoc issue)
• syntax highlighting of output is weird (quartodoc issue) addressed in Quarto api interlinks #7013
• quartodoc autodiscover (quartodoc issue) fixed in chore: render do_connect arguments and file support backend APIs using quartodoc #7007
• fill out top level api docs
• documented selectors classes are not useful for most users
• rewrite docstring interlinks to qmd syntax (which is just regular md syntax) addressed in docs: fix self and external links #7017

[cpcloud]

I think I've gotten most of the low hanging fruit from ☝🏻

[machow]

Thanks for flagging the quartodoc ones -- I have the repo open and am adding interlinks now! (Then will look through the other ones).

Can you elaborate a bit on some of the points, like..

• syntax highlighting of output is weird
• quartodoc autodiscover

(Or point to a page in the netlify build where the problem shows up)

[cpcloud]

@machow Thanks for taking a look!

Here's a link to some strangely highlighted output from pivot_longer examples:

https://ibis-quarto.netlify.app/reference/expression-tables#examples-22

Here's an example:

[cpcloud]

quartodoc autodiscover

This one is a feature request to be able to highlight everything except some objects matching specific patterns. This was a nice feature of mkdocstrings that made it easy to not have to have a giant list of all the things you want to document, but a small list of the things you don't.

[machow]

Pivot longer

For the pivot longer example, the code is inside a markdown python code block, so I wonder if the weirdness has to do with the syntax highlighter (either quarto's default, or a custom one if set?)

Here's the part of the qmd for pivot_longer (code block at the bottom):

### pivot_longer { #ibis.expr.types.relations.Table.pivot_longer }

`expr.types.relations.Table.pivot_longer(self, col, *, names_to='name', names_pattern='(.+)', names_transform=None, values_to='value', values_transform=None)`

Transform a table from wider to longer.

#### Parameters

| Name               | Type                                                                                                                                                                                                                                                                            | Description                                                                                            | Default    |
|--------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|------------|
| `col`              | [str](`str`) \| [s](`ibis.selectors`).[Selector](`ibis.selectors.Selector`)                                                                                                                                                                                                     | String column name or selector.                                                                        | _required_ |
| `names_to`         | [str](`str`) \| [Iterable](`typing.Iterable`)\[[str](`str`)\]                                                                                                                                                                                                                   | A string or iterable of strings indicating how to name the new pivoted columns.                        | `'name'`   |
| `names_pattern`    | [str](`str`) \| [re](`re`).[Pattern](`re.Pattern`)                                                                                                                                                                                                                              | Pattern to use to extract column names from the input. By default the entire column name is extracted. | `'(.+)'`   |
| `names_transform`  | [Callable](`typing.Callable`)\[\[[str](`str`)\], [ir](`ibis.expr.types`).[Value](`ibis.expr.types.Value`)\] \| [Mapping](`typing.Mapping`)\[[str](`str`), [Callable](`typing.Callable`)\[\[[str](`str`)\], [ir](`ibis.expr.types`).[Value](`ibis.expr.types.Value`)\]\] \| None | Function or mapping of a name in `names_to` to a function to transform a column name to a value.       | `None`     |
| `values_to`        | [str](`str`)                                                                                                                                                                                                                                                                    | Name of the pivoted value column.                                                                      | `'value'`  |
| `values_transform` | [Callable](`typing.Callable`)\[\[[ir](`ibis.expr.types`).[Value](`ibis.expr.types.Value`)\], [ir](`ibis.expr.types`).[Value](`ibis.expr.types.Value`)\] \| [Deferred](`ibis.expr.deferred.Deferred`) \| None                                                                    | Apply a function to the value column. This can be a lambda or deferred expression.                     | `None`     |

#### Returns

| Type                                       | Description   |
|--------------------------------------------|---------------|
| [Table](`ibis.expr.types.relations.Table`) | Pivoted table |

#### Examples

Basic usage

```python
>>> import ibis
>>> import ibis.selectors as s
>>> from ibis import _
>>> ibis.options.interactive = True
>>> relig_income = ibis.examples.relig_income_raw.fetch()
>>> relig_income
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━┓
┃ religion                ┃ <$10k ┃ $10-20k ┃ $20-30k ┃ $30-40k ┃ $40-50k ┃ … ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━┩
│ string                  │ int64 │ int64   │ int64   │ int64   │ int64   │ … │
├─────────────────────────┼───────┼─────────┼─────────┼─────────┼─────────┼───┤
│ Agnostic                │    27 │      34 │      60 │      81 │      76 │ … │
│ Atheist                 │    12 │      27 │      37 │      52 │      35 │ … │
│ Buddhist                │    27 │      21 │      30 │      34 │      33 │ … │
│ Catholic                │   418 │     617 │     732 │     670 │     638 │ … │
│ Don’t know/refused      │    15 │      14 │      15 │      11 │      10 │ … │
│ Evangelical Prot        │   575 │     869 │    1064 │     982 │     881 │ … │
│ Hindu                   │     1 │       9 │       7 │       9 │      11 │ … │
│ Historically Black Prot │   228 │     244 │     236 │     238 │     197 │ … │
│ Jehovah's Witness       │    20 │      27 │      24 │      24 │      21 │ … │
│ Jewish                  │    19 │      19 │      25 │      25 │      30 │ … │
│ …                       │     … │       … │       … │       … │       … │ … │
└─────────────────────────┴───────┴─────────┴─────────┴─────────┴─────────┴───┘
```

edit: if there's something more sensible than python to use for the codeblock here, we can def change it

Excluding

That makes sense, I've put it off for a long time, but it's an easy add 😅. Do you have an example (either here, or from the past docs) you can share? That way I can make sure it hits your exact use-case

[machow]

Oh, hmm.. I guess for mkdocstrings it's in a code block, but maybe not necessarily a python one? (Or maybe a python one, but a different highlighter?)

From the current docs (it's a div with class "highlight", wrapping a code element)

[cpcloud]

@machow For now, is it possible to not syntax highlight examples (but keep them inside triple backticks to keep the monospaced font), or to have an option to turn that off?

[machow]

That's super doable--I'll add it in a custom renderer in ibis for now, and try to reproduce in a quarto issue

[cpcloud]

@machow I'm seeing a funky rendering of a markdown table embedded in an argument description of the NumericValue.round method:

Am I doing something wrong in the docstring?

Code is here: https://github.com/ibis-project/ibis/blob/master/ibis/expr/types/numeric.py#L69-L85

[machow]

Oh gosh, quartodoc escapes pipes, since in a description they could mess up the parameter table. We can toggle how the parameter tables get generated pretty easily using the _render_table(), and tinker with disabling the escaping (it's here in the renderer).

Here's part of the escaped description.

expression output type:  \|   `digits`    \| `self.type()` \|  Output   \| \| :

I think it's basically a situation where if we can figure out some format that would work in a qmd, we can tweak the renderer to produce it (even if just for this particular docstring, etc..).

[machow]

I'll be out for the next week, but wanted to at least summarize some of the lingering issues from the list above! I think the biggest one for me to loop back to and complete when I get back is the last one (implement filtering in quartodoc).

round description is not rendered correctly (quartodoc issue)

See above, I think we might need to figure out some format quarto will accept. If we could hand-code an example parameters table in a qmd that has this nested table, then I should be able to tweak the renderer to produce it.

variable width font for types is ugly (quartodoc issue)

I think this was raised by in machow/quartodoc#206, and machow/quartodoc#208 by the shiny team. Based on this comment, I think we should be able to wrap renderered annotations in a <code> element.

Here's the relevant code from shiny docs

quartodoc autodiscover (quartodoc issue)

Leaving breadcrumbs for myself. It looks like the filters are here: https://github.com/ibis-project/ibis/blob/master/mkdocs.yml#L79

[cpcloud]

Huzzah!

Thanks for your work @machow!

We are probably going to cut over to the new site at the end of next week, so nothing is blocking us at the moment. I think the rest of our changes are content changes and don't need anymore infrastructural adjustments.

For the embedded table, we may just convert that into a bulleted list for the time being.

[cpcloud]

Oh, and yes the variable width font issue is fixed!

[machow]

Awww yeah! If you hit other issues next week, and raise them on quartodoc, I can probably beseech some members of the shiny team to weigh in 🙃

[cpcloud]

Closed by #7061.

Thanks everyone!