Search all pages in Pages[] by default

[Martien]

I have pages[] defined in config. It seams that search only indexes the top level of pages[].

How can I have it index all pages in pages[]? That is, without having to add index: true to all individual files?

[Fil]

All the pages are indexed, by default. Please provide more details so we can understand what's happening in your case.

[Martien]

Ah, well, sorry about being so terse. Here you go.

Unlike https://observablehq.com/framework/, which has pages (leaves) at level 1 and 2 (root being level 0), my setup has most pages at level 3, as I have to include an extra indirection layer because I potentially have hundreds of pages.

For example, http://aarde.aardrock.nl/money shows a TOC of all pages relating to Money and is listed in the sidebar under ‘Aardbron’. One of these ‘Money’ pages is about Flow Money. It’s logical path is aarde/aardbron/money/flow-money, so at level 3.

Searching for ‘evil’ does find the Money category page, as it is on level 2. But searching for ‘geld’ (Dutch for money) only finds two pages, one of them being http://aarde.aardrock.nl/aardbron. This while there are about 50 pages that mention ‘geld’.

pages[]:

• are collected and exported in zones/zones.js;
• imports a handful of other pages[]; and
• is imported in observable.config.js

For instance, the ‘Money’ category used by ‘Aardbron’, containing the page ‘Flow money’ is defined in zones/aardbron.js

As another example, searching for ‘martien’ lists only four results, while there are 15 distinct pages that contain that text.

My initial thought is that only two levels are supported by both the sidebar and search. I understand and respect the choice for not supporting nested sections in the sidebar. However, I very much want ALL pages to be indexed so they can be found. A global option to turn this off or on would be fine.

Thoughts?

[Fil]

The pages option is a list of pages and sections — sections being lists of pages. There is no third level. The option is normalized here:
https://github.com/observablehq/framework/blob/main/src/config.ts#L440

Anything that you add to this option and does not fit this expected shape is ignored. If your site structure is different from the expected, you can add a search: true statement in the relevant pages’ front-matter. It's what I do in pangea.

[Martien]

Clear. Pages beyond level 2 are ignored, which I also understand from https://observablehq.com/framework/config#pages: “(Sections can only contain pages; nested sections are not currently supported.)”.

I have trouble understanding the rationale behind this decision. Can you please enlighten me here?

Also, what is your advise for sites with hundreds of pages? How to best organise this?

Currently, I see two options:

1. add index: true to the front matter of all pages;
2. flatten the pages hierarchy to two levels only, which makes the sidebar very wide (long); probably too wide to be practical.

How can I best create a enhancement request for normalizePages and normalizeSection to traverse the a tree of any depth rather than just two levels?

[Martien]

How much sense does this make? For a depth-independent version, see end of https://chatgpt.com/share/675ef22f-93e4-8011-bf05-de9f06253109

[Martien]

@Fil, can you please share the reasoning behind ignoring pages beyond level 2?

[Fil]

We normalize the config to whatever is supported. Since we only support two levels, that's what happens here. If we wanted to support more levels, then we'd normalize differently. Supporting a third level for the pages config option would be a large amount of work.

Another approach could be to add a config option to say whether we should index a page that is unlisted (not present in pages), and has no explicit index param in its front-matter. Currently this defaults to false:

framework/src/search.ts

Line 84 in 6d455ac

const indexed = data?.index === undefined ? listed : Boolean(data.index);