How to automatically manage and make multiple (sub)Frameworks discoverable?

[dleeftink]

As I've settled on Framework for my blog, I am finding that rebuilding the whole site after a minor update to a single .md file can feel a little excessive (even though its not many pages now, I am thinking about the case of fully rebuilding 100+ pages each time). I am considering breaking out various blog categories into a separate repositories, and then link to to their contents from a main index page.

I wonder what would be an appropriate solution to (automatically) making the pages of subsites/Framework projects discoverable by the main page? A simple routing structure would resemble something like the following, with the root being the main index repository and each second-level (posts, notebooks, etc.) entry a separate repo.

.
└── root/
    ├── posts
    │   ├── 2024
    │   │   ├── Jul
    │   │   │   └── ..
    │   │   └── Jun
    │   │       ├── ..
    │   │       └── ..
    │   ├── 2023
    │   │   ├── Aug
    │   │   │   ├── ..
    │   │   │   └── ..
    │   │   └── Mar
    │   └── 2022
    │       └── Feb
    ├── notebooks
    │   ├── 2024
    │   │   └── Jun
    │   └── 2023
    ├── articles
    │   ├── 2021
    │   └── 2019
    └── prototypes
        └── 2024
            ├── Mar
            └── Feb

[Fil]

First I'd argue that it might be early to give up on rebuilding the whole site. What if you have 1k, 10k, 100k, 1M pages? How often would you touch these pages? What is the pain point/threshold that you'd find acceptable in terms of building/indexing/deploying time?

fwiw I'm regularly rebuilding and redeploying pangea (412 pages as of today); it generally takes about 50s, which in my book is "noticeable, but bearable". Of course, assuming this scales ~ linearly, I'm not sure I'll want to wait 10 minutes for a typo fix when I have 5,000 pages, and an hour when I have 25k.

But I think there's a lot of room to make builds much faster. The process is currently quite "stupid", in that it rebuilds all the HTML and the search index from scratch (only dependencies and data loaders outputs are cached). There would probably be a way to make rebuilding an order of magnitude faster by reusing a built page when we can guarantee nothing has changed. Another improvement could be to parallelize the build. We just haven't worked on this yet. Deploying is already very fast as it compares the list of files to deploy with the list of files already deployed and just sends the updates.

Also, if your site is separated into multiple small sites, you'll have to find a way to regroup and reconcile the minisearch indexes (assuming you want search!) You might be interested in an experiment here where I mix indexes from 7 sites using either Vitepress or Framework (since both generators use Minisearch under the hood) for an integrated search.

[Fil]

@jaanli Maybe I'm missing the point, but I'm not sure how using vike or next.js can help with Framework? They seem to be more like alternatives to Framework.

I'd like to clarify one difference. Having a ton of similar pages (HTML pages) that only differ with a few parameters stemming from a database is a very different problem from having a ton of Observable Markdown files. Since the latter can all have very different requirements, require different npm modules, create file attachments etc., they must be processed one by one, which is necessarily different (and certainly much slower) than a repetitive task where we just need to parse the template once then replace a few placeholders by actual data.

I can't really think of a project where you'd write a million different Framework pages, in fact, and find it more likely that you'd want the same page (or small lot of pages) to be replicated a million times, one for each data point. Say, for an Atlas of all the cities in the world, or a catalogue of all the stars in the galaxy. Parametrized data loaders and pages will in the future help with this, and are discussed in #1523.

If I wanted to make such a website now, I might prototype a few pages with Framework (with maybe different approaches for small and large cities, etc), then use the build as a template and run it in a tight loop against my database, maybe regenerating only the pages that correspond to fresh data.

[jaanli]

Thank you so much for your patience in helping clarify my naïve thinking @Fil !!!

Maybe I am confused about whether Observable Framework generates static HTML pages?

We both need a ton of markdown files and a ton of pages :(

That is a critical requirement for our search engine optimization and why alternatives like Vike and Next.JS have been on the table.

(If I misunderstood this, my bad! Then I need to edit #855 (reply in thread) as well which it seems a few other folks need by now :)

[Fil]

Framework does indeed publish static files, but in my understanding what we call "pages" in this context is more like the markdown source than the HTML output.

The OP by @dleeftink was about handling a blog with hundreds or potentially thousands of pages—I suspect they will be authored one by one as markdown files. Your requirement is "millions of pages" in the HTML sense, with an example that seemed to be pulling data from a database, and creating an HTML file for each datum.

We want Framework to be able to address both needs, but I wanted to clarify that these are two different classes of problems. If we make each of these faster, then "a ton of markdown pages" times "a ton of parameters for each page" will be faster as well.

Of course the Jevons paradox will apply and once millions of pages are fast, we'll want to create billions of pages, which will be slow again — but let's not anticipate too much.

[jaanli]

Touché @Fil - we actually needed trillions of pages! That makes sense - thank you so much for clarifying and we will hold our horses for now and do our best to help optimize the code.

The use case on our end is that we require 1 dashboard per legal entity in the United States.

For example, there is a map at https://www.payless.health/maps with all of the hospitals in the US (around 5-6k right now), and within each hospital there are several dashboards: https://www.payless.health/hospital/newyork-presbyterian that are each generated from the data we have published on 4000+ hospitals so far: https://data.payless.health/#hospital_price_transparency/ (1 data loader per hospital in Observable Framework terms :)

The fastest way to do this has been with @uwdata's Mosaic (https://github.com/uwdata/mosaic) and https://vike.dev/ so far.

We will likely use @pallets' Jinja for templating these dashboards: https://github.com/pallets/jinja alongside @leycec's @beartype (https://github.com/beartype/beartype) and @patrick-kidger's work on jaxtyping (https://github.com/patrick-kidger/jaxtyping - also compatible with numpy/scipy/pytorch for runtime static type checking of data loader input and output for CI/CD on our GitHub actions pipelines or TPU training scripts)

cc @Kariina for visibility.

[dleeftink]

The OP by @dleeftink was about handling a blog with hundreds or potentially thousands of pages—I suspect they will be authored one by one as markdown files. Your requirement is "millions of pages" in the HTML sense, with an example that seemed to be pulling data from a database, and creating an HTML file for each datum.

That's right @Fil, even though I don't imagine myself that prolific a writer, my concern has to do with splitting out potential pages based on type (long-form vs dashboardy-type content), where some pages may not need a long build process as there are little imports/data loaders to be resolved.

I think the idea of main frames and sub frames is worth discussing here, as this way dependencies can be neatly contained in individual repos (mitigating the need of having to manage different languages in a monorepo) while build steps won't have to touch anything outside their project scope.

If the published sub site endpoints are openly accessible, this opens the door to distributed collaboration, global search and aggregation of community content as well (configurable and opt-in, of course).

[leycec]

1 dashboard per legal entity in the United States.

That's... a whole lot of dashboarding. 😄

@leycec's @beartype and @patrick-kidger's work on jaxtyping (https://github.com/patrick-kidger/jaxtyping - also compatible with numpy/scipy/pytorch for runtime static type checking of data loader input and output for CI/CD on our GitHub actions pipelines or TPU training scripts)

This is the way. Just ping me with an issue on the @beartype issue tracker ^{...or anywhere with @leycec, really!} if you need a bit of friendly help from a Canadian redneck. Runtime tensor type-hinting can be "fun" stuff. Until then, may the Python be with your super-cool AI healthcare project. Robots need hospitals, too. 🤖 -> 🏥