TiddlyWiki: load tiddlers on-demand

[ichorid]

TiddlyWiki is a microcontent platform. The basic building block in TW is a "tiddler". Everything is tiddler: wiki entries, plugins, code blocks, even pictures.

The problem

While TW supports loading tiddlers dynamically through the backend synchronization mechanism, it still tries to load all the tiddlers in one go. This becomes problematic for channels with millions of entries/tiddlers.

The solution

According to the architecture of Channels 3.0, we want to be able to load tiddlers on-demand from the backend, possibly in pre-caching fashion. For instance, when the user opens a tiddler with a list of torrent categories for a channel, the tiddlers linked from this page should be silently loaded by the backend. So, as the user proceeds through the categories/folders hierarchy, the corresponding tiddlers must be queried by TW from the backend.

Existing solutions

TiddlyWeb

Historically, a project called "TiddlyWeb" provided a synchronization backend for TiddlyWiki. It is mostly dead now, but it had a nice synchronization plugins infrastructure that got us started with TW integration. We can still steal some code/solutions from them as TiddlyWeb was written in Python.

"Skinny Tiddlers"

Before v.5, TW featured support for so-called "Skinny Tiddlers". The idea was to load empty tiddlers for everything initially, then lazy load real tiddlers from backend on-demand. According to TW author @Jermolene, the mechanism is slightly broken now, as there is no way to determine if the tiddler is really skinny, or just empty. Nonetheless, the code of associated plugins should provide a good starting point for implementing a proper on-demand loading scheme.

ListField

Tiddlers may have an attribute (a "field" in TW notation) called list field. This field is supposed to contain links to "related" entries. We can try using it to store the list of child nodes that must be lazy-loaded on opening the toddler.

The Task

The result of solving this issue must be a plugin for TW that will implement a lazy-loading procedure for marked tiddlers, and a sample NodeJS and/or FastAPI interface to test this plugin.

Useful info

While developing the Channels 3.0 prototype, I had a couple of very insightful conversations with TiddliWiki developers. Reading these is highly recommended to anyone trying to hack TW internals. These contain some revelations that are never mentioned by in TW documentation, but are true staples of TW design (such as the fact that Tiddler objects are immutable):

• Custom saver/loader plugins: how to start? TiddlyWiki/TiddlyWiki5#5269
• How to assign unique IDs to tiddlers? TiddlyWiki/TiddlyWiki5#5304