Dynamic Configuration/Metadata - A proposed plugin hook & implementation

[brandonrobertz]

There have been several issues related to the ability to change certain things inside Datasette without rebooting or re-deploying the app. NextLI is trying to open Datasette to non-technical users (people who aren't CLI comfortable) and that has required us to be able to do things like live-import new datasets and change metadata (also known as "configuration") on-the-fly. In response to this need, I've developed a fork of Datasette that adds a new plugin hook around configuration:

get_metadata(
    datasette=self, key=key, database=database, table=table,
    fallback=fallback
)

This plugs into Datasette.metadata() and Dataette._metadata where the rest of the application (plugins included) pulls configuration from. This plugin hook merges anything returned from plugins with any local configuration provided (e.g., via metadata.yml). In the current implementation, the on-disk configuration can't be overridden by plugins as a security feature.

• The current implementation of this plugin can be found here: https://github.com/next-LI/datasette
• The current implementation of the corresponding "live configuration" plugin that uses this hook is here: https://github.com/next-LI/datasette-live-config/

Plugin allows us to do the following:

• Dynamically set metadata and other configuration -- no server reboots required for most trivial changes to Datasette.
• Bundle sqlite databases with their own metadata (hidden table _datasette_metadata or something).
• Allows users to DIY if they have different constraints

Related issues this would solve (non-exhaustive list):

• Ability to bundle metadata and templates inside the SQLite file #188 "Ability to bundle metadata and templates inside the SQLite file"
• updating metadata.json without recreating the app #639 "updating metadata.json without recreating the app"
• Plugin hook for instance/database/table metadata #860 "Plugin hook for instance/database/table metadata"
• Mechanism for storing metadata in _metadata tables #1168 "Mechanism for storing metadata in _metadata tables"
• Maintain an in-memory SQLite table of connected databases and their tables #1150 " Maintain an in-memory SQLite table of connected databases and their tables"

Open questions:

• async or sync? The current implementation I've been testing is synchronous. This is the easiest path because is requires the smallest amount of code to change. Switching it to async would allow for more complex configuration strategies, but would force more code to have their function signatures change to async (since async infects all sync code). I've benchmarked the plugin hook with the datasette-live-config plugin and even with a non-optimized hook implementation that opens the DB connection every time, it adds <30ms.

• What do we do with the non-defined areas of the metadata? I have JSON schema of metadata.yaml, but plugins, allow and allow_sql are undefined. In the UI I have for editing config, I leave these parts as JSON text for manual editing.

• Simon has been talking about re-naming metadata and/or splitting metadata up. Should this plugin be named get_configuration to be future safe? Should the other things change before this plugin gets merged?

[simonw]

Having thought about this for a while I'd love to review a pull request of what you have so far.

[brandonrobertz]

Very much a WIP, but here we go: #1368

I'm currently debating whether Datasette should be responsible for merging the returned metadatas, or if the plugin hook should be a "update_metadata" where the plugin modifies the metadata dict. Thoughts on this would be appreciated!