Move Metadata to --internal database

[asg017]

Work-in-progress, not ready for review. refs #2341

• Removes .metadata() and ._metadata() fields from the Datasette class
• Removes /-/metadata.json endpoint
• Removes metadata "fallback"
• Removes the get_metadata() hook
• Adds the following methods to Datasette:
  • get_instance_metadata()
  • get_database_metadata()
  • get_resource_metadata()
  • get_column_metadata()
  • set_instance_metadata()
  • set_database_metadata()
  • set_resource_metadata()
  • set_column_metadata()
• Adds the following tables to internal.db
  • datasette_metadata_instance_entries
  • datasette_metadata_database_entries
  • datasette_metadata_resource_entries
  • datasette_metadata_column_entries

TODO

• Support metadata.json
• Don't change facet logic, move facet_size to synchronous config
• Docs!

---

📚 Documentation preview 📚: https://datasette--2343.org.readthedocs.build/en/2343/

[simonw]

This inspired me to survey people to find out what SQLite versions are out there in the wild:

https://twitter.com/simonw/status/1800198142002115029
https://fedi.simonwillison.net/@simon/112593180340389914

[simonw]

This logic should live in datasette/utils/internal_db.py since that's where the other logic that creates tables and populates them lives.

[simonw]

I'm so happy to be rid of the synchronous (but undocumented) .metadata() method.

[simonw]

Ran Datasette like this (with the datasette-visible-internal-db plugin installed):

datasette -m metadata.json fixtures.db --internal _internal.db  -p 8003 --root

Here are the tables in _internal.db (output using SQLite dump):

CREATE TABLE datasette_metadata_column_entries(
                database_name text,
                resource_name text,
                column_name text,
                key text,
                value text,
                unique(database_name, resource_name, column_name, key)
              );
INSERT INTO "datasette_metadata_column_entries" VALUES('fixtures','roadside_attractions','name','description','The name of the attraction');
INSERT INTO "datasette_metadata_column_entries" VALUES('fixtures','roadside_attractions','address','description','The street address for the attraction');
CREATE TABLE datasette_metadata_database_entries(
                database_name text,
                key text,
                value text,
                unique(database_name, key)
              );
INSERT INTO "datasette_metadata_database_entries" VALUES('fixtures','description','Test tables description');
CREATE TABLE datasette_metadata_instance_entries(
                key text,
                value text,
                unique(key)
              );
INSERT INTO "datasette_metadata_instance_entries" VALUES('title','Datasette Fixtures');
INSERT INTO "datasette_metadata_instance_entries" VALUES('description_html','An example SQLite database demonstrating Datasette. <a href="/login-as-root">Sign in as root user</a>');
INSERT INTO "datasette_metadata_instance_entries" VALUES('license','Apache License 2.0');
INSERT INTO "datasette_metadata_instance_entries" VALUES('license_url','https://github.com/simonw/datasette/blob/main/LICENSE');
INSERT INTO "datasette_metadata_instance_entries" VALUES('source','tests/fixtures.py');
INSERT INTO "datasette_metadata_instance_entries" VALUES('source_url','https://github.com/simonw/datasette/blob/main/tests/fixtures.py');
INSERT INTO "datasette_metadata_instance_entries" VALUES('about','About Datasette');
INSERT INTO "datasette_metadata_instance_entries" VALUES('about_url','https://github.com/simonw/datasette');
CREATE TABLE datasette_metadata_resource_entries(
                database_name text,
                resource_name text,
                key text,
                value text,
                unique(database_name, resource_name, key)
              );
INSERT INTO "datasette_metadata_resource_entries" VALUES('fixtures','simple_primary_key','description_html','Simple <em>primary</em> key');
INSERT INTO "datasette_metadata_resource_entries" VALUES('fixtures','simple_primary_key','title','This <em>HTML</em> is escaped');

[simonw]

SELECT 
    ct.database_name,
    ct.table_name,
    ct.rootpage,
    ct.sql,
    COALESCE(
        json_group_object(dmre.key, dmre.value) 
        FILTER (WHERE dmre.key IS NOT NULL AND dmre.value IS NOT NULL),
        '{}'
    ) AS metadata
FROM 
    catalog_tables ct
LEFT JOIN 
    datasette_metadata_resource_entries dmre
ON 
    ct.database_name = dmre.database_name 
    AND ct.table_name = dmre.resource_name
GROUP BY 
    ct.database_name,
    ct.table_name
ORDER BY 
    ct.database_name,
    ct.table_name;

(Wrote with help of GPT-4o)

Returns results like this:

But FILTER in aggregates was added in 3.30.0 on 2019-10-04: https://sqlite.org/changes.html#version_3_30_0

[asg017]

Those pyodide errors look not too fun

[simonw]

Maybe we should rename self._metadata_local to self._metadata_on_startup or self._metadata_from_file to help clarify that it's the metadata that was passed to Datasette's constructor on startup?

I don't want people to get confused and think that reading from self._metadata_local is a useful thing to do, when they should be calling the new async methods instead.

Probably overthinking that though, since those methods will be documented but self._metadata_local will not.

[simonw]

I'm OK with us not doing this - software telling me to delete a file is a bit weird.