metabase-datomic

A Metabase driver for Datomic.

Commercial support is provided by Gaiwan.

Try it!

docker run -p 3000:3000 lambdaisland/metabase-datomic

Design decisions

See the Architecture Decision Log

Developing

To get a REPL based workflow, do a git clone of both metabase and metabase-datomic, such that they are in sibling directories

$ git clone [email protected]:metabase/metabase.git
$ git clone [email protected]:plexus/metabase-datomic.git

$ tree -L 1
.
│
├── metabase
└── metabase-datomic

Before you can use Metabase you need to build the frontend. This step you only need to do once.

cd metabase
yarn build

And install metabase locally

lein install

Now cd into the metabase-datomic directory, and run bin/start_metabase to lauch the process including nREPL running on port 4444.

cd metabase-datomic
bin/start_metabase

Now you can connect from Emacs/CIDER to port 4444, or use the bin/cider_connect script to automatically connect, and to associate the REPL session with both projects, so you can easily evaluate code in either, and navigate back and forth.

Once you have a REPL you can start the web app. This also opens a browser at localhost:3000

user=> (go)

The first time it will ask you to create a user and do some other initial setup. To skip this step, invoke setup!. This will create a user with username [email protected] and password dev. It will also create a Datomic database with URL datomic:free://localhost:4334/mbrainz. You are encouraged to run a datomic-free transactor, and import the MusicBrainz database for testing.

user=> (setup!)

Installing

The general process is to build an uberjar, and copy the result into your Metabase plugins/ directory. You can build a jar based on datomic-free, or datomic-pro (assuming you have a license). Metabase must be available as a local JAR.

cd metabase
lein install
mkdir plugins
cd ../metabase-datomic
lein with-profiles +datomic-free uberjar
# lein with-profiles +datomic-pro uberjar
cp target/uberjar/datomic.metabase-driver.jar ../metabase/plugins

Now you can start Metabase, and start adding Datomic databases

cd ../metabase
lein run -m metabase.core

Configuration EDN

When you configure a Datomic Database in Metabase you will notice a config field called "Configuration EDN". Here you can paste a snippet of EDN which will influence some of the Driver's behavior.

The EDN needs to represent a Clojure map. These keys are currently understood

• :inclusion-clauses
• :tx-filter
• :relationships

Other keys are ignored.

:inclusion-clauses

Datomic does not have tables, but nevertheless the driver will map your data to Metabase tables based on the attribute names in your schema. To limit results to the right entities it needs to do a check to see if a certain entity logically belongs to such a table.

By default these look like this

[(or [?eid :user/name]
     [?eid :user/password]
     [?eid :user/roles])]

In other words we look for entities that have any attribute starting with the given prefix. This can be both suboptimal (there might be a single attribute with an index that is faster to check), and it may be wrong, depending on your setup.

So we allow configuring this clause per table. The configured value should be a vector of datomic clauses. You have the full power of datalog available. Use the special symbol ?eid for the entity that is being filtered.

{:inclusion-clauses {"user" [[?eid :user/handle]]}}

:tx-filter

The datomic.api/filter function allows you to get a filtered view of the database. A common use case is to select datoms based on metadata added to transaction entities.

You can set :tx-filter to any form that evaluates to a Clojure function. Make sure any namespaces like datomic.api are fully qualified.

{:tx-filter
 (fn [db ^datomic.Datom datom]
   (let [tx-user (get-in (datomic.api/entity db (.tx datom)) [:tx/user :db/id])]
     (or (nil? tx-tenant) (= 17592186046521 tx-user))))}

:rules

This allows you to configure Datomic rules. These then become available in the native query editor, as well in :inclusion-clauses and :relationships.

{:rules
 [[(sub-accounts ?p ?c)
   [?p :account/children ?c]]
  [(sub-accounts ?p ?d)
   [?p :account/children ?c]
   (sub-accounts ?c ?d)]]}

:relationships

This features allows you to add "synthetic foreign keys" to tables. These are fields that Metabase will consider to be foreign keys, but in reality they are backed by an arbitrary lookup path in Datomic. This can include reverse reference (:foo/_bar) and rules.

To set up an extra relationship you start from the table where you want to add the relationship, then give it a name, give the path of attributes and rules needed to get to the other entity, and specifiy which table the resulting entity belongs to.

{:relationships
 {;; foreign keys added to the account table
  :account
  {:journal-entry-lines
   {:path [:journal-entry-line/_account]
    :target :journal-entry-line}

   :subaccounts
   {:path [sub-accounts]
    :target :account}

   :parent-accounts
   {:path [_sub-accounts] ;; apply a rule in reverse
    :target :account}}

  ;; foreign keys added to the journal-entry-line table
  :journal-entry-line
  {:fiscal-year
   {:path [:journal-entry/_journal-entry-lines
           :ledger/_journal-entries
           :fiscal-year/_ledgers]
    :target :fiscal-year}}}}

Status

Feature	Supported?
Basics
{:source-table integer-literal}	Yes
{:fields [& field]}	Yes
[:field-id field-id]	Yes
[:datetime-field local-field | fk unit]	Yes
{:breakout [& concrete-field]}	Yes
[:field-id field-id]	Yes
[:aggregation 0]	Yes
[:datetime-field local-field | fk unit]	Yes
{:filter filter-clause}
[:and & filter-clause]	Yes
[:or & filter-clause]	Yes
[:not filter-clause]	Yes
[:= concrete-field value & value]	Yes
[:!= concrete-field value & value]	Yes
[:< concrete-field orderable-value]	Yes
[:> concrete-field orderable-value]	Yes
[:<= concrete-field orderable-value]	Yes
[:>= concrete-field orderable-value]	Yes
[:is-null concrete-field]	Yes
[:not-null concrete-field]	Yes
[:between concrete-field min orderable-value max orderable-value]	Yes
[:inside lat concrete-field lon concrete-field lat-max numeric-literal lon-min numeric-literal lat-min numeric-literal lon-max numeric-literal]	Yes
[:starts-with concrete-field string-literal]	Yes
[:contains concrete-field string-literal]	Yes
[:does-not-contain concrete-field string-literal]	Yes
[:ends-with concrete-field string-literal]	Yes
[:time-interval field concrete-field n :current|:last|:next|integer-literal unit relative-datetime-unit]
{:limit integer-literal}	Yes
{:order-by [& order-by-clause]}	Yes
:basic-aggregations
{:aggregation aggregation-clause}
[:count]	Yes
[:count concrete-field]	Yes
[:cum-count concrete-field]	Yes
[:cum-sum concrete-field]	Yes
[:distinct concrete-field]	Yes
[:sum concrete-field]	Yes
[:min concrete-field]	Yes
[:max concrete-field]	Yes
[:share filter-clause]
:standard-deviation-aggregations	Yes
{:aggregation aggregation-clause}	Yes
[:stddev concrete-field]	Yes
:foreign-keys	Yes
{:fields [& field]}	Yes
[:fk-> fk-field-id dest-field-id]	Yes
:nested-fields
:set-timezone
:expressions
{:fields [& field]}
[:expression]
{:breakout [& concrete-field]}
[:expression]
{:expressions {expression-name expression}}
:native-parameters
:expression-aggregations
:nested-queries	Yes
{:source-query query}	Yes
:binning
:case-sensitivity-string-filter-options	Yes

License

Copyright © 2019 Arne Brasseur

Licensed under the term of the Mozilla Public License 2.0, see LICENSE.