Skip to content

database integration

Jump to bottom
Daniel Higginbotham edited this page Nov 2, 2022 · 9 revisions

Datapotato is written in a database-agnostic way so that you can use it with your standard relational dbs, or something more esoteric like datomic or xtdb, or anything you want.

It comes with helper libraries that should handle the majority of use cases if you're using one of the following:

This doc covers using the helper libraries, as well as writing your own integration. It also shows how you can specify how to insert records per-ent-type and per-record. It assumes you've gone through the Getting Started doc.

This doc also assumes you've added the matcher combinators library to your dependencies. This isn't necessary for working with datapotato, but it's generally useful for testing random data.

next-jdbc

Example namespace showing how you can use the next-jdbc helper in tests

Add a dependency

To use the next-jdbc helper, first add the library party.donut/datapotato-next-jdbc to your dependencies. Its version number should match datapotato's version number.

Require namespaces

Next, you'll want to include the following namespaces in your test namespace:

[donut.datapotato.core :as dc]
[donut.datapotato.next-jdbc :as dnj]
[matcher-combinators.test] ;; optional
[next.jdbc :as jdbc]
[next.jdbc.sql :as sql]

Define potato-db

Then define a potato-db:

(def potato-db
  {:schema   potato-schema ;; <1>
   :generate {:generator mg/generate} ;; <2>
   :fixtures (merge dnj/config  ;; <3>
                    {:dbspec {:dbtype         "sqlite"
                              :connection-uri "jdbc:sqlite::memory:"} ;; <4>
                     :setup  (fn [_]
                               (create-tables dc/*connection*))})}) ;; <5>

  1. Your potato-schema should be defined. See Getting Started for an explanation of potato-schemas.

  2. See Getting Started for data generation config details

  3. Here we're constructing the configuration for :fixtures by merging dnj/config with some project-specific values. dnj/config includes the keys :insert, :get-connection, and :close-connection.

    • :insert is a function used to actually insert a generated record in a database. See Writing your own integration below for more details.

    • :get-connection is a function that takes the potato-db as its argument and that should return a JDBC connection. The function provided in dnj/config is:

      (fn next-jdbc-get-connection [potato-db]
  (jdbc/get-connection (get-in potato-db [:fixtures :dbspec])))

      The with-fixtures macro binds the dc/*connection* dynamic var to the value returned by :get-connection.

    • If :close-connection is set, it gets called after everything else in the body of dc/with-fixtures

  4. The :fixtures map should include a :dbspec key. this is passed directly to next.jdbc/get-connection. Check out next-jdbc's Datasources and Connections docs for more info on the keys that :dbspec can take.

  5. The :setup function is called when you use the dc/with-fixtures macro before anything in the macro body is evaluated. This is a good place to empty your database or do whatever tidying you need for your tests.

OPTIONAL define a get-inserted method

NOTE: If you use a popular database like postgres, mysql, or sqlite, this section probably isn't relevant to you. If things aren't working as expected, this is a good place to start.

Datapotato and its database-specific helper libraries have to handle a very specific use case: setting foreign key values based on the data that actually exists in the database. Datapotato must handle this scenario because sometimes the database is responsible for generating values, like when it autogenerates primary keys.

For example, if you're inserting a DreamJournal that references a User via :created-by-id, the DreamJournal record's :created-by-id field must be set to the User's actual :id in the database. If you don't do this, then you'll likely violate a foreign key constraint and encounter an exception.

To deal with this, the next-jdbc helper library defines the get-inserted multimethod, which is used to retrieve a record after it has been inserted in the database. The retrieved record is then used to set foreign keys.

get-inserted is a multimethod because the method for retrieving a record after insertion varies across database vendors (postgres, sqlite, etc).

The default method for get-inserted will return the return value from next.jdbc.sql/insert!. For some databases, like postgres, this is the full record that exists in the database.

Consult the source to see how get-inserted is dispatched and what arguments it takes.

Write tests

Now you're ready to use datapotato in tests:

(deftest inserts-simple-generated-data
  (dc/with-fixtures potato-db
    (dc/insert-fixtures {:user [{:count 2}]})
    (is (match? [#:users{:id 1 :username string?}
                 #:users{:id 2 :username string?}]
                (sql/query dc/*connection* ["SELECT * FROM users"])))))

Now you're cooking with potatoes!

datomic

To use the datomic helper, first add the library party.donut/datapotato-datomic to your dependencies. Its version number should match datapotato's version number.

Require namespaces

Next, you'll want to include the following namespaces in your test namespace:

[datomic.api :as d]
[donut.datapotato.core :as dc]
[donut.datapotato.datomic :as dd]
[matcher-combinators.test] ;; optional

Define potato-db

Then define a potato-db:

(def uri "datomic:mem://datapotato-test") ;; <1>
(def potato-db
  {:schema   schema ;; <2>
   :generate {:generator mg/generate} ;; <3>
   :fixtures {:insert
              dd/insert ;; <4>

              :get-connection ;; <5>
              (fn get-connection [_]
                (d/delete-database uri)
                (d/create-database uri)
                (d/connect uri))

              :setup ;; <6>
              (fn setup [{:keys [fixtures]}]
                ;; set up database
                @(d/transact
                  dc/*connection*
                  []))}})
  1. In this example we connect to an in-memory datomic database
  2. Your potato-schema should be defined. See Getting Started for an explanation of potato-schemas.
  3. See Getting Started for data generation config details
  4. We call the helper donut.datapotato.datomic/insert to insert records
  5. :get-connection is a function that takes the potato-db as its argument and that should return a datomic connection. The with-fixtures macro binds the dc/*connection* dynamic var to the value returned by :get-connection.
  6. You can write a :setup function to set up your datomic db.

Write tests

Now you're ready to use datapotato in tests:

(defn q
  "lil helper helpin out"
  [connection query]
  (->> (d/q query (d/db connection))
       (map first)
       (sort-by :db/id)))

(deftest inserts-simple-generated-data
  (dc/with-fixtures potato-db
    (dc/insert-fixtures {:user [{:count 2}]})
    (is (match? [{:db/id 17592186045418 :user/username string?}
                 {:db/id 17592186045420 :user/username string?}]
                (q dc/*connection*
                   '{:find  [(pull ?u [*])]
                     :where [[?u :user/username]]})))))

xtdb

To use the xtdb helper, first add the library party.donut/datapotato-xtdb to your dependencies. Its version number should match datapotato's version number.

Require namespaces

Next, you'll want to include the following namespaces in your test namespace:

[donut.datapotato.core :as dc]
[donut.datapotato.xtdb :as dxtdb]
[matcher-combinators.test] ;; optional
[xtdb.api :as xt]

Define potato-db

Then define a potato-db:

(def node-atom (atom nil)) <1>

(def potato-db
  {:schema   potato-schema ;; <2>
   :generate {:generator mg/generate} ;; <3>
   :fixtures {:insert         dxtdb/insert <4>
              :get-connection (fn get-connection [_]
                                (when-let [node @node-atom]
                                  (.close node))
                                (reset! node-atom (xt/start-node {}))
                                @node-atom)}}) ;; <5>
  1. This atom stores the connection to an xtdb node
  2. Your potato-schema should be defined. See Getting Started for an explanation of potato-schemas.
  3. See Getting Started for data generation config details
  4. We call the helper donut.datapotato.xtdb/insert to insert records
  5. :get-connection is a function that takes the potato-db as its argument and that should return an XTDB node. The with-fixtures macro binds the dc/*connection* dynamic var to the value returned by :get-connection. Here, the "connection" is an XTDB in-memory node.

Write tests

Now you're ready to use datapotato in tests:

(defn q
  "query helper for tests"
  [query]
  (->> (xt/q (xt/db dc/*connection*) query)
       (map first)
       (sort-by :xt/id)))

(deftest inserts-simple-generated-data
  (dc/with-fixtures potato-db
    (dc/insert-fixtures {:user [{:count 2}]})
    (is (match? [{:xt/id 1 :user/username string?}
                 {:xt/id 2 :user/username string?}]
                (q '{:find  [(pull ?u [*])]
                     :where [[?u :user/username]]})))))

Now you're cooking with potatoes!

Writing your own integration

Writing your own integration means configuring your potato-db so that datapotato.core/with-fixtures and datapotato.core/insert-fixtures will work.

The with-fixtures macro is responsible for:

  • Calling the function under [:fixtures :get-connection] to get a connection and binding the connection to datapotato.core/*connection*
  • Calling the function under [:fixtures :setup]

insert-fixtures relies on a function defined at [:fixtures :insert], calling it for every record that needs to get inserted. It takes two arguments:

To do that, you need to add a map under the :fixtures key of your potato-db with the following keys:

  • potato-db - the first argument is the potato-db itself. You can use this to look up values under :fixtures, including a :connection key gets populated by the with-fixtures macro.
  • visit-data - this is a map with the following keys:
    • :visit-val - this is the data to be inserted
    • :query-opts - explained more below
    • :visit-visit-query-opts - explained more below

For reference, here's how the datomic insert function is defined:

(defn insert
  [{{:keys [connection]} dc/fixtures-visit-key :as _potato-db}
   {:keys [visit-val] :as _visit-data}]
  (let [result @(d/transact connection [visit-val])
        dbid (first (vals (:tempids result)))]
      (assoc (d/pull (d/db connection) '[*] dbid)
           :db/id dbid)))

:query-opts and :visit-query-opts

:query-opts and :visit-query-opts allow you refine how your data is inserted and generated. They refer to the values passed in to your datapotato query:

(dc/insert {:users {:fixtures {:option-a "whatever"}
                    :other-opts {:option-b "bloop"}}})

For the query above, :query-opts would be the map

{:fixtures {:option-a "whatever"}
            :other-opts {:option-b "bloop"}}

and :visit-query-opts would be the map

{:option-a "whatever"}

Configuring insertion

If for some reason you need to use different databases per entity type or even per entity, you can override the potato-db-wide configuration in your potato-schema and even in your queries. Here's what that could look like:

(def potato-db
  {:schema   {:user          {:prefix   :u
                              :generate {:schema User}
                              :fixtures {:insert custom-insert}}
              :dream-journal {:prefix   :dj
                              :generate {:schema DreamJournal}}
              :entry         {:prefix   :e
                              :generate {:schema Entry}}}
   :fixtures {:insert donut.datapotato.datomic/insert}})


(dc/with-fixtures
  (dc/insert {:entry [{:fixtures {:insert custom-dream-journal-insert}}]}))

In this example, a custom insert is defined for users at the potato-schema level, and a custom insert function is defined for entries at the query level.

Clone this wiki locally