Skip to content

antimer/lmdb

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

87 Commits
.github/workflows
.github/workflows
 
 
src
src
 
 
test
test
 
 
.gitignore
.gitignore
 
 
COPYING
COPYING
 
 
README
README
 
 
README.md
README.md
 
 
lmdb.asd
lmdb.asd
 
 

Repository files navigation

LMDB Manual

Table of Contents

[in package LMDB]

1 LMDB ASDF System

2 Links

Here is the official repository and the HTML documentation for the latest version.

3 Introduction

LMDB, the Lightning Memory-mapped Database, is an ACID key-value database with MVCC. It is a small C library ("C lmdb" from now on), around which LMDB is a Common Lisp wrapper. LMDB covers most of C lmdb's functionality, has a simplified API, much needed Safety checks, and comprehensive documentation.

Compared to other key-value stores, LMDB's distuingishing features are:

  • Transactions span multiple keys.

  • Embedded. It has no server but can be used concurrently not only by multiple threads but by multiple OS processes, too.

  • Extremely high read performance: millions of transactions per second.

  • Very low maintenance.

Other notable things:

  • With its default - the most durable - settings, it has average write performance, which is bottlenecked by fsync().

  • Readers don't block readers or writers, but there is at most one writer at a time.

  • Extremely simple, crash-proof design.

  • The entire database (called environment) is backed by a single memory-mapped file, with a copy-on-write B+ tree.

  • No transaction log.

  • It is very much like Berkeley DB done right, without the fluff and much improved administration.

Do read the Caveats, though. On the Lisp side, this library will not work with virtual threads because LMDB's write locking is tied to native threads.

Using LMDB is easy:

(with-temporary-env (*env*)
  (let ((db (get-db "test")))
    (with-txn (:write t)
      (put db "k1" #(2 3))
      (print (g3t db "k1")) ; => #(2 3)
      (del db "k1"))))

More typically, the environment and databases are opened once so that multiple threads and transactions can access them:

(defvar *test-db*)

(unless *env*
  (setq *env* (open-env "/tmp/lmdb-test-env/" :if-does-not-exist :create))
  (setq *test-db* (get-db "test" :value-encoding :utf-8)))

(with-txn (:write t)
  (put *test-db* 1 "hello")
  (print (g3t *test-db* 1)) ; => "hello"
  (del *test-db* 1))

Note how :VALUE-ENCODING sneaked in above. This was so to make G3T return a string instead of an octet vector.

LMDB treats keys and values as opaque byte arrays to be hung on a B+ tree, and only requires a comparison function to be defined over keys. LMDB knows how to serialize the types (UNSIGNED-BYTE 64) and STRING (which are often used as keys so sorting must work as expected). Serialization of the rest of the datatypes is left to the client. See Encoding and decoding data for more.

4 Design and implementation

4.1 Safety

The lmdb C API trusts client code to respect its rules. Being C, managing object lifetimes is the biggest burden. There are also rules that are documented, but not enforced. This Lisp wrapper tries to enforce these rules itself and manage object lifetimes in a safe way to avoid data corruption. How and what it does is described in the following.

Environments

  • OPEN-ENV checks that the same path is not used in multiple open environments to prevent locking issues documented in Caveats.

  • CLOSE-ENV waits until all active transactions are finished before actually closing the environment. Alternatively, if OPEN-ENV was called with :SYNCHRONIZED NIL, to avoid the overhead of synchronization, the environment is closed only when garbage collected.

Transactions

  • Checks are made to detect illegal operations on parent transactions (see LMDB-ILLEGAL-ACCESS-TO-PARENT-TXN-ERROR).

  • Access to closed transactions is reliably detected.

  • C LMDB allows read transactions to be used in multiple threads. The synchronization cost of performing this safely (i.e. without risking access to closed and freed transaction objects) is significant so this is not supported.

Databases

  • mdb_dbi_open() is wrapped by GET-DB in a transaction and is protected by a mutex to comply with C lmdb's requirements:

       A transaction that opens a database must finish (either
   commit or abort) before another transaction may open it.
   Multiple concurrent transactions cannot open the same
   database.

  • mdb_dbi_close() is too dangerous to be exposed as explained in the GET-DB documentation.

  • For similar reasons, DROP-DB is wrapped in WITH-ENV.

  • mdb_env_set_mapsize(), mdb_env_set_max_readers(), and mdb_env_set_maxdbs() are only available through OPEN-ENV because they either require that there are no write transactions or do not work on open environments.

Cursors
  • As even read transactions are restricted to a single thread, so are cursors. Using a cursor from a thread other than the one in which it was created (i.e. the thread of its transaction) raises LMDB-CURSOR-THREAD-ERROR. In return for this restriction, access to cursors belonging to closed transactions is reliably detected.
Signal handling

The C lmdb library handles system calls being interrupted (EINTR and EAGAIN), but unwinding the stack from interrupts in the middle of LMDB calls can leave the in-memory data structures such as transactions inconsistent. If this happens, their further use risks data corruption. For this reason, calls to LMDB are performed with interrupts disabled. For SBCL, this means SB-SYS:WITHOUT-INTERRUPTS. It is an error when compiling LMDB if an equivalent facility is not found in the Lisp implementation. A warning is signalled if no substitute is found for SB-SYS:WITH-INTERRUPTS because this makes the body of WITH-ENV, WITH-TXN, WITH-CURSOR and similar uninterruptible.

Operations that do not modify the database (G3T, CURSOR-FIRST, CURSOR-VALUE, etc) are async unwind safe, and for performance they are called without the above provisions.

Note that the library is not reentrant, so don't call LMDB from signal handlers.

4.2 Deviations from the C lmdb API

The following are the most prominent deviations and omissions from the C lmdb API in addition to those listed in Safety.

Environments
Transactions
  • Read-only WITH-TXNs are turned into noops when "nested" (unless IGNORE-PARENT).
Databases

  • mdb_set_compare() and mdb_set_dupsort() are not exposed. If they are needed, implement a foreign comparison function and call LIBLMDB:SET-COMPARE or LIBLMDB:SET-DUPSORT directly or perhaps change the encoding of the data.

  • Working with multiple contiguous values with DUPFIXED is not yet implemented. This functionality would belong in PUT, CURSOR-PUT, CURSOR-NEXT and CURSOR-VALUE.

  • PUT, CURSOR-PUT do not support the RESERVE flag.

5 Library versions

  • [function] LMDB-FOREIGN-VERSION

    Return the version of the C lmdb library as a string like 0.9.26.

    Wraps mdb_version().

  • [function] LMDB-BINDING-VERSION

    Return a string representing the version of C lmdb based on which the CFFI bindings were created. The version string has the same format as LMDB-FOREIGN-VERSION.

6 Environments

An environment (class ENV) is basically a single memory-mapped file holding all the data, plus some flags determining how we interact it. An environment can have multiple databases (class DB), each of which is a B+ tree within the same file. An environment is like a database in a relational db, and the databases in it are like tables and indices. The terminology comes from Berkeley DB.

6.1 Environments reference

  • [class] ENV

    An environment object through which a memory-mapped data file can be accessed. Always to be created by OPEN-ENV.

  • [reader] ENV-PATH ENV (:PATH)

    The location of the memory-mapped file and the environment lock file.

  • [reader] ENV-MAX-DBS ENV (:MAX-DBS)

    The maximum number of named databases in the environment. Currently a moderate number is cheap, but a huge number gets expensive: 7-120 words per transaction, and every GET-DB does a linear search of the opened database.

  • [reader] ENV-MAX-READERS ENV (:MAX-READERS)

    The maximum number of threads/reader slots. See the documentation of the reader lock table for more.

  • [reader] ENV-MAP-SIZE ENV (:MAP-SIZE)

    Specifies the size of the data file in bytes.

  • [reader] ENV-MODE ENV (:MODE)

  • [reader] ENV-FLAGS ENV (:FLAGS)

    A plist of the options as captured by OPEN-ENV. For example, (:FIXED-MAP NIL :SUBDIR T ...).

6.2 Opening and closing environments

  • [variable] *ENV-CLASS* ENV

    The default class OPEN-ENV instaniates. Must be a subclass of ENV. This provides a way to associate application specific data with ENV objects.

  • [function] OPEN-ENV PATH &KEY (CLASS *ENV-CLASS*) (IF-DOES-NOT-EXIST :ERROR) (SYNCHRONIZED T) (MAX-DBS 1) (MAX-READERS 126) (MAP-SIZE (* 1024 1024)) (MODE 436) (SUBDIR T) (SYNC T) (META-SYNC T) READ-ONLY (TLS T) (READ-AHEAD T) (LOCK T) (MEM-INIT T) FIXED-MAP WRITE-MAP MAP-ASYNC

    Create an ENV object through which the LMDB environment can be accessed and open it. To prevent corruption, an error is signalled if the same data file is opened multiple times. However, the checks performed do not work on remote filesystems (see ENV-PATH).

    LMDB-ERROR is signalled if opening the environment fails for any other reason.

    Unless explicitly noted, none of the arguments persist (i.e. they are not saved in the data file).

    PATH is the filesystem location of the environment files (see SUBDIR below for more). Do not use LMDB data files on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and certainly sync between programs on different hosts.

    IF-DOES-NOT-EXIST determines what happens if ENV-PATH does not exists:

    • :ERROR: An error is signalled.

    • :CREATE: A new memory-mapped file is created ensuring that all containing directories exist.

    • NIL: Return NIL without doing anything.

    See CLOSE-ENV for the description of SYNCHRONIZED.

    • MAX-DBS: The maximum number of named databases in the environment. Currently a moderate number is cheap, but a huge number gets expensive: 7-120 words per transaction, and every GET-DB does a linear search of the opened database.

    • MAP-SIZE: Specifies the size of the data file in bytes. The new size takes effect immediately for the current process, but will not be persisted to any others until a write transaction has been committed by the current process. Also, only map size increases are persisted into the environment. If the map size is increased by another process, and data has grown beyond the range of the current mapsize, starting a new transaction (see WITH-TXN) will signal LMDB-MAP-RESIZED-ERROR. If zero is specified for MAP-SIZE, then the persisted size is used from the data file. Also see LMDB-MAP-FULL-ERROR.

    • MODE: Unix file mode for files created. The default is #o664. Has no effect when opening an existing environment.

    The rest of the arguments correspond to LMDB environment flags and are available in the plist ENV-FLAGS.

    • SUBDIR: If SUBDIR, then the path is a directory which holds the data.mdb and the lock.mdb files. If SUBDIR is NIL, the path is the filename of the data file and the lock file has the same name plus a -lock suffix.

    • SYNC: If NIL, don't fsync after commit. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often SYNC-ENV is called. However, if the filesystem preserves write order (very few do) and the WRITE-MAP (currently unsupported) flag is not used, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). I.e. database integrity is maintained, but a system crash may undo the final transactions.

    • META-SYNC: If NIL, flush system buffers to disk only once per transaction, but omit the metadata flush. Defer that until the system flushes files to disk, the next commit of a non-read-only transaction or SYNC-ENV. This optimization maintains database integrity, but a system crash may undo the last committed transaction. I.e. it preserves the ACI (atomicity, consistency, isolation) but not D (durability) database property.

    • READ-ONLY: Map the data file in read-only mode. It is an error to try to modify anything in it.

    • TLS: Setting it to NIL allows each OS thread to have multiple read-only transactions (see WITH-TXN's IGNORE-PARENT argument). It also allows and transactions not to be tied to a single thread, but that's quite dangerous, see Safety.

    • READ-AHEAD: Turn off readahead as in madvise(MADV_RANDOM). Most operating systems perform read-ahead on read requests by default. This option turns it off if the OS supports it. Turning it off may help random read performance when the DB is larger than RAM and system RAM is full. This option is not implemented on Windows.

    • LOCK: Data corruption lurks here. If NIL, don't do any locking. If concurrent access is anticipated, the caller must manage all concurrency itself. For proper operation the caller must enforce single-writer semantics, and must ensure that no readers are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that no readers may be active at all when a writer begins.

    • MEM-INIT: If NIL, don't initialize malloced memory before writing to unused spaces in the data file. By default, memory for pages written to the data file is obtained using malloc. While these pages may be reused in subsequent transactions, freshly malloced pages will be initialized to zeroes before use. This avoids persisting leftover data from other code (that used the heap and subsequently freed the memory) into the data file. Note that many other system libraries may allocate and free memory from the heap for arbitrary uses. E.g., stdio may use the heap for file I/O buffers. This initialization step has a modest performance cost so some applications may want to disable it using this flag. This option can be a problem for applications which handle sensitive data like passwords, and it makes memory checkers like Valgrind noisy. This flag is not needed with WRITE-MAP, which writes directly to the mmap instead of using malloc for pages.

    • FIXED-MAP (experimental): This flag must be specified when creating the environment and is stored persistently in the data file. If successful, the memory map will always reside at the same virtual address and pointers used to reference data items in the database will be constant across multiple invocations. This option may not always work, depending on how the operating system has allocated memory to shared libraries and other uses.

    Unsupported flags (an error is signalled if they are changed from their default values):

    • WRITE-MAP: Use a writable memory map unless READ-ONLY is set. This is faster and uses fewer mallocs, but loses protection from application bugs like wild pointer writes and other bad updates into the database. Incompatible with nested transactions. This may be slightly faster for DBs that fit entirely in RAM, but is slower for DBs larger than RAM. Do not mix processes with and without WRITE-MAP on the same environment. This can defeat durability (SYNC-ENV, etc).

    • MAP-ASYNC: When using WRITE-MAP, use asynchronous flushes to disk. As with SYNC NIL, a system crash can then corrupt the database or lose the last transactions. Calling #sync ensures on-disk database integrity until next commit.

    Open environments have a finalizer attached to them that takes care of freeing foreign resources. Thus, the common idiom:

    (setq *env* (open-env "some-path"))

    is okay for development, too. No need to always do WITH-ENV, which does not mesh with threads anyway.

    Wraps mdb_env_create() and mdb_env_open().

  • [function] CLOSE-ENV ENV &KEY FORCE

    Close ENV and free the memory. Closing an already closed ENV has no effect.

    Since accessing Transactions, Databases and Cursors after closing their environment would risk database curruption, CLOSE-ENV makes sure that they are not in use. There are two ways this can happen:

    • If ENV was opened :SYNCHRONIZED (see OPEN-ENV), then CLOSE-ENV waits until there are no active transactions in ENV before closing it. This requires synchronization and introduces some overhead, which might be noticable for workloads involving lots of quick read transactions. It is an LMDB-ERROR to attempt to close an environment in a WITH-TXN to avoid deadlocks.

    • On the other hand, if SYNCHRONIZED was NIL, then - unless FORCE is true - calling CLOSE-ENV signals an LMDB-ERROR to avoid the Safety issues involved in closing the environment. Environments opened with :SYNCHRONIZED NIL are only closed when they are garbage collected and their finalizer is run. Still, for production it might be worth it to gain the last bit of performance.

    Wraps mdb_env_close().

  • [variable] *ENV* NIL

    The default ENV for macros and function that take an environment argument.

  • [macro] WITH-ENV (ENV PATH &REST OPEN-ENV-ARGS) &BODY BODY

    Bind the variable ENV to a new enviroment returned by OPEN-ENV called with PATH and OPEN-ENV-ARGS, execute BODY, and CLOSE-ENV. The following example binds the default environment:

    (with-env (*env* "/tmp/lmdb-test" :if-does-not-exist :create)
  ...)

  • [function] OPEN-ENV-P ENV

    See if ENV is open, i.e. OPEN-ENV has been called on it without a corresponding CLOSE-ENV.

6.3 Miscellaneous environment functions

  • [function] CHECK-FOR-STALE-READERS &OPTIONAL (ENV *ENV*)

    Check for stale entries in the reader lock table. See Caveats. This function is called automatically by OPEN-ENV. If other OS processes or threads accessing ENV abort without closing read transactions, call this function periodically to get rid off them. Alternatively, close all environments accessing the data file.

    Wraps mdb_reader_check().

  • [function] ENV-STATISTICS &OPTIONAL (ENV *ENV*)

    Return statistics about ENV as a plist.

    • :PAGE-SIZE: The size of a database page in bytes.

    • :DEPTH: The height of the B-tree.

    • :BRANCH-PAGES: The number of internal (non-leaf) pages.

    • :LEAF-PAGES: The number of leaf pages.

    • :OVERFLOW-PAGES: The number of overflow pages.

    • :ENTRIES: The number of data items.

    Wraps mdb_env_stat().

  • [function] ENV-INFO &OPTIONAL (ENV *ENV*)

    Return information about ENV as a plist.

    • :MAP-ADDRESS: Address of memory map, if fixed (see OPEN-ENV's FIXED-MAP).

    • :MAP-SIZE: Size of the memory map in bytes.

    • :LAST-PAGE-NUMBER: Id of the last used page.

    • :LAST-TXN-ID: Id of the last committed transaction.

    • :MAXIMUM-READERS: The number of reader slots.

    • :N-READERS: The number of reader slots current used.

    Wraps mdb_env_info().

  • [function] SYNC-ENV &OPTIONAL (ENV *ENV*)

    Flush the data buffers to disk as in calling fsync(). When ENV had been opened with :SYNC NIL or :META-SYNC NIL, this may be handy to force flushing the OS buffers to disk, which avoids potential durability and integrity issues.

    Wraps mdb_env_sync().

  • [function] ENV-MAX-KEY-SIZE &OPTIONAL (ENV *ENV*)

    Return the maximum size of keys and DUPSORT data in bytes. Depends on the compile-time constant MDB_MAXKEYSIZE in the C library. The default is 511. If this limit is exceeded LMDB-BAD-VALSIZE-ERROR is signalled.

    Wraps mdb_env_get_maxkeysize().

  • [macro] WITH-TEMPORARY-ENV (ENV &REST OPEN-ENV-ARGS) &BODY BODY

    Run BODY with an open temporary environment bound to ENV. In more detail, create an environment in a fresh temporary directory in an OS specific location. OPEN-ENV-ARGS is a list of keyword arguments and values for OPEN-ENV. This macro is intended for testing and examples.

    (with-temporary-env (*env*)
  (let ((db (get-db "test")))
    (with-txn (:write t)
      (put db "k1" #(2 3))
      (print (g3t db "k1")) ; => #(2 3)
      (del db "k1"))))

    Since data corruption in temporary environments is not a concern, unlike WITH-ENV, WITH-TEMPORARY-ENV closes the environment even if it was opened with :SYNCHRONIZED NIL (see OPEN-ENV and CLOSE-ENV).

7 Transactions

The LMDB environment supports transactional reads and writes. By default, these provide the standard ACID (atomicity, consistency, isolation, durability) guarantees. Writes from a transaction are not immediately visible to other transactions. When the transaction is committed, all its writes become visible atomically for future transactions even if Lisp crashes or there is power failure. If the transaction is aborted, its writes are discarded.

Transactions span the entire environment (see ENV). All the updates made in the course of an update transaction - writing records across all databases, creating databases, and destroying databases - are either completed atomically or rolled back.

Write transactions can be nested. Child transactions see the uncommitted writes of their parent. The child transaction can commit or abort, at which point its writes become visible to the parent transaction or are discarded. If the parent aborts, all of the writes performed in the context of the parent, including those from committed child transactions, are discarded.

  • [macro] WITH-TXN (&KEY (ENV '*ENV*) WRITE IGNORE-PARENT (SYNC T) (META-SYNC T)) &BODY BODY

    Start a transaction in ENV, execute BODY. Then, if the transaction is open (see OPEN-TXN-P) and BODY returned normally, attempt to commit the transaction. Next, if BODY performed a non-local exit or committing failed, but the transaction is still open, then abort it. It is explicitly allowed to call COMMIT-TXN or ABORT-TXN within WITH-TXN.

    Transactions provide ACID guarantees (with SYNC and META-SYNC both on). They span the entire environment, they are not specific to individual DB.

    • If WRITE is NIL, the transaction is read-only and no writes (e.g. PUT) may be performed in the transaction. On the flipside, many read-only transactions can run concurrently (see ENV-MAX-READERS), while write transactions are mutually exclusive. Furthermore, the single write transaction can also run concurrently with read transactions, just keep in mind that read transactions hold on to the state of the environment at the time of their creation and thus prevent pages since replaced from being reused.

    • If IGNORE-PARENT is true, then in an enclosing WITH-TXN, instead of creating a child transaction, start an independent transaction.

    • If SYNC is NIL, then no flushing of buffers will take place after a commit as if the environment had been opened with :SYNC NIL.

    • Likewise, META-SYNC is the per-transaction equivalent of the OPEN-ENV's META-SYNC.

    Also see Nesting transactions.

    Wraps mdb_txn_begin().

  • [glossary-term] active transaction

    The active transaction in some environment and thread is the transaction of the innermost WITH-TXN being executed in the thread that belongs to the environment. In most cases, this is simply the enclosing WITH-TXN, but if WITH-TXNs with different :ENV arguments are nested, then it may not be:

    (with-temporary-env (env)
  (let ((db (get-db "db" :env env)))
    (with-temporary-env (inner-env)
      (with-txn (:env env :write t)
        (with-txn (:env inner-env)
          (put db #(1) #(2)))))))

    In the above example, DB is known to belong to ENV so although the immediately enclosing transaction belongs to INNER-ENV, PUT is executed in context of the outer, write transaction because that's the innermost in ENV.

    Operations that require a transaction always attempt to use the active transaction even if it is not open (see OPEN-TXN-P).

  • [function] OPEN-TXN-P &OPTIONAL ENV

    See if there is an active transaction and it is open, i.e. COMMIT-TXN or ABORT-TXN have not been called on it. Also, RESET-TXN without a corresponding RENEW-TXN closes the transaction.

  • [function] TXN-ID

    The ID of TXN. IDs are integers incrementing from 1. For a read-only transaction, this corresponds to the snapshot being read; concurrent readers will frequently have the same transaction ID. Only committed write transactions increment the ID. If a transaction aborts, the ID may be re-used by the next writer.

  • [function] COMMIT-TXN &OPTIONAL ENV

    Commit the innermost enclosig transaction (or active transaction belonging to ENV if ENV is specified) or signal an error if it is not open. If TXN is not nested in another transaction, committing makes updates performed visible to future transactions. If TXN is a child transaction, then committing makes updates visible to its parent only. For read-only transactions, committing releases the reference to a historical version environment, allowing reuse of pages replaced since.

    Wraps mdb_txn_commit().

  • [function] ABORT-TXN &OPTIONAL ENV

    Close TXN by discarding all updates performed, which will then not be visible to either parent or future transactions. Aborting an already closed transaction is a noop. Always succeeds.

    Wraps mdb_txn_abort().

  • [function] RENEW-TXN &OPTIONAL ENV

    Renew TXN that was reset by RESET-TXN. This acquires a new reader lock that had been released by RESET-TXN. After renewal, it is as if TXN had just been started.

    Wraps mdb_txn_renew().

  • [function] RESET-TXN &OPTIONAL ENV

    Abort the open, read-only TXN, release the reference to the historical version of the environment, but make it faster to start another read-only transaction with RENEW-TXN. This is accomplished by not deallocating some data structures, and keeping the slot in the reader table. Cursors opened within the transaction must not be used again, except if renewed (see RENEW-CURSOR). If TXN is an open, read-only transaction, this function always succeeds.

    Wraps mdb_txn_reset().

7.1 Nesting transactions

When WITH-TXNs are nested (i.e. one is executed in the dynamic extent of another), we speak of nested transactions. Transaction can be nested to arbitrary levels. Child transactions may be committed or aborted independently from their parent transaction (the immediately enclosing WITH-TXN). Committing a child transaction only makes the updates made by it visible to the parent. If the parent then aborts, the child's updates are aborted too. If the parent commits, all child transactions that were not aborted are committed, too.

Actually, the C lmdb library only supports nesting write transactions. To simplify usage, the Lisp side turns read-only WITH-TXNs nested in another WITH-TXNs into noops.

(with-temporary-env (*env*)
  (let ((db (get-db "test" :value-encoding :uint64)))
    ;; Create a top-level write transaction.
    (with-txn (:write t)
      (put db "p" 0)
      ;; First child transaction
      (with-txn (:write t)
        ;; Writes of the parent are visible in children.
        (assert (= (g3t db "p") 0))
        (put db "c1" 1))
      ;; Parent sees what the child committed (but it's not visible to
      ;; unrelated transactions).
      (assert (= (g3t db "c1") 1))
      ;; Second child transaction
      (with-txn (:write t)
        ;; Sees writes from the parent that came from the first child.
        (assert (= (g3t db "c1") 1))
        (put db "c1" 2)
        (put db "c2" 2)
        (abort-txn)))
    ;; Create a top-level read transaction to check what was committed.
    (with-txn ()
      ;; Since the second child aborted, its writes are discarded.
      (assert (= (g3t db "p") 0))
      (assert (= (g3t db "c1") 1))
      (assert (null (g3t db "c2"))))))

COMMIT-TXN, ABORT-TXN, and RESET-TXN all close the active transaction (see OPEN-TXN-P). When the active transaction is not open, database operations such as G3T, PUT, DEL signal LMDB-BAD-TXN-ERROR. Furthermore, any Cursors created in the context of the transaction will no longer be valid (but see CURSOR-RENEW).

An LMDB parent transaction and its cursors must not issue operations other than COMMIT-TXN and ABORT-TXN while there are active child transactions. As the Lisp side does not expose transaction objects directly, performing Basic operations in the parent transaction is not possible, but it is possible with Cursors as they are tied to the transaction in which they were created.

IGNORE-PARENT true overrides the default nesting semantics of WITH-TXN and creates a new top-level transaction, which is not a child of the enclosing WITH-TXN.

  • Since LMDB is single-writer, on nesting an IGNORE-PARENT write transaction in another write transaction, LMDB-BAD-TXN-ERROR is signalled to avoid the deadlock.

  • Nesting a read-only WITH-TXN with IGNORE-PARENT in another read-only WITH-TXN is LMDB-BAD-RSLOT-ERROR error with the TLS option because it would create two read-only transactions in the same thread.

Nesting a read transaction in another transaction would be an LMDB-BAD-RSLOT-ERROR according to the C lmdb library, but a read-only WITH-TXN with IGNORE-PARENT NIL nested in another WITH-TXN is turned into a noop so this edge case is papered over.

8 Databases

8.1 The unnamed database

LMDB has a default, unnamed database backed by a B+ tree. This db can hold normal key-value pairs and named databases. The unnamed database can be accessed by passing NIL as the database name to GET-DB. There are some restrictions on the flags of the unnamed database, see LMDB-INCOMPATIBLE-ERROR.

8.2 DUPSORT

A prominent feature of LMDB is the ability to associate multiple sorted values with keys, which is enabled by the DUPSORT argument of GET-DB. Just as a named database is a B+ tree associated with a key (its name) in the B+ tree of the unnamed database, so do these sorted duplicates form a B+ tree under a key in a named or the unnamed database. Among the Basic operations, PUT and DEL are equipped to deal with duplicate values, but G3T is too limited, and Cursors are needed to make full use of DUPSORT.

When using this feature the limit on the maximum key size applies to duplicate data, as well. See ENV-MAX-KEY-SIZE.

8.3 Database API

  • [variable] *DB-CLASS* DB

    The default class that GET-DB instantiates. Must a subclass of DB. This provides a way to associate application specific data with DB objects.

  • [function] GET-DB NAME &KEY (CLASS *DB-CLASS*) (ENV *ENV*) (IF-DOES-NOT-EXIST :CREATE) KEY-ENCODING VALUE-ENCODING INTEGER-KEY REVERSE-KEY DUPSORT INTEGER-DUP REVERSE-DUP DUPFIXED

    Open the database with NAME in the open environment ENV, and return a DB object. If NAME is NIL, then the The unnamed database is opened.

    If GET-DB is called with the same name multiple times, the returned DB objects will be associated with the same database (although they may not be EQ). The first time GET-DB is called with any given name and environment, it must not be from an open transaction. This is because GET-DB starts a transaction itself to comply with C lmdb's requirements on mdb_dbi_open() (see Safety). Since dbi handles are cached within ENV, subsequent calls do not involve mdb_dbi_open() and are thus permissible within transactions.

    CLASS designates the class which will instantiated. See *DB-CLASS*.

    If IF-DOES-NOT-EXIST is :CREATE, then a new named database is created. If IF-DOES-NOT-EXIST is :ERROR, then an error is signalled if the database does not exists.

    KEY-ENCODING and VALUE-ENCODING are both one of NIL, :UINT64, :OCTETS or :UTF-8. KEY-ENCODING is set to :UINT64 when INTEGER-KEY is true. VALUE-ENCODING is set to :UINT64 when INTEGER-DUP is true. Note that changing the encoding does not reencode already existing data. See Encoding and decoding data for the full semantics.

    GET-DB may be called more than once with the same NAME and ENV, and the returned DB objects will have the same underlying C lmdb database, but they may have different KEY-ENCODING and VALUE-ENCODING.

    The following flags are for database creation, they do not have any effect in subsequent calls (except for the The unnamed database).

    • INTEGER-KEY: Keys in the database are C unsigned or size_t integers encoded in native byte order. Keys must all be either unsigned or size_t, they cannot be mixed in a single database.

    • REVERSE-KEY: Keys are strings to be compared in reverse order, from the end of the strings to the beginning. By default, keys are treated as strings and compared from beginning to end.

    • DUPSORT: Duplicate keys may be used in the database (or, from another perspective, keys may have multiple values, stored in sorted order). By default, keys must be unique and may have only a single value. Also, see DUPSORT.

    • INTEGER-DUP: This option specifies that duplicate data items are binary integers, similarly to INTEGER-KEY. Only matters if DUPSORT.

    • REVERSE-DUP: This option specifies that duplicate data items should be compared as strings in reverse order. Only matters if DUPSORT.

    • DUPFIXED: This flag may only be used in combination DUPSORT. When true, data items for this database must all be the same size, which allows further optimizations in storage and retrieval. Currently, the wrapper functions that could take advantage of this (e.g. PUT, CURSOR-PUT, CURSOR-NEXT and CURSOR-VALUE), do not.

    No function to close a database (an equivalent to mdb_dbi_close()) is provided due to subtle races and corruption it could cause when an MDB_dbi (unsigned integer, similar to an fd) is assigned by a subsequent open to another named database.

    Wraps mdb_dbi_open().

  • [class] DB

    A database in an environment (class ENV). Always to be created by GET-DB.

  • [reader] DB-NAME DB (:NAME)

    The name of the database.

  • [reader] DB-KEY-ENCODING DB (:KEY-ENCODING)

    The ENCODING that was passed as KEY-ENCODING to GET-DB.

  • [reader] DB-VALUE-ENCODING DB (:VALUE-ENCODING)

    The ENCODING that was passed as VALUE-ENCODING to GET-DB.

  • [function] DROP-DB NAME PATH &KEY OPEN-ENV-ARGS (DELETE T)

    Empty the database with NAME in the environment denoted by PATH. If DELETE, then delete the database. Since closing a database is dangerous (see GET-DB), DROP-DB opens and closes the environment itself.

    Wraps mdb_drop().

  • [function] DB-STATISTICS DB

    Return statistics about the database.

    Wraps mdb_stat().

9 Encoding and decoding data

In the C lmdb library, keys and values are opaque byte vectors only ever inspected internally to maintain the sort order (of keys and also duplicate values if DUPSORT). The client is given the freedom and the responsibility to choose how to perform conversion to and from byte vectors.

LMDB exposes this full flexibility while at the same time providing reasonable defaults for the common cases. In particular, with the KEY-ENCODING and VALUE-ENCODING arguments of GET-DB, the data (meaning the key or value here) encoding can be declared explicitly.

Even if the encoding is undeclared, it is recommended to use a single type for keys (and duplicate values) to avoid unexpected conflicts that could arise, for example, when the UTF-8 encoding of a string and the :UINT64 encoding of an integer coincide. The same consideration doubly applies to named databases, which share the key space with normal key-value pairs in the default database (see The unnamed database).

Together, :UINT64 and :UTF-8 cover the common cases for keys. They trade off dynamic typing for easy sortability (using the default C lmdb behaviour). On the other hand, when sorting is not concern (either for keys and values), serialization may be done more freely. For this purpose, using an encoding of :OCTETS or NIL with cl-conspack is recommended because it works with complex objects, it encodes object types, it is fast and space-efficient, has a stable specification and an alternative implementation in C. For example:

(with-temporary-env (*env*)
  (let ((db (get-db "test")))
    (with-txn (:write t)
      (put db "key1" (cpk:encode (list :some "stuff" 42)))
      (cpk:decode (g3t db "key1")))))
=> (:SOME "stuff" 42)

Note that multiple DB objects with different encodings can be associated with the same C lmdb database, which declutters the code:

(defvar *cpk-encoding*
  (cons #'cpk:encode (alexandria:compose #'cpk:decode #'mdb-val-to-octets)))

(with-temporary-env (*env*)
  (let ((next-id-db (get-db "test" :key-encoding *cpk-encoding*
                                   :value-encoding :uint64))
        (db (get-db "test" :key-encoding *cpk-encoding*
                           :value-encoding *cpk-encoding*)))
    (with-txn (:write t)
      (let ((id (or (g3t next-id-db :next-id) 0)))
        (put next-id-db :next-id (1+ id))
        (put db id (list :some "stuff" 42))
        (g3t db id)))))
=> (:SOME "stuff" 42)
=> T

  • [type] ENCODING

    The following values are supported:

    • :UINT64: Data to be encoded must be of type (UNSIGNED-BYTE 64), which is then encoded as an 8 byte array in native byte order with UINT64-TO-OCTETS. The reverse transformation takes place when returning values. This is the encoding used for INTEGER-KEY and INTEGER-DUP DBs.

    • :OCTETS: Note the plural. Data to be encoded (e.g. KEY argument of G3T) must be a 1D byte array. If its element type is (UNSIGNED-BYTE 8), then the data can be passed to the foreign code more efficiently, but declaring the element type is not required. For example, VECTORs can be used as long as the actual elements are of type (UNSIGNED-BYTE 8). Foreign byte arrays to be decoded (e.g. the value returned by G3T) are returned as OCTETS.

    • :UTF-8: Data to be encoded must be a string, which is converted to octets by TRIVIAL-UTF-8. Null-terminated. Foreign byte arrays are decoded the same way.

    • NIL: Data is encoded using the default encoding according to its Lisp type: strings as :UTF-8, vectors as :OCTETS, (UNSIGNED-BYTE 64) as :UINT64. Decoding is always performed as :OCTETS.

    • A CONS: Data is encoded by the function in the CAR of the cons and decoded by the function in the CDR. For example, :UINT64 is equivalent to (CONS #'UINT64-TO-OCTETS #'MDB-VAL-TO-UINT64).

  • [macro] WITH-MDB-VAL-SLOTS (%BYTES SIZE MDB-VAL) &BODY BODY

    Bind %BYTES and SIZE locally to the corresponding slots of MDB-VAL. MDB-VAL is an opaque handle for a foreign MDB_val struct, that holds the pointer to a byte array and the number of bytes in the array. This macro is needed to access the foreign data in a function used as *KEY-DECODER* or *VALUE-DECODER*. MDB-VAL is dynamic extent, so don't hold on to it. Also, the pointer to which %BYTES is bound is valid only within the context of current top-level transaction.

  • [type] OCTETS &OPTIONAL (SIZE '*)

    A 1D SIMPLE-ARRAY of (UNSIGNED-BYTE 8).

  • [function] MDB-VAL-TO-OCTETS MDB-VAL

    A utility function provided for writing *KEY-DECODER* and *VALUE-DECODER* functions. It returns a Lisp octet vector that holds the same bytes as MDB-VAL.

  • [function] UINT64-TO-OCTETS N

    Convert an (UNSIGNED-BYTE 64) to OCTETS of length 8 taking the native byte order representation of N. Suitable as a *KEY-ENCODER* or *VALUE-ENCODER*.

  • [function] MDB-VAL-TO-UINT64 MDB-VAL

    Like OCTETS-TO-UINT64, but suitable for *KEY-DECODER* or *VALUE-DECODER* that decodes unsigned 64 bit integers in native byte order. This function is called automatically when the encoding is known to require it (see GET-DB's INTEGER-KEY, :VALUE-ENCODING, etc).

9.1 Overriding encodings

Using multiple DB objects with different encodings is the recommended practice (see the example in Encoding and decoding data), but when that is inconvenient, one can override the encodings with the following variables.

  • [variable] *KEY-ENCODER* NIL

    A function designator, NIL or an ENCODING. If non-NIL, it overrides the encoding method determined by KEY-ENCODING (see GET-DB). It is called with a single argument, the key, when it is to be converted to an octet vector.

  • [variable] *KEY-DECODER* NIL

    A function designator, NIL or an ENCODING. If non-NIL, it is called with a single MDB-VAL argument (see WITH-MDB-VAL-SLOTS), that holds a pointer to data to be decoded and its size. This function is called whenever a key is to be decoded and overrides the KEY-ENCODING argument of GET-DB.

    For example, if we are only interested in the length of the value and want to avoid creating a lisp vector on the heap, we can do this:

    (with-temporary-env (*env*)
  (let ((db (get-db "test")))
    (with-txn (:write t)
      (put db "key1" "abc")
      (let ((*value-decoder* (lambda (mdb-val)
                               (with-mdb-val-slots (%bytes size mdb-val)
                                 (declare (ignore %bytes))
                                 ;; Take null termination into account.
                                 (1- size)))))
        (g3t db "key1")))))
=> 3
=> T

  • [variable] *VALUE-ENCODER* NIL

    Like *KEY-ENCODER*, but for values.

  • [variable] *VALUE-DECODER* NIL

    Like *KEY-DECODER*, but for values.

    Apart from performing actual decoding, the main purpose of *VALUE-DECODER*, one can also pass the foreign data on to other foreign functions such as write() directly from the decoder function and returning a constant such as T to avoid consing.

10 Basic operations

  • [function] G3T DB KEY

    Return the value from DB associated with KEY and T as the second value. If KEY is not found in DB, then NIL is returned. If DB supports DUPSORT, then the first value for KEY will be returned. Retrieval of other values requires the use of Cursors.

    This function is called G3T instead of GET to avoid having to shadow CL:GET when importing the LMDB package. On the other hand, importing the LMDB+ package, which has LMDB::GET exported, requires some shadowing.

    The LMDB+ package is like the LMDB package, but it has #'LMDB:G3T fbound to LMDB+:G3T so it probably needs shadowing to avoid conflict with CL:GET:

    (defpackage lmdb/test
  (:shadow #:get)
  (:use #:cl #:lmdb+))

    Wraps mdb_get().

  • [function] PUT DB KEY VALUE &KEY (OVERWRITE T) (DUPDATA T) APPEND APPEND-DUP (KEY-EXISTS-ERROR-P T)

    Add a KEY, VALUE pair to DB within TXN (which must support writes). Returns T on success.

    • OVERWRITE: If NIL, signal LMDB-KEY-EXISTS-ERROR if KEY already appears in DB.

    • DUPDATA: If NIL, signal LMDB-KEY-EXISTS-ERROR if the KEY, VALUE pair already appears in DB. Has no effect if DB doesn't have DUPSORT.

    • APPEND: Append the KEY, VALUE pair to the end of DB instead of finding KEY's location in the B+ tree by performing comparisons. The client effectively promises that keys are inserted in sort order, which allows for fast bulk loading. If the promise is broken, a LMDB-KEY-EXISTS-ERROR is signalled.

    • APPEND-DUP: The client promises that duplicate values are inserted in sort order. If the promise is broken, a LMDB-KEY-EXISTS-ERROR is signalled.

    • If KEY-EXISTS-ERROR-P is NIL, then instead of signalling LMDB-KEY-EXISTS-ERROR return NIL.

    May signal LMDB-MAP-FULL-ERROR, LMDB-TXN-FULL-ERROR, LMDB-TXN-READ-ONLY-ERROR.

    Wraps mdb_put().

  • [function] DEL DB KEY &KEY VALUE

    Delete KEY from DB. Returns T if data was deleted, NIL otherwise. If DB supports sorted duplicates (DUPSORT), then VALUE is taken into account: if it's NIL, then all duplicate values for KEY are deleted, if it's not NIL, then only the matching value. May signal LMDB-TXN-READ-ONLY-ERROR.

    Wraps mdb_del().

11 Cursors

  • [macro] WITH-IMPLICIT-CURSOR (DB) &BODY BODY

    Like WITH-CURSOR but the cursor object is not accessible directly, only through the default cursor mechanism. The cursor is stack-allocated, which eliminates the consing of WITH-CURSOR. Note that stack allocation of cursors in WITH-CURSOR would risk data corruption if the cursor were accessed beyond its dynamic extent.

    Use WITH-IMPLICIT-CURSOR instead of WITH-CURSOR if a single cursor at a time will suffice. Conversely, use WITH-CURSOR if a second cursor is needed. That is, use

    (with-implicit-cursor (db)
  (cursor-set-key 1))

    but when two cursors iterate in an interleaved manner, use WITH-CURSOR:

    (with-cursor (c1 db)
  (with-cursor (c2 db)
    (cursor-first c1)
    (cursor-last c2)
    (if (some-pred (cursor-value c1) (cursor-value c2))
        (cursor-next c1)
        (cursor-prev c2))
    ...))

    Wraps mdb_cursor_open() and mdb_cursor_close().

  • [class] CURSOR STRUCTURE-OBJECT

  • [function] CURSOR-DB INSTANCE

  • [glossary-term] default cursor

    All operations, described below, that take cursor arguments accept NIL instead of a CURSOR object, in which case the cursor from the immediately enclosing WITH-CURSOR or WITH-IMPLICIT-CURSOR is used. This cursor is referred to as the default cursor.

    To reduce syntactic clutter, some operations thus make cursor arguments &OPTIONAL. When this is undesirable - because there are keyword arguments as well - the cursor may be a required argument as in CURSOR-PUT. Still NIL can be passed explicitly.

11.1 Positioning cursors

The following functions position or initialize a cursor while returning the value (a value with DUPSORT) associated with a key, or both the key and the value. Initialization is successful if there is the cursor points to a key-value pair, which is indicated by the last return value being T.

  • [function] CURSOR-FIRST &OPTIONAL CURSOR

    Move CURSOR to the first key of its database. Return the key, the value and T, or NIL if the database is empty. If DUPSORT, position CURSOR on the first value of the first key.

    Wraps mdb_cursor_get() with MDB_FIRST.

  • [function] CURSOR-FIRST-DUP &OPTIONAL CURSOR

    Move CURSOR to the first duplicate value of the current key. Return the value and T. Return NIL if CURSOR is not positioned.

    Wraps mdb_cursor_get() with MDB_FIRST_DUP.

  • [function] CURSOR-LAST &OPTIONAL CURSOR

    Move CURSOR to the last key of its database. Return the key, the value and T, or NIL if the database is empty. If DUPSORT, position CURSOR on the last value of the last key.

    Wraps mdb_cursor_get() with MDB_LAST.

  • [function] CURSOR-LAST-DUP &OPTIONAL CURSOR

    Move CURSOR to the last duplicate value of the current key. Return the value and T. Return NIL if CURSOR is not positioned.

    Wraps mdb_cursor_get() with MDB_LAST_DUP.

  • [function] CURSOR-NEXT &OPTIONAL CURSOR

    Move CURSOR to the next key-value pair of its database and return the key, the value, and T. Return NIL if there is no next item. If DUPSORT, position CURSOR on the next value of the current key if exists, else the first value of next key. If CURSOR is uninitialized, then CURSOR-FIRST is called on it first.

    Wraps mdb_cursor_get() with MDB_NEXT.

  • [function] CURSOR-NEXT-NODUP &OPTIONAL CURSOR

    Move CURSOR to the first value of next key pair of its database (skipping over duplicate values of the current key). Return the key, the value and T. Return NIL if there is no next item. If CURSOR is uninitialized, then CURSOR-FIRST is called on it first.

    Wraps mdb_cursor_get() with MDB_NEXT_NODUP.

  • [function] CURSOR-NEXT-DUP &OPTIONAL CURSOR

    Move CURSOR to the next value of current key pair of its database. Return the value and T. Return NIL if there is no next value. If CURSOR is uninitialized, then CURSOR-FIRST is called on it first.

    Wraps mdb_cursor_get() with MDB_NEXT_DUP.

  • [function] CURSOR-PREV &OPTIONAL CURSOR

    Move CURSOR to the previous key-value pair of its database. Return the key, the value and T. Return NIL if there is no previous item. If DUPSORT, position CURSOR on the previous value of the current key if exists, else the last value of previous key. If CURSOR is uninitialized, then CURSOR-LAST is called on it first.

    Wraps mdb_cursor_get() with MDB_PREV.

  • [function] CURSOR-PREV-NODUP &OPTIONAL CURSOR

    Move CURSOR to the last value of previous key pair of its database (skipping over duplicate values of the current and the previous key). Return the key, the value, and T. Return NIL if there is no prev item. If CURSOR is uninitialized, then CURSOR-LAST is called on it first.

    Wraps mdb_cursor_get() with MDB_PREV_NODUP.

  • [function] CURSOR-PREV-DUP &OPTIONAL CURSOR

    Move CURSOR to the previous duplicate value of current key pair of its database. Return the value and T. Return NIL if there is no prev value. If CURSOR is uninitialized, then CURSOR-LAST is called on it first.

    Wraps mdb_cursor_get() with MDB_PREV_DUP.

  • [function] CURSOR-SET-KEY KEY &OPTIONAL CURSOR

    Move CURSOR to KEY of its database. Return the corresponding value and T. Return NIL if KEY was not found. If DUPSORT, position CURSOR on the first value of KEY.

    Wraps mdb_cursor_get() with MDB_SET_KEY.

  • [function] CURSOR-SET-KEY-DUP KEY VALUE &OPTIONAL CURSOR

    Move CURSOR to the KEY, VALUE pair of its database and return T on success. Return NIL if the pair was not found.

    Wraps mdb_cursor_get() with MDB_GET_BOTH.

  • [function] CURSOR-SET-RANGE KEY &OPTIONAL CURSOR

    Position CURSOR on the first key equal to or greater than KEY. Return the found key, the value and T. Return NIL if KEY was not found. If DUPSORT, position CURSOR on the first value of the found key.

    Wraps mdb_cursor_get() with MDB_SET_RANGE.

  • [function] CURSOR-SET-RANGE-DUP KEY VALUE &OPTIONAL CURSOR

    Position CURSOR exactly at KEY on the first value greater than or equal to VALUE. Return the value at the position and T on success, or NIL if there is no such value associated with KEY.

    Wraps mdb_cursor_get() with MDB_GET_BOTH_RANGE.

11.2 Basic cursor operations

The following operations are similar to G3T, PUT, DEL (the Basic operations), but G3T has three variants (CURSOR-KEY-VALUE, CURSOR-KEY, and CURSOR-VALUE). All of them require the cursor to be positioned (see Positioning cursors).

  • [function] CURSOR-KEY-VALUE &OPTIONAL CURSOR

    Return the key and value CURSOR is positioned at and T. Return NIL if CURSOR is uninitialized.

    Wraps mdb_cursor_get() with MDB_GET_CURRENT.

  • [function] CURSOR-KEY &OPTIONAL CURSOR

    Return the key CURSOR is positioned at and T. Return NIL if CURSOR is uninitialized.

    Wraps mdb_cursor_get() with MDB_GET_CURRENT.

  • [function] CURSOR-VALUE &OPTIONAL CURSOR

    Return the value CURSOR is positioned at and T. Return NIL if CURSOR is uninitialized.

    Wraps mdb_cursor_get() with MDB_GET_CURRENT.

  • [function] CURSOR-PUT KEY VALUE CURSOR &KEY CURRENT (OVERWRITE T) (DUPDATA T) APPEND APPEND-DUP

    Like PUT, store key-value pairs into CURSOR's database. CURSOR is positioned at the new item, or on failure usually near it. Return VALUE.

    • CURRENT: Replace the item at the current cursor position. KEY must still be provided, and must match it. If using sorted duplicates (DUPSORT), VALUE must still sort into the same place. This is intended to be used when the new data is the same size as the old. Otherwise it will simply perform a delete of the old record followed by an insert.

    • OVERWRITE: If NIL, signal LMDB-KEY-EXISTS-ERROR if KEY already appears in CURSOR-DB.

    • DUPDATA: If NIL, signal LMDB-KEY-EXISTS-ERROR if the KEY, VALUE pair already appears in DB. Has no effect if CURSOR-DB doesn't have DUPSORT.

    • APPEND: Append the KEY, VALUE pair to the end of CURSOR-DB instead of finding KEY's location in the B+ tree by performing comparisons. The client effectively promises that keys are inserted in sort order, which allows for fast bulk loading. If the promise is broken, LMDB-KEY-EXISTS-ERROR is signalled.

    • APPEND-DUP: The client promises that duplicate values are inserted in sort order. If the promise is broken, LMDB-KEY-EXISTS-ERROR is signalled.

    May signal LMDB-MAP-FULL-ERROR, LMDB-TXN-FULL-ERROR, LMDB-TXN-READ-ONLY-ERROR.

    Wraps mdb_cursor_put().

11.3 Miscellaneous cursor operations

  • [function] CURSOR-RENEW &OPTIONAL CURSOR

    Associate CURSOR with the active transaction (which must be read-only) as if it had been created with that transaction to begin with to avoid allocation overhead. CURSOR-DB stays the same. This may be done whether the previous transaction is open or closed (see OPEN-TXN-P). No values are returned.

    Wraps mdb_cursor_renew().

  • [macro] DO-CURSOR (KEY-VAR VALUE-VAR CURSOR &KEY FROM-END NODUP) &BODY BODY

    Iterate over key-value pairs starting from the position of CURSOR. If CURSOR is not positioned then no key-value pairs will be seen. If FROM-END, then iterate with CURSOR-PREV instead of CURSOR-NEXT. If NODUP, then make that CURSOR-PREV-NODUP and CURSOR-NEXT-NODUP.

    If CURSOR is NIL, the default cursor is used.

    If NODUP and not FROM-END, then the first duplicate of each key will be seen. If NODUP and FROM-END, then the last duplicate of each key will be seen.

    To iterate over all key-value pairs with keys >= 7:

    (with-cursor (cursor db)
  (cursor-set-key 7 cursor)
  (do-cursor (key value cursor)
    (print (cons key value))))

  • [macro] DO-CURSOR-DUP (VALUE-VAR CURSOR &KEY FROM-END) &BODY BODY

    Iterate over duplicate values with starting from the position of CURSOR. If CURSOR is not positioned then no values will be seen. If FROM-END, then iterate with CURSOR-PREV-DUP instead of CURSOR-NEXT-DUP.

    If CURSOR is NIL, the default cursor is used.

    To iterate over all values that not smaller than #(3 4 5), associated with the key 7:

    (with-cursor (cursor db)
  (cursor-set-key-dup cursor 7 #(3 4 5))
  (do-cursor-dup (value cursor)
    (print value)))

  • [macro] DO-DB (KEY-VAR VALUE-VAR DB &KEY FROM-END NODUP) &BODY BODY

    Iterate over all keys and values in DB. If NODUP, then all but the first (or last if FROM-END) value for each key are skipped. If FROM-END, then iterate in reverse order.

    To iterate over all values in DB:

    (do-db (key value db)
  (print (cons key value)))

    This macro establishes a default cursor.

  • [macro] DO-DB-DUP (VALUE-VAR DB KEY &KEY FROM-END) &BODY BODY

    Iterate over all values associated with KEY in DB. If FROM-END, then iteration starts at the largest value.

    To iterate over all values associated with the key 7:

    (do-db-dup (value db 7)
  (print value))

    This macro establishes a default cursor.

  • [function] LIST-DUPS DB KEY &KEY FROM-END

    A thin wrapper around DO-DB-DUP, this function returns all values associated with KEY in DB as a list. If FROM-END, then the first element of the list is the largest value.

12 Conditions

  • [condition] LMDB-SERIOUS-CONDITION SERIOUS-CONDITION

    The base class of all LMDB conditions. Conditions that are LMDB-SERIOUS-CONDITIONs, but not LMDB-ERRORs are corruption and internal errors, which are hard to recover from.

  • [condition] LMDB-ERROR LMDB-SERIOUS-CONDITION ERROR

    Base class for normal, recoverable LMDB errors.

12.1 Conditions for C lmdb error codes

The following conditions correspond to C lmdb error codes.

  • [condition] LMDB-KEY-EXISTS-ERROR LMDB-ERROR

    Key-value pair already exists. Signalled by PUT and CURSOR-PUT.

  • [condition] LMDB-NOT-FOUND-ERROR LMDB-ERROR

    Key-value pair does not exist. All functions (G3T, CURSOR-NEXT, ...) should return NIL instead of signalling this error. If it is signalled, that's a bug.

  • [condition] LMDB-PAGE-NOT-FOUND-ERROR LMDB-SERIOUS-CONDITION

    Requested page not found - this usually indicates corruption.

  • [condition] LMDB-CORRUPTED-ERROR LMDB-SERIOUS-CONDITION

    Located page was wrong type.

  • [condition] LMDB-PANIC-ERROR LMDB-SERIOUS-CONDITION

    Update of meta page failed or environment had fatal error.

  • [condition] LMDB-VERSION-MISMATCH-ERROR LMDB-ERROR

    Environment version mismatch.

  • [condition] LMDB-INVALID-ERROR LMDB-SERIOUS-CONDITION

    File is not a valid LMDB file.

  • [condition] LMDB-MAP-FULL-ERROR LMDB-ERROR

    ENV-MAP-SIZE reached. Reopen the environment with a larger :MAP-SIZE.

  • [condition] LMDB-DBS-FULL-ERROR LMDB-ERROR

    ENV-MAX-DBS reached. Reopen the environment with a higher :MAX-DBS.

  • [condition] LMDB-READERS-FULL-ERROR LMDB-ERROR

    ENV-MAX-READERS reached. Reopen the environment with a higher :MAX-READERS.

  • [condition] LMDB-TXN-FULL-ERROR LMDB-ERROR

    TXN has too many dirty pages. This condition is expected to occur only when using nested read-write transactions or operations multiple items (currently not supported by this wrapper).

  • [condition] LMDB-CURSOR-FULL-ERROR LMDB-SERIOUS-CONDITION

    Cursor stack too deep - internal error.

  • [condition] LMDB-PAGE-FULL-ERROR LMDB-SERIOUS-CONDITION

    Page has not enough space - internal error.

  • [condition] LMDB-MAP-RESIZED-ERROR LMDB-ERROR

    Data file contents grew beyond ENV-MAP-SIZE. This can happen if another OS process using the same environment path set a larger map size than this process did.

  • [condition] LMDB-INCOMPATIBLE-ERROR LMDB-ERROR

    Operation and DB incompatible, or DB type changed. This can mean:

    • The operation expects a DUPSORT or DUPFIXED database.

    • Opening a named DB when the unnamed DB has DUPSORT or INTEGER-KEY.

    • Accessing a data record as a database, or vice versa.

    • The database was dropped and recreated with different flags.

  • [condition] LMDB-BAD-RSLOT-ERROR LMDB-ERROR

    Invalid reuse of reader locktable slot. May be signalled by WITH-TXN.

  • [condition] LMDB-BAD-TXN-ERROR LMDB-ERROR

    Transaction must abort, has a child, or is invalid. Signalled, for example, when a read-only transaction is nested in a read-write transaction, or when a cursor is used whose transaction has been closed (committed, aborted, or reset).

  • [condition] LMDB-BAD-VALSIZE-ERROR LMDB-ERROR

    Unsupported size of key/DB name/data, or wrong DUPFIXED, INTEGER-KEY or INTEGER-DUP. See ENV-MAX-KEY-SIZE.

  • [condition] LMDB-BAD-DBI-ERROR LMDB-ERROR

    The specified DBI was changed unexpectedly.

12.2 Additional conditions

The following conditions do not have a dedicated C lmdb error code.

  • [condition] LMDB-CURSOR-UNINITIALIZED-ERROR LMDB-ERROR

    Cursor was not initialized. Position the cursor at a key-value pair with a function like CURSOR-FIRST or CURSOR-SET-KEY. Signalled when some functions return the C error code EINVAL.

  • [condition] LMDB-CURSOR-THREAD-ERROR LMDB-ERROR

    Cursor was accessed from a thread other than the one in which it was created. Since the foreign cursor object's lifetime is tied to the dynamic extent of its WITH-CURSOR, this might mean accessing garbage in foreign memory with unpredictable consequences.

  • [condition] LMDB-TXN-READ-ONLY-ERROR LMDB-ERROR

    Attempt was made to write in a read-only transaction. Signalled when some functions return the C error code EACCESS.

  • [condition] LMDB-ILLEGAL-ACCESS-TO-PARENT-TXN-ERROR LMDB-ERROR

    A parent transaction and its cursors may not issue any other operations than COMMIT-TXN and ABORT-TXN while it has active child transactions. In LMDB, Basic operations are always executed in the active transaction, but Cursors can refer to the parent transaction:

    (with-temporary-env (*env*)
  (let ((db (get-db "db")))
    (with-txn (:write t)
      (put db #(1) #(1))
      (with-cursor (cursor db)
        (with-txn (:write t)
          (assert-error lmdb-illegal-access-to-parent-txn-error
            (cursor-set-key #(1) cursor)))))))
[generated by MGL-PAX]

About

LMDB bindings for Common Lisp

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

43 stars

Watchers

6 watching

Forks

7 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •  

Languages