ethdb readme, db objects diagram (erigontech#1281)

* db objects diagram * db objects diagram * db objects diagram * db objects diagram * db objects diagram * db objects diagram * db objects diagram
bobanetwork · Oct 23, 2020 · 0bc61c0 · 0bc61c0
1 parent dd5258d
commit 0bc61c0
Show file tree

Hide file tree

Showing 2 changed files with 130 additions and 232 deletions.
diff --git a/ethdb/AbstractKV.md b/ethdb/AbstractKV.md
diff --git a/ethdb/Readme.md b/ethdb/Readme.md
@@ -0,0 +1,130 @@
+#### `Ethdb` package hold's bouquet of objects to access DB
+
+Words "KV" and "DB" have special meaning here: 
+- KV - key-value-style API to access data: let developer manage transactions, stateful cursors. 
+- DB - object-oriented-style API to access data: Get/Put/Delete/WalkOverTable/MultiPut, managing transactions internally.
+
+So, DB abstraction fits 95% times and leads to more maintainable code - because it's looks stateless. 
+
+About "key-value-style": Modern key-value databases don't provide Get/Put/Delete methods, 
+  because it's very hard-drive-unfriendly - it pushes developers do random-disk-access which is [order of magnitude slower than sequential read](https://www.seagate.com/sg/en/tech-insights/lies-damn-lies-and-ssd-benchmark-master-ti/).
+  To enforce sequential-reads - introduced stateful cursors/iterators - they intentionally look as file-api: open_cursor/seek/write_data_from_current_position/move_to_end/step_back/step_forward/delete_key_on_current_position/append.
+
+## Class diagram: 
+
+```asciiflow.com
+// This is not call graph, just show classes from low-level to high-level. 
+// And show which classes satisfy which interfaces.
+
++-----------------------------------+   +-----------------------------------+   +-----------------------------------+ 
+|  github.com/ledgerwatch/lmdb-go   |   |  github.com/torquem-ch/mdbx-go    |   | google.golang.org/grpc.ClientConn |                    
+|  (app-agnostic LMDB go bindings)  |   |  (app-agnostic MDBX go bindings)  |   | (app-agnostic RPC and streaming)  |
++-----------------------------------+   +-----------------------------------+   +-----------------------------------+
+                 |                                        |                                      |
+                 |                                        |                                      |
+                 v                                        v                                      v
++-----------------------------------+   +-----------------------------------+   +-----------------------------------+
+|      ethdb/kv_lmdb.go             |   |       ethdb/kv_mdbx.go            |   |       ethdb/kv_remote.go          |                
+| (tg-specific LMDB implementaion)  |   |  (tg-specific MDBX implementaion) |   |   (tg-specific remote DB access)  |              
++-----------------------------------+   +-----------------------------------+   +-----------------------------------+
+                 |                                        |                                      |
+                 |                                        |                                      |
+                 v                                        v                                      v
+            +----------------------------------------------------------------------------------------------+
+            |                                       ethdb/kv_abstract.go                                   |  
+            |         (Common KV interface. DB-friendly, disk-friendly, cpu-cache-friendly.                |
+            |           Same app code can work with local or remote database.                              |
+            |           Allows experiment with another database implementations.                           |
+            |          Supports context.Context for cancelation. Any operation can return error)           |
+            +----------------------------------------------------------------------------------------------+
+                 |                                        |                                      |
+                 |                                        |                                      |
+                 v                                        v                                      v
++-----------------------------------+   +-----------------------------------+   +-----------------------------------+
+|       ethdb/object_db.go          |   |          ethdb/tx_db.go           |   |    ethdb/remote/remotedbserver    |                
+|     (thread-safe, stateless,      |   | (non-thread-safe, more performant |   | (grpc server, using kv_abstract,  |  
+|   opens/close short transactions  |   |   than object_db, method Begin    |   |   kv_remote call this server, 1   |
+|      internally when need)        |   |  DOESN'T create new TxDb object)  |   | transaction maps on 1 grpc stream |
++-----------------------------------+   +-----------------------------------+   +-----------------------------------+
+                |                                          |                                     
+                |                                          |                                     
+                v                                          v                                     
+            +-----------------------------------------------------------------------------------------------+
+            |                                    ethdb/interface.go                                         |  
+            |     (Common DB interfaces. ethdb.Database and ethdb.DbWithPendingMutations are widely used)   |
+            +-----------------------------------------------------------------------------------------------+
+                |                      
+                |                      
+                v                      
++--------------------------------------------------+ 
+|             ethdb/mutation.go                    |                 
+| (also known as "batch", recording all writes and |  
+|   them flush to DB in sorted way only when call  | 
+|     .Commit(), use it to avoid random-writes.    | 
+|   It use and satisfy ethdb.Database in same time |
++--------------------------------------------------+ 
+
+```
+
+
+## ethdb.AbstractKV design:
+
+- InMemory, ReadOnly: `NewLMDB().InMem().ReadOnly().Open()`
+- MultipleDatabases, Customization: `NewLMDB().Path(path).WithBucketsConfig(config).Open()`
+
+
+- 1 Transaction object can be used only withing 1 goroutine.
+- Only 1 write transaction can be active at a time (other will wait). 
+- Unlimited read transactions can be active concurrently (not blocked by write transaction).
+
+
+- Methods db.Update, db.View - can be used to open and close short transaction.
+- Methods Begin/Commit/Rollback - for long transaction.
+- it's safe to call .Rollback() after .Commit(), multiple rollbacks are also safe. Common transaction patter:
+```
+tx, err := db.Begin(true, nil, ethdb.RW)
+if err != nil {
+    return err
+}
+defer tx.Rollback() // important to avoid transactions leak at panic or early return
+
+// ... code which uses database in transaction
+ 
+err := tx.Commit()
+if err != nil {
+    return err
+}
+```
+
+
+- No internal copies/allocations. It means: 1. app must copy keys/values before put to database. 2. Data after read from db - valid only during current transaction - copy it if plan use data after transaction Commit/Rollback.
+- Methods .Bucket() and .Cursor(), can’t return nil, can't return error.
+- Bucket and Cursor - are interfaces - means different classes can satisfy it: for example `LmdbCursor`, `LmdbDupSortCursor`, `LmdbDupFixedCursor` classes satisfy it. 
+  If your are not familiar with "DupSort" concept, please read [indices.md](./../docs/programmers_guide/indices.md) first.
+
+
+- If Cursor returns err!=nil then key SHOULD be != nil (can be []byte{} for example). 
+Then traversal code look as: 
+```go
+for k, v, err := c.First(); k != nil; k, v, err = c.Next() {
+    if err != nil {
+        return err
+    }
+    // logic
+}
+``` 
+- Move cursor: `cursor.Seek(key)`
+
+
+
+## ethdb.Database design:
+
+- Allows pass multiple implementations 
+- Allows traversal tables by `db.Walk` and `db.MultiWalk` 
+
+## ethdb.TxDb design:
+- holds inside 1 long-running transaction and 1 cursor per table
+- method Begin DOESN'T create new TxDb object, it means this object can be passed into other objects by pointer, 
+  and high-level app code can start/commit transactions when it needs without re-creating all objects which holds 
+  TxDb pointer.
+- This is reason why txDb.CommitAndBegin() method works: inside it creating new transaction object, pinter to TxDb stays valid.