From 41f457c3b52be50b74911212a2e47ef677ca2f06 Mon Sep 17 00:00:00 2001 From: gyuwonMoon <78714820+MoonGyu1@users.noreply.github.com> Date: Fri, 25 Aug 2023 18:16:31 +0900 Subject: [PATCH] Add tree document (#608) --- design/data-structure.md | 15 ++-- design/tree.md | 166 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+), 5 deletions(-) create mode 100644 design/tree.md diff --git a/design/data-structure.md b/design/data-structure.md index b736810a2..bfab2f745 100644 --- a/design/data-structure.md +++ b/design/data-structure.md @@ -1,6 +1,6 @@ --- title: data-structure -target-version: 0.3.1 +target-version: 0.4.6 --- # Data Structures @@ -28,7 +28,9 @@ The `json` and `crdt` package has data structures for representing the contents Below is the dependency graph of data structures used in a JSON-like document. -![data-structure](./media/data-structure.png) + + + The data structures can be divided into three groups: @@ -45,7 +47,9 @@ JSON-like data strucutres are used when editing JSON-like documents. - `Primitive`: represents primitive data like `string`, `number`, `boolean`, `null`, etc. - `Object`: represents [object type](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object) of JavaScript. Just like JavaScript, you can use `Object` as [hash table](https://en.wikipedia.org/wiki/Hash_table). - `Array`: represents [array type](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array) of JavaScript. You can also use `Array` as [list](https://en.wikipedia.org/wiki/List_(abstract_data_type)). -- `Text`: represents text with style attributes in rich text editors such as [Quill](https://github.com/yorkie-team/yorkie-js-sdk/blob/main/examples/quill.html). Users can express styles such as bold, italic, and underline to text content. Of course, it can represent just a plain text in text-based editors such as [CodeMirror](https://github.com/yorkie-team/yorkie-js-sdk/blob/main/examples/index.html). It supports collaborative editing; multiple users can modify parts of the contents without conflict. +- `Text`: represents text with style attributes in rich text editors such as [Quill](https://quilljs.com/). Users can express styles such as bold, italic, and underline to text content. Of course, it can represent just a plain text in text-based editors such as [CodeMirror](https://codemirror.net). It supports collaborative editing; multiple users can modify parts of the contents without conflict. +- `Counter`: represents a counter in the document. As a proxy for the CRDT counter, it is used when the user manipulates the counter from the outside. +- `Tree`: represents CRDT-based tree structure that is used to represent the document tree of text-based editor such as [ProseMirror](https://prosemirror.net/). JSON-like data structures can be edited through proxies. For example: @@ -72,7 +76,7 @@ CRDT data structures are used by JSON-like group to resolve conflicts in concurr - `ElementRHT`: similar to `RHT`, but has elements as values. - `RGATreeList`: extended `RGA(Replicated Growable Array)` with an additional index tree. The index tree manages the indices of elements and provides faster access to elements at the int-based index. - `RGATreeSplit`: extended `RGATreeList` allowing characters to be represented as blocks rather than each single character. - +- `CRDTTree`: represents the CRDT tree with an index tree structure'. It resolves conflicts arising from concurrent editing. ### Common Group Common data structures can be used for general purposes. @@ -80,7 +84,8 @@ Common data structures can be used for general purposes. - [`SplayTree`](https://en.wikipedia.org/wiki/Splay_tree): A tree that moves nodes to the root by splaying. This is effective when user frequently access the same location, such as text editing. We use `SplayTree` as an index tree to give each node a weight, and to quickly access the node based on the index. - [`LLRBTree`](https://en.wikipedia.org/wiki/Left-leaning_red%E2%80%93black_tree): A tree simpler than Red-Black Tree. Newly added `floor` method finds the node of the largest key less than or equal to the given key. - [`Trie`](https://en.wikipedia.org/wiki/Trie): A data structure that can quickly search for prefixes of sequence data such as strings. We use `Trie` to remove nested events when the contents of the `Document` are modified at once. - +- `IndexTree`: A tree implementation to represent a document of text-based editors. + ### Risks and Mitigation We can replace the data structures with better ones for some reason, such as performance. For example, `SplayTree` used in `RGATreeList` can be replaced with [TreeList](https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/list/TreeList.html). diff --git a/design/tree.md b/design/tree.md new file mode 100644 index 000000000..ab7071acc --- /dev/null +++ b/design/tree.md @@ -0,0 +1,166 @@ +--- +title: tree +target-version: 0.4.6 +--- + +# Tree + +## Summary + +In Yorkie, users can create and edit JSON-like documents using JSON-like data structures such as `Primitive`, `Object`, `Array`, `Text`, and `Tree`. Among these, the `Tree` structure is used to represent the document model of a tree-based text editor, similar to XML. + +This document introduces the `Tree` data structure, and explains the operations provided by `Tree`, focusing on the `Tree` coordinate system and the logic of the `Tree.Edit` operation. Furthermore, it explains how this logic ensures eventual consistency in concurrent document editing scenarios. + +### Goals + +This document aims to help new SDK contributors understand the overall `Tree` data structure and explain how Yorkie ensures consistency when multiple clients are editing concurrently. + +### Non-Goals + +This document focuses on `Tree.Edit` operations rather than `Tree.Style`. + +## Proposal Details + +### XML-like Tree + +In yorkie, a XML-like `Tree` is used to represent the document model of a tree-based text editor. + +This tree-based document model resembles XML tree and consists of element nodes and text nodes. element nodes can have attributes, and text nodes contain a string as their value. For example: + + + +**Operation** + +The XML-like `Tree` provides specialized operations tailored for text editing rather than typical operations of a general tree. To specify the operation's range, an `index` or `path` is used. For example: + + + +These `index`es are assigned in order at positions where the user's cursor can reach. These `index`es draw inspiration from ProseMirror's index and share a similar structural concept. + +In the case of a `path`, it contains `offset`s of each node from the root node as elements except the last. The last element of the `path` represents the position in the parent node. For example, the `path` of the position between '`k`' and '`i`' is `[1, 4]`. The first element of the `path` is the `offset` of the `` in `

` and the second element represents the position between '`k`' and '`i`' in ``. + +1. `Tree.Edit` + +Users can use the `Edit` operation to insert or delete nodes within the `Tree`. + +https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L115-L131 + +Where `fromIdx` is the starting position of editing, `toIdx` is the ending position, and `contents` represent the nodes to be inserted. If `contents` are omitted, the operation only deletes nodes between `fromIdx` and `toIdx`. + + + +Similarly, users can specify the editing range using a `path` that leads to the `Tree`'s node in the type of `[]int`. + +https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L217-L237 + +2. `Tree.Style` + +Users can use the `Style` operation to specify attributes for the element nodes in the `Tree`. + +https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L239-L268 + +### Implementation of Edit Operation + +**Tree Coordinate System** + + + +Yorkie implements the above [data structure](https://github.com/yorkie-team/yorkie/blob/main/design/data-structure.md) to create a JSON-like `Document`, which consists of different layers, each with its own coordinate system. The dependency graph above can be divided into three main groups. The **JSON-like** group directly used by users to edit JSON-like `Document`s. The **CRDT** Group is utilized from the JSON-like group to resolve conflicts in concurrent editing situations. Finally, the **common** group is used for the detailed implementation of CRDT group and serves general purposes. + +Thus, the `Tree`, introduced in this document, has dependencies such as '`Tree` → `CRDTTree` → `IndexTree`', and each layer has its own coordinate system: + + + +These coordinate systems transform in the order of '`index(path)` → `IndexTree.TreePos` → `CRDTTree.TreeNodeID` → `CRDTTree.TreePos`'. + + + +1. `index` → `IndexTree.TreePos` + +The `index` is the coordinate system used by users for local editing. This `index` is received from the user, and is converted to `IndexTree.TreePos`. This `IndexTree.TreePos` represents the physical position within the local tree and is used for actual tree editing. + +2. `IndexTree.TreePos` → (`CRDTTree.TreeNodeID`) → `CRDTTree.TreePos` + +Next, the obtained `IndexTree.TreePos` is transformed into the logical coordinate system of the distributed tree, represented by `CRDTTree.TreePos`. To achieve this, the given physical position, `IndexTree.TreePos`, is used to find the parent node and left sibling node. Then, a `CRDTTree.TreePos` is created using the unique IDs of the parent node and left sibling node, which are `CRDTTree.TreeNodeID`. This coordinate system is used in subsequent `Tree.Edit` and `Tree.Style` operations. + +In the case of remote editing, where the local coordinate system is received from the user in local editing, there is no need for Step 1 since changes are pulled from the server using `ChangePack` to synchronize the changes. Refer to the [document-editing](https://github.com/yorkie-team/yorkie/blob/main/design/document-editing.md) for more details. + +**Tree.Edit Logic** + +The core process of the `Tree.Edit` operation is as follows: + +1. Find `CRDTTree.TreePos` from the given `fromIdx` and `toIdx` (local editing only). +2. Find the corresponding left sibling node and parent node within the `IndexTree` based on `CRDTTree.TreePos`. +3. Delete nodes in the range of `fromTreePos` to `toTreePos`. +4. Insert the given nodes at the appropriate positions (insert operation only). + +**[[STEP 1]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L121C1-L128)** Find `CRDTTree.TreePos` from the given `fromIdx` and `toIdx` (local editing only) + +In the case of local editing, the given `index`es are converted to `CRDTTree.TreePos`. The detailed process is the same as described in the 'Tree Coordinate System' above. + +**[[STEP 2]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/crdt/tree.go#L572C1-L580C3)** Find the corresponding left sibling node and parent node within the `IndexTree` based on `CRDTTree.TreePos` + +2-1. For text nodes, if necessary, split nodes at the appropriate positions to find the left sibling node. + +2-2. Determine the sequence of nodes and find the appropriate position. Since `Clone`s of each client might exist in different states, the `findFloorNode` function is used to find the closest node (lower bound). + +**[[STEP 3]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/crdt/tree.go#L582-L640)** Delete nodes in the range of `fromTreePos` to `toTreePos` + +3-1. Traverse the range and identify nodes to be removed. If a node is an element node and doesn't include both opening and closing tags, it is excluded from removal. + +3-2. Update the `latestCreatedAtMapByActor` information for each node and mark nodes with tombstones in the `IndexTree` to indicate removal. + +**[[STEP 4]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/crdt/tree.go#L642-L681)** Insert the given nodes at the appropriate positions (insert operation only) + +4-1. If the left sibling node at the insertion position is the same as the parent node, it means the node will be inserted as the leftmost child of the parent. Hence, the node is inserted at the leftmost position of the parent's children list. + +4-2. Otherwise, the new node is inserted to the right of the left sibling node. + +### How to Guarantee Eventual Consistency + +**Coverage** + + + +Using conditions such as range type, node type, and edit type, 27 possible cases of concurrent editing can be represented. + + + +Eventual consistency is guaranteed for these [27 cases](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/test/integration/tree_test.go#L736-L2094). In addition, eventual consistency is ensured for the following edge cases: + +- Selecting multiple nodes in a multi-level range +- Selecting only a part of nodes (e.g., selecting only the opening tag or closing tag of the node) + +**How does it work?** + +- `lastCreatedAtMapByActor` + +https://github.com/yorkie-team/yorkie/blob/81137b32d0d1d3d36be5b63652e5ab0273f536de/pkg/document/operations/tree_edit.go#L36-L38 + +`latestCreatedAtMapByActor` is a map that stores the latest creation time by actor for the nodes included in the editing range. However, relying solely on the typical `lamport` clocks that represent local clock of clients, it's not possible to determine if two events are causally related or concurrent. For instance: + + + +In the case of the example above, during the process of synchronizing operations between clients A and B, client A is unaware of the existence of '`c`' when client B performs `Edit(0,2)`. As a result, an issue arises where the element '`c`', which is within the contained range, gets deleted together. + +To address this, the `lastCreatedAtMapByActor` is utilized during operation execution to store final timestamp information for each actor. Subsequently, this information allows us to ascertain the causal relationship between the two events. + +- Restricted to only `insertAfter` + +https://github.com/yorkie-team/yorkie/blob/422901861aedbd3a86fdcb9cf3b5740d6daf38eb/pkg/index/tree.go#L552-L570 + +To ensure consistency in concurrent editing scenarios, only the `insertAfter` operation is allowed, rather than `insertBefore`, similar to conventional CRDT algorithms. To achieve this, `CRDTTree.TreePos` takes a form that includes `LeftSiblingID`, thus always maintaining a reference to the left sibling node. + +If the left sibling node is the same as the parent node, it indicates that the node is positioned at the far left of the parent's children list. + +- `FindOffset` + +https://github.com/yorkie-team/yorkie/blob/422901861aedbd3a86fdcb9cf3b5740d6daf38eb/pkg/index/tree.go#L393-L412 + +During the traversal of the given range in `traverseInPosRange` (STEP3), the process of converting the provided `CRDTTree.TreePos` to an `IndexTree.TreePos` is executed. To determine the `offset` for this conversion, the `FindOffset` function is utilized. In doing so, calculating the `offset` excluding the removed nodes prevents potential issues that can arise in concurrent editing scenarios. + +### Risks and Mitigation + +- In the current conflict resolution policy of Yorkie, when both insert and delete operations occur simultaneously, even if the insert range is included in the delete range, the inserted node remains after synchronization. This might not always reflect the user's intention accurately. + +- The `Tree.Edit` logic uses index-based traversal instead of node-based traversal for a clearer implementation. This might lead to a performance impact. If this becomes a concern, switching to node-based traversal can be considered.