Skip to content

Commit

Permalink
Implementation of batch replace
Browse files Browse the repository at this point in the history
Batch upsert is mostly used for operation with
one bucket / one Tarantool node in a transaction.
In this case batch replace is more efficient
then replacing tuple-by-tuple.
Right now CRUD cannot provide batch replace with full consistency.
CRUD offers batch upsert with partial consistency. That means
that full consistency can be provided only on single replicaset
using `box` transactions.

Part of #193
  • Loading branch information
AnaNek committed Jun 27, 2022
1 parent 063c4ba commit 598e447
Show file tree
Hide file tree
Showing 9 changed files with 2,556 additions and 0 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
## [Unreleased]

### Added
* Batch insert/upsert operation
`crud.insert_many()`/`crud.insert_object_many()`/
`crud.upsert_many()`/`crud.upsert_object_many()`
`crud.replace_many()`/`crud.replace_object_many()`
with partial consistency

### Changed

Expand Down
96 changes: 96 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ It also provides the `crud-storage` and `crud-router` roles for
- [Update](#update)
- [Delete](#delete)
- [Replace](#replace)
- [Replace many](#replace-many)
- [Upsert](#upsert)
- [Upsert many](#upsert-many)
- [Select](#select)
Expand Down Expand Up @@ -477,6 +478,101 @@ crud.replace_object('customers', {
...
```

### Replace many

```lua
-- Replace batch of tuples
local result, err = crud.replace_many(space_name, tuples, opts)
-- Replace batch of objects
local result, err = crud.replace_object_many(space_name, objects, opts)
```

where:

* `space_name` (`string`) - name of the space to insert/replace an object
* `tuples` / `objects` (`table`) - array of tuples/objects to insert
* `opts`:
* `timeout` (`?number`) - `vshard.call` timeout (in seconds)
* `fields` (`?table`) - field names for getting only a subset of fields
* `stop_on_error` (`?boolean`) - stop on a first error and report error
regarding the failed operation and error about what tuples were not
performed, default is `false`
* `rollback_on_error` (`?boolean`) - any failed operation will lead to
rollback on a storage, where the operation is failed, report error
about what tuples were rollback, default is `false`

Returns metadata and array contains inserted rows, array of errors.
Error object can contain field `operation_data`.

This field can contain:
* tuple for which the error occurred;
* object with an incorrect format;
* tuple the operation on which was performed but
operation was rollback;
* tuple the operation on which was not performed
because operation was stopped by error.

Right now CRUD cannot provide batch replace with full consistency.
CRUD offers batch replace with partial consistency. That means
that full consistency can be provided only on single replicaset
using `box` transactions.

**Example:**

```lua
crud.replace_many('developers', {
{1, box.NULL, 'Elizabeth', 'lizaaa'},
{2, box.NULL, 'Anastasia', 'iamnewdeveloper'},
})
---
- metadata:
- {'name': 'id', 'type': 'unsigned'}
- {'name': 'bucket_id', 'type': 'unsigned'}
- {'name': 'name', 'type': 'string'}
- {'name': 'login', 'type': 'string'}
rows:
- [1, 477, 'Elizabeth', 'lizaaa']
- [2, 401, 'Anastasia', 'iamnewdeveloper']
...
crud.replace_object_many('developers', {
{id = 1, name = 'Inga', login = 'mylogin'},
{id = 10, name = 'Anastasia', login = 'qwerty'},
})
---
- metadata:
- {'name': 'id', 'type': 'unsigned'}
- {'name': 'bucket_id', 'type': 'unsigned'}
- {'name': 'name', 'type': 'string'}
- {'name': 'age', 'type': 'number'}
rows:
- [1, 477, 'Inga', 'mylogin']
- [10, 569, 'Anastasia', 'qwerty']

-- Partial success
-- Let's say login has unique secondary index
local res, errs = crud.replace_object_many('developers', {
{id = 22, name = 'Alex', login = 'pushkinn'},
{id = 3, name = 'Anastasia', login = 'qwerty'},
{id = 5, name = 'Sergey', login = 's.petrenko'},
})
---
res
- metadata:
- {'name': 'id', 'type': 'unsigned'}
- {'name': 'bucket_id', 'type': 'unsigned'}
- {'name': 'name', 'type': 'string'}
- {'name': 'age', 'type': 'number'}
rows:
- [5, 1172, 'Sergey', 'crudisthebest'],
- [22, 655, 'Alex', 'pushkinn'],

#errs -- 1
errs[1].class_name -- ReplaceManyError
errs[1].err -- 'Duplicate key exists <...>'
errs[1].tuple -- {3, 2804, 'Anastasia', 'qwerty'}
...
```

### Upsert

```lua
Expand Down
10 changes: 10 additions & 0 deletions crud.lua
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ local cfg = require('crud.cfg')
local insert = require('crud.insert')
local insert_many = require('crud.insert_many')
local replace = require('crud.replace')
local replace_many = require('crud.replace_many')
local get = require('crud.get')
local update = require('crud.update')
local upsert = require('crud.upsert')
Expand Down Expand Up @@ -53,6 +54,14 @@ crud.replace = stats.wrap(replace.tuple, stats.op.REPLACE)
-- @function replace_object
crud.replace_object = stats.wrap(replace.object, stats.op.REPLACE)

-- @refer replace_many.tuples
-- @function replace_many
crud.replace_many = replace_many.tuples

-- @refer replace_many.objects
-- @function replace_object_many
crud.replace_object_many = replace_many.objects

-- @refer update.call
-- @function update
crud.update = stats.wrap(update.call, stats.op.UPDATE)
Expand Down Expand Up @@ -145,6 +154,7 @@ function crud.init_storage()
insert_many.init()
get.init()
replace.init()
replace_many.init()
update.init()
upsert.init()
upsert_many.init()
Expand Down
Loading

0 comments on commit 598e447

Please sign in to comment.