Skip to content

Commit

Permalink
Add statistics for CRUD operations on router
Browse files Browse the repository at this point in the history
Add statistics module for collecting metrics of CRUD operations on
router. Wrap all CRUD operation calls in statistics collector.
Statistics must be enabled manually, and they can be disabled, restarted
or re-enabled later.

`crud.stats()` returns

---
- spaces:
    my_space:
      insert:
        ok:
          latency: 0.002
          count: 19800
          time: 39.6
        error:
          latency: 0.000001
          count: 4
          time: 0.000004
  space_not_found: 0
...

`spaces` section contains statistics for each observed space.
If operation has never been called for space, corresponding
field will be empty. If no operations has been called for a
space, it will not be represented. Operations called for
spaces not found both on storages and in statistics registry
(including cases when crud can't verify space existence)
will increment `space_not_found` counter.

Possible statistics operation labels are
`insert` (for `insert` and `insert_object` calls),
`get`, `replace`  (for `replace` and `replace_object` calls), `update`,
`upsert` (for `upsert` and `upsert_object` calls), `delete`,
`select` (for `select` and `pairs` calls), `truncate`, `len` and
`borders` (for `min` and `max` calls).

Each operation section contains of different collectors
for success calls and error (both error throw and `nil, err`)
returns. `count` is total requests count since instance start
or stats restart. `latency` is average time of requests execution,
`time` is total time of requests execution.

Part of #224
  • Loading branch information
DifferentialOrange committed Dec 20, 2021
1 parent 6fcf363 commit 5685009
Show file tree
Hide file tree
Showing 10 changed files with 1,471 additions and 15 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
## [Unreleased]

### Added
* Statistics for CRUD operations on router (#224).

### Changed

Expand Down
71 changes: 71 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,77 @@ crud.len('customers')
...
```

### Statistics

`crud` routers can provide statistics on called operations.
```lua
-- Enable statistics collect.
crud.enable_stats()

-- Returns table with statistics information.
crud.stats()

-- Returns table with statistics information for specific space.
crud.stats('my_space')

-- Disables statistics collect and destroys all collectors.
crud.disable_stats()

-- Destroys all statistics collectors and creates them again.
crud.reset_stats()
```

Format is as follows.
```lua
crud.stats()
---
- spaces:
my_space:
insert:
ok:
latency: 0.002
count: 19800
time: 39.6
error:
latency: 0.000001
count: 4
time: 0.000004
space_not_found: 0
...
crud.stats('my_space')
---
- insert:
ok:
latency: 0.002
count: 19800
time: 39.6
error:
latency: 0.000001
count: 4
time: 0.000004
...
```
`spaces` section contains statistics for each observed space.
If operation has never been called for space, corresponding
field will be empty. If no operations has been called for a
space, it will not be represented. Operations called for
spaces not found both on storages and in statistics registry
(including cases when crud can't verify space existence)
will increment `space_not_found` counter.

Possible statistics operation labels are
`insert` (for `insert` and `insert_object` calls),
`get`, `replace` (for `replace` and `replace_object` calls), `update`,
`upsert` (for `upsert` and `upsert_object` calls), `delete`,
`select` (for `select` and `pairs` calls), `truncate`, `len` and
`borders` (for `min` and `max` calls).

Each operation section contains of different collectors
for success calls and error (both error throw and `nil, err`)
returns. `count` is total requests count since instance start
or stats restart. `latency` is average time of requests execution,
`time` is total time of requests execution.

## Cartridge roles

`cartridge.roles.crud-storage` is a Tarantool Cartridge role that depends on the
Expand Down
47 changes: 32 additions & 15 deletions crud.lua
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ local len = require('crud.len')
local borders = require('crud.borders')
local sharding_key = require('crud.common.sharding_key')
local utils = require('crud.common.utils')
local stats = require('crud.stats.module')

local crud = {}

Expand All @@ -22,67 +23,67 @@ local crud = {}

-- @refer insert.tuple
-- @function insert
crud.insert = insert.tuple
crud.insert = stats.wrap(insert.tuple, stats.op.INSERT)

-- @refer insert.object
-- @function insert_object
crud.insert_object = insert.object
crud.insert_object = stats.wrap(insert.object, stats.op.INSERT)

-- @refer get.call
-- @function get
crud.get = get.call
crud.get = stats.wrap(get.call, stats.op.GET)

-- @refer replace.tuple
-- @function replace
crud.replace = replace.tuple
crud.replace = stats.wrap(replace.tuple, stats.op.REPLACE)

-- @refer replace.object
-- @function replace_object
crud.replace_object = replace.object
crud.replace_object = stats.wrap(replace.object, stats.op.REPLACE)

-- @refer update.call
-- @function update
crud.update = update.call
crud.update = stats.wrap(update.call, stats.op.UPDATE)

-- @refer upsert.tuple
-- @function upsert
crud.upsert = upsert.tuple
crud.upsert = stats.wrap(upsert.tuple, stats.op.UPSERT)

-- @refer upsert.object
-- @function upsert
crud.upsert_object = upsert.object
crud.upsert_object = stats.wrap(upsert.object, stats.op.UPSERT)

-- @refer delete.call
-- @function delete
crud.delete = delete.call
crud.delete = stats.wrap(delete.call, stats.op.DELETE)

-- @refer select.call
-- @function select
crud.select = select.call
crud.select = stats.wrap(select.call, stats.op.SELECT)

-- @refer select.pairs
-- @function pairs
crud.pairs = select.pairs
crud.pairs = stats.wrap(select.pairs, stats.op.SELECT, { pairs = true })

-- @refer utils.unflatten_rows
-- @function unflatten_rows
crud.unflatten_rows = utils.unflatten_rows

-- @refer truncate.call
-- @function truncate
crud.truncate = truncate.call
crud.truncate = stats.wrap(truncate.call, stats.op.TRUNCATE)

-- @refer len.call
-- @function len
crud.len = len.call
crud.len = stats.wrap(len.call, stats.op.LEN)

-- @refer borders.min
-- @function min
crud.min = borders.min
crud.min = stats.wrap(borders.min, stats.op.BORDERS)

-- @refer borders.max
-- @function max
crud.max = borders.max
crud.max = stats.wrap(borders.max, stats.op.BORDERS)

-- @refer utils.cut_rows
-- @function cut_rows
Expand All @@ -92,6 +93,22 @@ crud.cut_rows = utils.cut_rows
-- @function cut_objects
crud.cut_objects = utils.cut_objects

-- @refer stats.enable
-- @function enable_stats
crud.enable_stats = stats.enable

-- @refer stats.get
-- @function stats
crud.stats = stats.get

-- @refer stats.disable
-- @function disable_stats
crud.disable_stats = stats.disable

-- @refer stats.reset
-- @function reset_stats
crud.reset_stats = stats.reset

--- Initializes crud on node
--
-- Exports all functions that are used for calls
Expand Down
124 changes: 124 additions & 0 deletions crud/stats/local_registry.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
local dev_checks = require('crud.common.dev_checks')
local registry_common = require('crud.stats.registry_common')

local registry = {}
local internal_registry = {}

--- Initialize local metrics registry
--
-- Registries are not meant to used explicitly
-- by users, init is not guaranteed to be idempotent.
--
-- @function init
--
-- @treturn boolean Returns true.
--
function registry.init()
internal_registry.spaces = {}
internal_registry.space_not_found = 0

return true
end

--- Destroy local metrics registry
--
-- Registries are not meant to used explicitly
-- by users, destroy is not guaranteed to be idempotent.
--
-- @function destroy
--
-- @treturn boolean Returns true.
--
function registry.destroy()
internal_registry = {}

return true
end

--- Get copy of local metrics registry
--
-- Registries are not meant to used explicitly
-- by users, get is not guaranteed to work without init.
--
-- @function get
--
-- @tparam string space_name
-- (Optional) If specified, returns table with statistics
-- of operations on table, separated by operation type and
-- execution status. If there wasn't any requests for table,
-- returns {}. In not specified, returns table with statistics
-- about all existing spaces and count of calls to spaces
-- that wasn't found.
--
-- @treturn table Returns copy of metrics registry (or registry section).
--
function registry.get(space_name)
dev_checks('?string')

if space_name ~= nil then
return table.deepcopy(internal_registry.spaces[space_name]) or {}
end

return table.deepcopy(internal_registry)
end

--- Check if space statistics are present in registry
--
-- @function is_unknown_space
--
-- @tparam string space_name
-- Name of space.
--
-- @treturn boolean True, if space stats found. False otherwise.
--
function registry.is_unknown_space(space_name)
dev_checks('string')

return internal_registry.spaces[space_name] == nil
end

--- Increase requests count and update latency info
--
-- @function observe
--
-- @tparam string space_name
-- Name of space.
--
-- @tparam number latency
-- Time of call execution.
--
-- @tparam string op
-- Label of registry collectors.
-- Use `require('crud.common.const').OP` to pick one.
--
-- @tparam string success
-- 'ok' if no errors on execution, 'error' otherwise.
--
-- @treturn boolean Returns true.
--
function registry.observe(latency, space_name, op, status)
dev_checks('number', 'string', 'string', 'string')

registry_common.init_collectors_if_required(internal_registry.spaces, space_name, op)
local collectors = internal_registry.spaces[space_name][op][status]

collectors.count = collectors.count + 1
collectors.time = collectors.time + latency
collectors.latency = collectors.time / collectors.count

return true
end

--- Increase count of "space not found" collector by one
--
-- @function observe_space_not_found
--
-- @treturn boolean Returns true.
--
function registry.observe_space_not_found()
internal_registry.space_not_found = internal_registry.space_not_found + 1

return true
end

return registry
Loading

0 comments on commit 5685009

Please sign in to comment.