serialize and unserialize large dictionary data (such as result field,
etc.).

serialize and unserialize large dictionary data (such as result field,
etc.).

(opt-in) by default.

escaping the values adds tons of overhead.

using two different approaches for serializing execution / live action
result.

single key with a large value.

all JSON fixture files so they contain at least one key with character
which needs to be escaped.

…st2 into optimize_escaped_dict_fields

field type.

object.

This will provide us better visibility into how long action runner needs
to process the execution completely - this means not just the runner
running the action, but also the action runner container persisting the
result and corresponding objects to the database.

for a dict field.

execution result save times.

header for the binary field value.

This header tells us which serialization format and compression (if any)
is used for a specific field value.

Using a header format gives us more, flexibility, makes it more future
proof (e.g. ability to change the format in the future) and also ability
to implement things such as per-field compression.

related models.

Based on end to end testings, this results in massive speed ups for
workflows which pass larger data sets around.

See #4846 (comment)
for some numbers and details.

purposes already "immutable" and make sure we always write them out to
the database, even on partial dict update.

Also add tests for it.

This dict value tracking allows us to track when a dict item value has
changed and only write the value to the database on existing document /
model update in case it has changed.

This is a very important property since it allows us to implement
efficient partial document updates.

With that change, JSONDictField now also works in exactly the same
manner as existing mongoengine DictField field type.

Also add tests for various edge cases which would fail if value change
tracking was not correctly implemented or working.

around around (both - returning it as a result and also as a next task
context).

directly.

This way we also get better throughput and lower CPU utilization for
rules engine when working with larger trigger instances.

also for the native dict field type.

in dict list items and mark parent dict field as changed if any dict
list item has changed.

attribute, update end_timestamp instead at the very end.

This way execution duration will be more accurately reported.

default.

This should make it easier to infer actual execution run time duration
and state transitions.

now be a bit different, depending on how long it takes to persist each
corresponding object in the database.

Also fix tests to utilize correct dict type for the result.

nightly scheduled basis.

faster (especially visible with packs which have many actions such as
the aws one):

* Utilize ``fast_deepcopy`` for making deep copies of dicts in json
  schema code (that code only works with simple native JSON type so this
  function can be used without any issues).
* Update registrator code to use runner db cache. This means that
  instead of doing N queries where N is number of actions to be
  registered, now we will do only M queries where M is number of unique
  runners actions utilize (in most cases thats < 4).
* Update existing action retrieval code to only retrieve fields we need
  (id, pack, ref). We really only need ID to check if the object already
  exists and perform upsert. Retrieving all the fields we don't use
  is wasteful and slow for actions with many parameters.
* Use C version of the YAML safe loader when loading YAML metadata. C
  version is a lot faster.

…ped_dict_fields

JSON types to utilize fast_deepcopy() instead of copy.deepcopy().

This should result in fast copy times and as such faster secret masking,
etc.

messages by default.

This should make troubleshooting failures a lot easier - before that
change, those log messages would add tons of noise (we load resource
fixtures for each single test) and make actual test failures hard to
troubleshoot.

…ped_dict_fieldsA

Also format new code with black.

benchmarks and it's a testing dependency.

instances.

values since it's not required and may break things by decoding bytes to
string and adding trailing character.

better operational visibility into some steps of the action runner.

…ped_dict_fields

…ped_dict_fields

action runner.

That exception does not represent a fatal error so we should not log
anything.

In fact, it's quite a common and expected scenario that a key doesn't
contain a JSON string.

versions of StackStorm.

This field is populated lazily on model save.

It will allow us to implement more efficient data retrieval in the web
ui and other clients since we will be able to avoid retrieving the whole
result for executions with very large results.
"

execution result.

This endpoint is to be used with webui for executions with large
results.

inside action runner at the end after save.

I was hoping we will be able to avoid one additional serialization, but
sadly we can't if we don't want to massively hack and monkey patch
mongoengine.

And that monkeypatching is not worth it since serialization is fast
enough.

To put things into perspective - takes takes 7ms for 4 MB result which
is nothing compared to other DB operations durations. And for smaller
results it even gets to the sub ms aka nanosecond range.

update_execution() serivce method and don't update end timestamp for
liveaction and execution DB model at the end.

Technically with new perf optimizations code, DB operations are very
fast already and this way we avoid 2 additional queries and save up to
500ms when storing very large executions.

And doing it inside that function also means we can correctly update it
for workflow executions when they finish.

…ped_dict_fields

from a database using string and binary field type.

…st2 into optimize_escaped_dict_fields

Python 3.

formatting execution result for display and orjson when parsing API
response.

This should result in "st2 execution get" and other commands to finish
faster, especially when working with large executions.

For example, locally running st2 execution get on execution with 8 MB
result takes 18 seconds before this change and less than 6 seconds with
this change.

means YAML loading and serialization will be significantly slower.

functions.

iso8601 date strings where possible.

tests-requirements.