Allow building caches based on the resolver's content

[pondzix]

Current state

Client essentially consists of 2 steps:

1. Lookup schema (or list of schemas) - by resolver.
2. Validate data against resolved schema - by validator.

Some applications (e.g. enrich) use whole package - steps 1) + 2) in order to validate entities of events. There are also applications (e.g. rdb-loader or bigquery-loader) which do not need to validate data. Only step 1) - schema lookup - is the feature used by those components in order to build desired model. Usually it means converting resolved JSON schemas to proper schema-ddl model.

Potential improvements

It's all about caching. There is already cache for lookup in a resolver. In general, JSON schemas returned by resolver are often used by downstream components to perform some costly operation: like data validation or building schema-ddl models. And we want to cache it as well, by building different caches on top of the one from resolver.

Timestamps in resolver

At the moment resolver returns:

• in schema lookup - for given schema key -> schema's content
• in listing schemas - for given combination of vendor, name and model -> matching list of schema keys

Internally, those values before being returned by resolver are cached. It is possible to configure TTL, what means all stored items can eventually expire. If another cache is built on top of the one from resolver, we need a way to inform new cache about any changes in the original resolver's cache content.

Returning timestamp, indicating a moment when value was cached, alongside actual content would do the job. If content in resolver's cache expires:

• Fresh one is fetched from Iglu registry and stored with a new timestamp.
• New timestamp could trigger recalculation when used as a part of a key in downstream cache.

Cache for validator

We can take advantage of changes in resolver and create correlated cache in validator. Currently, for every validation in CirceValidator, Json (circe) schema instance is converted to JsonSchema instance from com.networknt library. This operation may be expensive from CPU point of view, so let's cache it!

The structure of a new cache could look like:

• Key - (schemaKey, timestamp)
• Value - schema evaluation result Either[ValidatorError.InvalidSchema, JsonSchema]

The goal is also to make it backward compatible, therefore separate client and new methods in resolver, mimicking the originals but returning the new data type (with timestamp).

See related caching changes in rdb-loader snowplow/snowplow-rdb-loader#1086