Pack config: Process defaults and datastore values in patternProperties and additionalItems

[cognifloyd]

Building on #5225 allowed pack config to get defaults populated and get datastore values injected when needed under additionalProperties

This PR does the same for patternProperties and additionalItems.

We're were not processing defaults or datastore values in pack config under:

• patternProperties on objects (discovered in Contribute new pack for Cisco ISE StackStorm-Exchange/exchange-incubator#166 (comment))
• additionalItems on arrays (noticed while adding patternProperties support)

This makes sure we are applying defaults on nested items in arrays as well.

TODO:

• tests

[amanda11]

This is great. I've not had chance to check the arrayItems changes, but went through patternProperties. Just raised a couple of minor comments on the patternProperties.

[amanda11]

Thanks for update, the additionalProperties handling is really clear now, and looks good.
I haven't looked at additionalItems, but will try and do that when you've added the extra tests.

Thanks for the refactor.

[cognifloyd]

K. I added a test showing the order of precedence in of properties, patternProperties, and additionalProperties in pack config.

[cognifloyd]

I'm reviewing the schema to make sure I implemented patternProperties correctly.

We use draft 4 (and draft 3's required):

st2/st2common/st2common/util/schema/draft4.json

Line 2 in bb04575

"id": "http://json-schema.org/draft-04/schema#",

st2/st2common/st2common/util/schema/__init__.py

Lines 46 to 50 in 9e336d8

	# https://github.com/json-schema/json-schema/blob/master/draft-04/schema
	# The source material is licensed under the AFL or BSD license.
	# Both draft 4 and custom schema has additionalProperties set to false by default.
	# The custom schema differs from draft 4 with the extension of position, immutable,
	# and draft 3 version of required.

In draft 4, patternProperties creates an all-of set of schemas for each property. Each property gets compared against all patterns. And it must match all of the schemas for the patterns it matches. So, we can't avoid compiling some of the patterns like I thought. 😥
https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00#section-8.3

So, I need to refactor a bit more to check all patterns against all keys not included in properties.

[cognifloyd]

K. Refactored and ready for more reviews

[amanda11]

this looks good to me.
The only thing I thought of was whether it was worth having additional additionalItems tests, e.g. are we covering the case where additionalItems is "true" and additionalIItems is "false".

I've approved though, if you don't think those are needed. As the logic looks good, and other than that tests look great.

Thanks for all the work.

[cognifloyd]

With additionalItems: true, the schema for each of those additional items is effectively {}. So, there are:

• no defaults to interpolate
• no automatically decrypted secrets (because nothing has secret: true unda a {} schema.

But, I suppose there can still be jinja templates without auto-decryption. So, I just added a test for a template that uses |decrypt_kv

[rush-skills]

LGTM Code-wise, didn't get a chance to set up and test.