Refactor(eos_designs,eos_cli_config_gen): Default validation_mode to …

…error and remove conversion_mode (aristanetworks#4327)

Co-authored-by: Mahesh Kumar <[email protected]>

ansible_collections/arista/avd/docs/contribution/eos_designs_internal_notes.md

@@ -43,8 +43,7 @@ The module is used in `arista.avd.eos_designs` to set facts for devices, which a
 
 ```yaml
   template_output: <true | false | default -> false>
-  conversion_mode: <"warning" | "info" | "debug" | "quiet" | "disabled" | default -> "debug">
-  validation_mode: <"error" | "warning" | "info" | "debug" | "disabled" | default -> "warning">
+  validation_mode: <"error" | "warning" | default -> "error">
   cprofile_file: <Filename for storing cprofile data used to debug performance issues>
 ```
 

ansible_collections/arista/avd/docs/contribution/input-variable-validation.md

@@ -28,75 +28,35 @@ The current implementation supports the automatic conversions listed below.
 | `bool`, `str` | `int` |
 | `int`, `str` | `bool` |
 | `bool`, `int` | `str` |
-| `dict`*, `list`** | `list` |
-
-\* If `primary_key` is set on the `list` schema, conversion from `dict`-of-`dicts` to `list`-of-`dicts` will insert the `primary_key`
-with the value of the outer dictionary key in each `dict`. If `primary_key` is *not* set on the `list` schema, only the input `dict`
-keys are returned as `list` items (any input `dict` values are lost).
-\*\* If `primary_key` is set on the `list` schema, conversion from `list` to `list`-of-`dicts` will insert the `primary_key` with the
-value of the input `list` items in each `dict`.
 
 An example of input variable conversion is `bgp_as`. `bgp_as` is expected as a string (`str`) since 32-bit AS numbers can be
-written with dot-notation such as `"65001.10000"`. Most deployments use 16-bit AS numbers, where the setting `bgp_as: 65001` will be interpreted as an integer in YAML, unless it is enclosed in quotes `bgp_as: "65001"`. In the schema for `bgp_as` the `convert_types` option is configured to `['int']` which means AVD Action Plugins will automatically convert
+written with dot-notation such as `"65001.10000"`. Most deployments use 16-bit AS numbers, where the setting `bgp_as: 65001`
+will be interpreted as an integer in YAML, unless it is enclosed in quotes `bgp_as: "65001"`. In the schema for `bgp_as` the
+`convert_types` option is configured to `['int']` which means AVD Action Plugins will automatically convert
 `bgp_as: 65001` to `bgp_as: "65001"`.
 
-Type conversion is also used for introducing changes to the data models without affecting existing deployments. For example,
-in AVD 4.0, the data models using "Dictionaries with wildcard keys" has been be changed to lists.
-Ex. the old data model:
-
-```yaml
-tenants:
-  mytenant:
-    vrfs:
-      myvrf:
-        ...
-```
-
-has been be changed to:
-
-```yaml
-tenants:
-  - name: mytenant
-    vrfs:
-      - name: myvrf
-        ...
-```
-
-The input data will be converted automatically and validated according to the new model to allow for a gracious introduction of this data-model changes in late 3.x versions.
-
-Type conversion is turned on by default.
-
-!!! CAUTION
-    Although conversion can be disabled, this is **not** recommended, and will most likely lead to issues with AVD or the generated
-    configuration. Data conversion also handles data deprecation and upgrades.
+Data conversion processing also handles deprecation warnings.
 
 ## Validation Options
 
 Schema validation is built in to the central Action plugins used in AVD. Each plugin runs variable type conversion first and then
 performs validation of the converted data.
 
-By default the data conversions are logged with `-v` and data validation issues will trigger warnings - not blocking further processing.
+By default any data validation issues will trigger errors - blocking further processing.
 This behavior can be adjusted by setting the variables described below.
 
-```yaml
-# Conversion Mode for AVD input data conversion | Optional
-# During conversion, messages will generated with information about the host(s) and key(s) which required conversion.
-# "disabled" means that conversion will not run - avoid this since conversion is also handling data deprecation and upgrade.
-# "error" will produce error messages and fail the task.
-# "warning" will produce warning messages.
-# "info" will produce regular log messages.
-# "debug" will produce hidden debug messages viewable with -v.
-# "quiet" will not produce any messages
-avd_data_conversion_mode: < "disabled" | "error" | "warning" | "info" | "debug" | "quiet" | default -> "debug" >
+!!! danger
+    The input variable `avd_data_validation_mode` now has a default value of `error`, and while it can be set to `warning`, this is highly discouraged.
+    All AVD code relies on the validation to ensure valid data, so the code assumes that the data is valid.
 
+    If the validation mode is set to `warning`, execution will continue with invalid data, which can lead to hard-to-read errors or incorrect behavior.
+
+```yaml
 # Validation Mode for AVD input data validation | Optional
 # During validation, messages will generated with information about the host(s) and key(s) which failed validation.
-# "disabled" means that validation will not run.
 # "error" will produce error messages and fail the task.
 # "warning" will produce warning messages.
-# "info" will produce regular log messages.
-# "debug" will produce hidden debug messages viewable with -v.
-avd_data_validation_mode: < "disabled" | "error" | "warning" | "info" | "debug" | default -> "warning" >
+avd_data_validation_mode: < "error" | "warning" | default -> "error" >
 ```
 
 ## Schema Details
@@ -228,13 +188,11 @@ The meta-schema does not allow for other keys to be set in the schema.
 | Key | Type | Required | Default | Value Restrictions | Description |
 | ----| ---- | -------- | ------- | ------------------ | ----------- |
 | <samp>type</samp> | String | True | | Valid Values:<br>- `"list"` | Type of variable using the Python short names for each type.<br>`list` for List |
-| <samp>convert_types</samp> | List, items: String | | | | List of types to auto-convert from.<br>For `list` auto-conversion is supported from `dict` if `primary_key` is set on the list schema |
 | <samp>default</samp> | List | | | | Default value |
 | <samp>items</samp> | Dict | | | | Dictionary describing the schema of each list item. This is a recursive schema, so the value must conform to AVD Schema |
 | <samp>max_length</samp> | Integer | | | | Maximum length |
 | <samp>min_length</samp> | Integer | | | | Minimum length |
 | <samp>primary_key</samp> | String | | | Pattern: `^[a-z][a-z0-9_]*$` | Name of a primary key in a list of dictionaries.<br>The configured key is implicitly required and must have unique values between the list elements |
-| <samp>secondary_key</samp> | String | | | Pattern: `^[a-z][a-z0-9_]*$` | Name of a secondary key, which is used with `convert_types:['dict']` in case of values not being dictionaries |
 | <samp>unique_keys</samp> | List, items: String | | | Item Pattern: `^[a-z][a-z0-9_]*$` | Name of a key in a list of dictionaries.<br>The configured key must have unique values between the list elements.<br>This can also be a variable path using dot-notation like `parent_key.child_key` in case of nested lists of dictionaries. |
 | <samp>display_name</samp> | String | | | Regex Pattern: `"^[^\n]+$"` | Free text display name for forms and documentation (single line) |
 | <samp>description</samp> | String | | | Minimum Length: 1 | Free text description for forms and documentation (multi line) |

ansible_collections/arista/avd/docs/plugins/Modules_and_action_plugins/eos_cli_config_gen.md

@@ -34,8 +34,7 @@ The `arista.avd.eos_cli_config_gen` module is an Ansible Action Plugin providing
 | <samp>generate_device_config</samp> | bool | optional | True |  | Flag to generate the device configuration. |
 | <samp>generate_device_doc</samp> | bool | optional | True |  | Flag to generate the device documentation. |
 | <samp>device_doc_toc</samp> | bool | optional | True |  | Flag to generate the table of content for the device documentation. |
-| <samp>conversion_mode</samp> | str | False | debug | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>quiet</code><br>- <code>disabled</code> | Run data conversion in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34;, &#34;quiet&#34; or &#34;disabled&#34; mode.<br>Conversion will perform type conversion of input variables as defined in the schema.<br>Conversion is intended to help the user to identify minor issues with the input data, while still allowing the data to be validated.<br>During conversion, messages will be generated with information about the host(s) and key(s) which required conversion.<br>conversion_mode:disabled means that conversion will not run.<br>conversion_mode:error will produce error messages and fail the task.<br>conversion_mode:warning will produce warning messages.<br>conversion_mode:info will produce regular log messages.<br>conversion_mode:debug will produce hidden messages viewable with -v.<br>conversion_mode:quiet will not produce any messages. |
-| <samp>validation_mode</samp> | str | False | warning | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>disabled</code> | Run validation in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34; or &#34;disabled&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:disabled means that validation will not run.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages.<br>validation_mode:info will produce regular log messages.<br>validation_mode:debug will produce hidden messages viewable with -v. |
+| <samp>validation_mode</samp> | str | False | error | Valid values:<br>- <code>error</code><br>- <code>warning</code> | Run validation in either &#34;error&#34; or &#34;warning&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages. |
 | <samp>cprofile_file</samp> | str | False | None |  | Filename for storing cprofile data used to debug performance issues.<br>Running cprofile will slow down performance in it self, so only set this while troubleshooting. |
 
 ## Examples

ansible_collections/arista/avd/docs/plugins/Modules_and_action_plugins/eos_designs_facts.md

@@ -36,8 +36,7 @@ The module is used in `arista.avd.eos_designs` to set facts for devices, which a
 | Argument | Type | Required | Default | Value Restrictions | Description |
 | -------- | ---- | -------- | ------- | ------------------ | ----------- |
 | <samp>template_output</samp> | bool | False | None |  | If true, the output data will be run through another jinja2 rendering before returning. This is to resolve any input values with inline jinja using variables/facts set by the input templates. |
-| <samp>conversion_mode</samp> | str | False | debug | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>quiet</code><br>- <code>disabled</code> | Run data conversion in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34;, &#34;quiet&#34; or &#34;disabled&#34; mode.<br>Conversion will perform type conversion of input variables as defined in the schema.<br>Conversion is intended to help the user to identify minor issues with the input data, while still allowing the data to be validated.<br>During conversion, messages will be generated with information about the host(s) and key(s) which required conversion.<br>conversion_mode:disabled means that conversion will not run.<br>conversion_mode:error will produce error messages and fail the task.<br>conversion_mode:warning will produce warning messages.<br>conversion_mode:info will produce regular log messages.<br>conversion_mode:debug will produce hidden messages viewable with -v.<br>conversion_mode:quiet will not produce any messages. |
-| <samp>validation_mode</samp> | str | False | warning | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>disabled</code> | Run validation in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34; or &#34;disabled&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:disabled means that validation will not run.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages.<br>validation_mode:info will produce regular log messages.<br>validation_mode:debug will produce hidden messages viewable with -v. |
+| <samp>validation_mode</samp> | str | False | error | Valid values:<br>- <code>error</code><br>- <code>warning</code> | Run validation in either &#34;error&#34; or &#34;warning&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages. |
 | <samp>cprofile_file</samp> | str | False | None |  | Filename for storing cprofile data used to debug performance issues.<br>Running cprofile will slow down performance in it self, so only set this while troubleshooting. |
 
 ## Examples

ansible_collections/arista/avd/docs/plugins/Modules_and_action_plugins/eos_designs_structured_config.md

@@ -38,8 +38,7 @@ The `arista.avd.eos_designs_structured_config` module is an Ansible Action Plugi
 | <samp>dest</samp> | str | False | None |  | Destination path. If set, the output facts will also be written to this path.<br>Autodetects data format based on file suffix. &#39;.yml&#39;, &#39;.yaml&#39; -&gt; YAML, default -&gt; JSON |
 | <samp>mode</samp> | str | False | None |  | File mode (ex. &#34;0o664&#34;) for dest file. See &#39;ansible.builtin.copy&#39; module for details. |
 | <samp>template_output</samp> | bool | False | None |  | If true, the output data will be run through another jinja2 rendering before returning.<br>This is to resolve any input values with inline jinja using variables/facts set by the input templates. |
-| <samp>conversion_mode</samp> | str | False | debug | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>quiet</code><br>- <code>disabled</code> | Run data conversion in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34;, &#34;quiet&#34; or &#34;disabled&#34; mode.<br>Conversion will perform type conversion of input variables as defined in the schema.<br>Conversion is intended to help the user to identify minor issues with the input data, while still allowing the data to be validated.<br>During conversion, messages will be generated with information about the host(s) and key(s) which required conversion.<br>conversion_mode:disabled means that conversion will not run.<br>conversion_mode:error will produce error messages and fail the task.<br>conversion_mode:warning will produce warning messages.<br>conversion_mode:info will produce regular log messages.<br>conversion_mode:debug will produce hidden messages viewable with -v.<br>conversion_mode:quiet will not produce any messages. |
-| <samp>validation_mode</samp> | str | False | warning | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>disabled</code> | Run validation in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34; or &#34;disabled&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:disabled means that validation will not run.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages.<br>validation_mode:info will produce regular log messages.<br>validation_mode:debug will produce hidden messages viewable with -v. |
+| <samp>validation_mode</samp> | str | False | error | Valid values:<br>- <code>error</code><br>- <code>warning</code> | Run validation in either &#34;error&#34; or &#34;warning&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages. |
 | <samp>cprofile_file</samp> | str | False | None |  | Filename for storing cprofile data used to debug performance issues.<br>Running cprofile will slow down performance in it self, so only set this while troubleshooting. |
 
 ## Examples

ansible_collections/arista/avd/docs/porting-guides/5.x.x.md

@@ -28,6 +28,7 @@ In AVD 4.0.0 and across both `eos_designs` and `eos_cli_config_gen`, *all* "dict
 In AVD 4.x inputs were auto-converted to the new data models.
 
 As of AVD 5.0.0 the automatic conversion from dict-of-dicts has been removed, so it is required to update the input data to the new data models.
+The `avd_data_conversion_mode` key has also been removed.
 
 Errors will be raised for each data model not conforming to the list type when running AVD:
 

ansible_collections/arista/avd/docs/release-notes/5.x.x.md

@@ -46,6 +46,7 @@ In AVD 4.0.0 and across both `eos_designs` and `eos_cli_config_gen`, *all* "dict
 In AVD 4.x inputs were auto-converted to the new data models.
 
 As of AVD 5.0.0 the automatic conversion from dict-of-dicts has been removed, so it is required to update the input data to the new data models.
+The `avd_data_conversion_mode` key has also been removed.
 
 See the [porting guide](../porting-guides/5.x.x.md#data-model-changes-from-dict-of-dicts-to-list-of-dicts) for details.
 
@@ -54,6 +55,12 @@ See the [porting guide](../porting-guides/5.x.x.md#data-model-changes-from-dict-
 In AVD 5.0.0, all `eos_designs` and `eos_cli_config_gen` keys are validated by schemas and execution will stop for any violations.
 If additional custom keys are desired, a key starting with an underscore `_`, will be ignored.
 
+!!! danger
+    The input variable `avd_data_validation_mode` now has a default value of `error`, and while it can be set to `warning`, this is highly discouraged.
+    All AVD code relies on the validation to ensure valid data, so the code assumes that the data is valid.
+
+    If the validation mode is set to `warning`, execution will continue with invalid data, which can lead to hard-to-read errors or incorrect behavior.
+
 ### Breaking or behavioral changes in eos_cli_config_gen
 
 Breaking changes may require modifications to the inventory or playbook. See the [Porting guide for AVD 5.x.x](../porting-guides/5.x.x.md)

ansible_collections/arista/avd/molecule/eos_cli_config_gen/inventory/group_vars/all.yml

@@ -8,6 +8,3 @@ management_interfaces:
     vrf: MGMT
     ip_address: 10.73.255.122/24
     gateway: 10.73.255.2
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "error"

ansible_collections/arista/avd/molecule/eos_cli_config_gen_deprecated_vars/inventory/group_vars/all.yml

@@ -1,6 +1,3 @@
 ---
 root_dir: '{{ playbook_dir }}'
 is_for_test: true
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "info"

ansible_collections/arista/avd/molecule/eos_cli_config_gen_negative_unit_tests/inventory/group_vars/all.yml

@@ -1,5 +1,5 @@
 ---
 root_dir: '{{ playbook_dir }}'
 
-# Only warning for validation (default) to trigger the errors
+# Only warning for validation to trigger the errors
 avd_data_validation_mode: "warning"

ansible_collections/arista/avd/molecule/eos_config_deploy_cvp/molecule.yml

@@ -35,6 +35,5 @@ provisioner:
     - --tags=generate,build
     - --skip-tags=online
     - --extra-vars
-    - "avd_data_conversion_mode=error"
 verifier:
   name: ansible

ansible_collections/arista/avd/molecule/eos_designs-l2ls/inventory/group_vars/all.yml

@@ -1,5 +1,2 @@
 ---
 root_dir: '{{ playbook_dir }}'
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "error"

ansible_collections/arista/avd/molecule/eos_designs-mpls-isis-sr-ldp/inventory/group_vars/all.yml

@@ -1,4 +1 @@
 ---
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "error"

ansible_collections/arista/avd/molecule/eos_designs-twodc-5stage-clos/inventory/group_vars/all.yml

@@ -1,5 +1,2 @@
 ---
 root_dir: '{{ playbook_dir }}'
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "error"

ansible_collections/arista/avd/molecule/eos_designs_deprecated_vars/inventory/group_vars/all.yml

@@ -2,7 +2,4 @@
 root_dir: '{{ playbook_dir }}'
 is_for_test: true
 
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "info"
-
 fabric_name: EOS_DESIGNS_DEPRECATED_VARS

ansible_collections/arista/avd/molecule/eos_designs_negative_unit_tests/inventory/group_vars/all.yml

@@ -1,5 +1,2 @@
 ---
 root_dir: '{{ playbook_dir }}'
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "error"

ansible_collections/arista/avd/molecule/eos_designs_unit_tests/inventory/group_vars/all.yml

@@ -1,5 +1,2 @@
 ---
 root_dir: '{{ playbook_dir }}'
-
-avd_data_validation_mode: "error"
-avd_data_conversion_mode: "error"