Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat(eos_validate_state): Add support for ANTA v1.0.0 #4123

Merged
merged 25 commits into from
Jun 20, 2024
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
2c81dc9
Feat(eos_validate_state): Add support for ANTA v1.0.0
carl-baillargeon Jun 17, 2024
cedbe6e
Fix CI
carl-baillargeon Jun 17, 2024
fb18dea
Remove more preview stuff
carl-baillargeon Jun 17, 2024
bf1c0f4
Remove cv_server.yml hostvars
carl-baillargeon Jun 18, 2024
d57f89d
Update anta_integration.md
carl-baillargeon Jun 18, 2024
c3948a0
Added anta.arista.com
carl-baillargeon Jun 18, 2024
c9bc4c7
documentation improvements
carlbuchmann Jun 18, 2024
2e050c2
update logging levels
carlbuchmann Jun 18, 2024
21a80f3
skipped_tests -> skip_tests
carlbuchmann Jun 18, 2024
12e4869
remove preview from eos_validate_state_reports/runner modules
carlbuchmann Jun 18, 2024
0b726de
Fix link title
carlbuchmann Jun 18, 2024
17c11ea
update link anta.ninja -> anta.arista.com
carlbuchmann Jun 18, 2024
f41dca2
fix link!
carlbuchmann Jun 18, 2024
1ab3d1f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 18, 2024
b52a00d
Fix typo!
carlbuchmann Jun 18, 2024
13071e4
Merge branch 'devel' into feat/anta_v1.0.0
gmuloc Jun 19, 2024
dcd90d4
update skip_tests documentation
carlbuchmann Jun 19, 2024
7e556d0
Merge branch 'devel' into feat/anta_v1.0.0
carlbuchmann Jun 19, 2024
e03517c
Update ansible_collections/arista/avd/roles/eos_validate_state/anta_i…
gmuloc Jun 19, 2024
028f7ee
Merge branch 'devel' into feat/anta_v1.0.0
gmuloc Jun 19, 2024
ecd0ae3
Update hardware tests to follow documentation
carl-baillargeon Jun 19, 2024
ac590e5
Merge branch 'devel' into feat/anta_v1.0.0
gmuloc Jun 19, 2024
27be919
Update molecule and doc
carl-baillargeon Jun 19, 2024
52ea7b2
Merge branch 'devel' into feat/anta_v1.0.0
carlbuchmann Jun 19, 2024
52e211a
Merge branch 'devel' into feat/anta_v1.0.0
gmuloc Jun 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions .github/requirements-ci-anta.txt

This file was deleted.

3 changes: 1 addition & 2 deletions .github/workflows/pull-request-management.yml
Original file line number Diff line number Diff line change
Expand Up @@ -322,8 +322,7 @@ jobs:
molecule_parentdir: 'ansible_collections/arista/avd'
molecule_command: 'test'
molecule_args: '--scenario-name ${{ matrix.avd_scenario }}'
# Temporarily using special requirements to install anta while the feature is in preview.
pip_file: .github/requirements-ci-anta.txt
pip_file: .github/requirements-ci.txt
galaxy_file: "ansible_collections/arista/avd/collections.yml"
ansible: ${{ matrix.ansible_version }}
check_git: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ title: arista.avd.eos_validate_state_reports
!!! note
Always use the FQCN (Fully Qualified Collection Name) `arista.avd.eos_validate_state_reports` when using this plugin.

!!! warning "This module is in **preview** mode"
This module is not guaranteed to have a backwards compatible interface.

Generates validation reports for the eos_validate_state role

## Synopsis
Expand Down Expand Up @@ -52,7 +49,7 @@ The plugin offers the following functionalities:

## See Also

- ANTA website: [https://anta.ninja](https://anta.ninja)<br>Documentation for the ANTA test framework
- ANTA website: [https://anta.arista.com](https://anta.arista.com)<br>Documentation for the ANTA test framework

## Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ title: arista.avd.eos_validate_state_runner
!!! note
Always use the FQCN (Fully Qualified Collection Name) `arista.avd.eos_validate_state_runner` when using this plugin.

!!! warning "This module is in **preview** mode"
This module is not guaranteed to have a backwards compatible interface.

Uses ANTA for eos_validate_state role

## Synopsis
Expand All @@ -40,9 +37,9 @@ The plugin offers the following capabilities:
| <samp>device_catalog_path</samp> | str | optional | None | | The absolute path where the device test catalog will be saved.<br>Required if `save_catalog` is set to `True`. |
| <samp>test_results_dir</samp> | str | optional | None | | The directory where the test results JSON file for each host will be saved. |
| <samp>custom_anta_catalogs_dir</samp> | any | optional | None | | The directory where custom ANTA test catalogs are stored.<br>Files must be named after the device hostname or the Ansible group name and have a `.yml` or `.yaml` extension. |
| <samp>skipped_tests</samp> | list | optional | None | | A list of dictionaries specifying categories and, optionally, tests to skip.<br>Each dictionary must have a key `category` and can optionally include a `tests` key. |
| <samp>skip_tests</samp> | list | optional | None | | A list of dictionaries specifying categories and, optionally, tests to skip.<br>Each dictionary must have a key `category` and can optionally include a `tests` key. |
| <samp>&nbsp;&nbsp;&nbsp;&nbsp;category</samp> | str | optional | None | | The name of an AvdTest category (e.g., `AvdTestHardware`). |
| <samp>&nbsp;&nbsp;&nbsp;&nbsp;tests</samp> | list | optional | None | | An optional list of specific tests in the category to skip (e.g., `VerifyRoutingProtocolModel` in `AvdTestBGP`).<br>If not specified, all tests in the category are considered.<br>For a complete list of available tests, see [link to the test list](https://avd.sh/en/stable/roles/eos_validate_state/ANTA-Preview.html#test-categories). |
| <samp>&nbsp;&nbsp;&nbsp;&nbsp;tests</samp> | list | optional | None | | An optional list of specific tests in the category to skip (e.g., `VerifyRoutingProtocolModel` in `AvdTestBGP`).<br>If not specified, all tests in the category are considered.<br>For a complete list of available tests, see [link to the test list](https://avd.sh/en/stable/roles/eos_validate_state/anta_integration.html#test-categories). |
carlbuchmann marked this conversation as resolved.
Show resolved Hide resolved
| <samp>cprofile_file</samp> | any | optional | None | | The filename for storing cProfile data, useful for debugging performance issues.<br>Be aware that enabling cProfile can affect performance, so use it only for troubleshooting. |

## Notes
Expand All @@ -55,7 +52,7 @@ The plugin offers the following capabilities:

## See Also

- ANTA website: [https://anta.ninja](https://anta.ninja)<br>Documentation for the ANTA test framework
- ANTA website: [https://anta.arista.com](https://anta.arista.com)<br>Documentation for the ANTA test framework

## Examples

Expand All @@ -67,7 +64,7 @@ The plugin offers the following capabilities:
device_catalog_path: "/my_avd_project/intended/test_catalogs/{{ inventory_hostname }}-catalog.yml"
test_results_dir: "/my_avd_project/reports/test_results"
custom_anta_catalogs_dir: "/my_avd_project/custom_anta_catalogs"
skipped_tests:
skip_tests:
- category: AvdTestHardware
- category: AvdTestBGP
tests:
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def run(self, tmp=None, task_vars=None):
logging_level = get_validated_value(
data=self._task.args, key="logging_level", expected_type=str, default_value="WARNING", allowed_values=LOGGING_LEVELS
)
skipped_tests = get_validated_value(data=self._task.args, key="skipped_tests", expected_type=list, default_value=[])
skip_tests = get_validated_value(data=self._task.args, key="skip_tests", expected_type=list, default_value=[])
save_catalog = get_validated_value(data=self._task.args, key="save_catalog", expected_type=bool, default_value=False)
catalog_path = get_validated_path(path_input=self._task.args.get("device_catalog_path"), parent=True) if save_catalog else None
test_results_dir = get_validated_path(path_input=self._task.args.get("test_results_dir"), parent=False)
Expand All @@ -91,7 +91,7 @@ def run(self, tmp=None, task_vars=None):
anta_device=anta_device,
config_manager=config_manager,
logging_level=logging_level,
skipped_tests=skipped_tests,
skip_tests=skip_tests,
ansible_tags=ansible_tags,
save_catalog_name=catalog_path,
custom_anta_catalogs=custom_anta_catalogs,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,6 @@
# Use of this source code is governed by the Apache License 2.0
# that can be found in the LICENSE file.

# NOTE: This is supposed to be deprecated as per
# https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#ansible-metadata-block
# But our doc Jinja2 template renders it as preview which is what we want
ANSIBLE_METADATA = {"metadata_version": "1.0", "status": ["preview"]}

DOCUMENTATION = r"""
---
module: eos_validate_state_reports
Expand Down Expand Up @@ -59,7 +54,7 @@
seealso:
- name: ANTA website
description: Documentation for the ANTA test framework
link: https://anta.ninja
link: https://anta.arista.com
notes:
- Enabling the cProfile feature for performance profiling may impact the plugin's performance, especially in production environments.
- Hosts with `is_deployed` is False are automatically skipped, and no test results are processed for these hosts.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,6 @@
# Use of this source code is governed by the Apache License 2.0
# that can be found in the LICENSE file.

# NOTE: This is supposed to be deprecated as per
# https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#ansible-metadata-block
# But our doc Jinja2 template renders it as preview which is what we want
ANSIBLE_METADATA = {"metadata_version": "1.0", "status": ["preview"]}

DOCUMENTATION = r"""
---
module: eos_validate_state_runner
Expand Down Expand Up @@ -48,7 +43,7 @@
description:
- The directory where custom ANTA test catalogs are stored.
- Files must be named after the device hostname or the Ansible group name and have a `.yml` or `.yaml` extension.
skipped_tests:
skip_tests:
description:
- A list of dictionaries specifying categories and, optionally, tests to skip.
- Each dictionary must have a key `category` and can optionally include a `tests` key.
Expand All @@ -65,15 +60,15 @@
- An optional list of specific tests in the category to skip (e.g., `VerifyRoutingProtocolModel` in `AvdTestBGP`).
- If not specified, all tests in the category are considered.
- For a complete list of available tests,
see [link to the test list](https://avd.sh/en/stable/roles/eos_validate_state/ANTA-Preview.html#test-categories).
see [link to the test list](https://avd.sh/en/stable/roles/eos_validate_state/anta_integration.html#test-categories).
carlbuchmann marked this conversation as resolved.
Show resolved Hide resolved
cprofile_file:
description:
- The filename for storing cProfile data, useful for debugging performance issues.
- Be aware that enabling cProfile can affect performance, so use it only for troubleshooting.
seealso:
- name: ANTA website
description: Documentation for the ANTA test framework
link: https://anta.ninja
link: https://anta.arista.com
notes:
- Enabling the cProfile feature for performance profiling may impact the plugin's performance, especially in production environments.
- The plugin manages the creation of JSON files, which are used for storing test results.
Expand All @@ -92,7 +87,7 @@
device_catalog_path: "/my_avd_project/intended/test_catalogs/{{ inventory_hostname }}-catalog.yml"
test_results_dir: "/my_avd_project/reports/test_results"
custom_anta_catalogs_dir: "/my_avd_project/custom_anta_catalogs"
skipped_tests:
skip_tests:
- category: AvdTestHardware
- category: AvdTestBGP
tests:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,7 @@

logger = getLogger(__name__)

REQUIRED_ANTA_VERSION = "v0.14.0"
"""This is temporary until the ANTA mode is out of preview and the `anta` Python library requirement is added to
the AVD repository's `requirements.txt` file. This constant and the condition below must be removed once the requirement is added."""

try:
from anta import __version__ as anta_version

if anta_version != REQUIRED_ANTA_VERSION:
raise AristaAvdError(message=f"AVD requires 'anta' Python library version {REQUIRED_ANTA_VERSION}, found {anta_version}")
from anta import __DEBUG__
from anta.device import AntaDevice
from anta.logger import anta_log_exception
Expand All @@ -35,20 +27,6 @@
HAS_ANTA = False
# Next line to make ansible-test sanity happy
AntaDevice = object
except TypeError as e:
# Known bug with Python 3.9.7 and Pydantic `conint`, impacting ANTA. Issue: https://github.com/arista-netdevops-community/anta/issues/557
if "Interval() takes no arguments" in str(e):
msg = (
"The ANTA testing framework, utilized in the AVD eos_validate_state role, has identified a compatibility issue with Python 3.9.x. "
"We recommend trying a different Python version; 3.9.13 has been confirmed to work, or consider upgrading to version 3.10 or newer.\n"
"For further assistance or to report your Python version, please visit the AVD and ANTA GitHub repositories:\n"
"https://github.com/aristanetworks/avd/\n"
"https://github.com/arista-netdevops-community/anta"
)
raise AristaAvdError(msg) from e
else:
# If the TypeError is not related to the known bug, raise it to avoid silencing other issues
raise

if TYPE_CHECKING:
from ansible.plugins.connection import ConnectionBase
Expand Down Expand Up @@ -114,14 +92,15 @@ def __rich_repr__(self) -> Generator:
if __DEBUG__:
yield "_connection", connection_vars

async def _collect(self, command: AntaCommand) -> None:
async def _collect(self, command: AntaCommand, *, collection_id: str | None = None) -> None:
"""Collect device command result using Ansible HttpApi connection plugin.

Supports outformat 'json' and 'text' as output structure.

Args:
----
command (AntaCommand): The command to collect.
collection_id (str, optional): This parameter is not used in this implementation. Defaults to None.

If there is an exception while collecting the command, the exception will be propagated
and handled in ANTA. That means ANTA will set the test result to 'error', the play will
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def get_anta_results(
anta_device: AntaDevice,
config_manager: ConfigManager,
logging_level: str,
skipped_tests: list[dict],
skip_tests: list[dict],
ansible_tags: dict | None = None,
save_catalog_name: Path | None = None,
custom_anta_catalogs: list[Path] | None = None,
Expand All @@ -59,9 +59,9 @@ def get_anta_results(
When running in Ansible, the action plugin will pass an AnsibleEOSDevice instance.
config_manager (ConfigManager): The device ConfigManager object containing data to be used by the tests.
logging_level (str): The level at which ANTA should be logging.
skipped_tests (list[dict]): A list of dictionary containing the categories and/or tests to skip.
skip_tests (list[dict]): A list of dictionary containing the categories and/or tests to skip.
ansible_tags (dict): An optional dictionary containing the tags to maintain legacy filtering behavior for
`eos_validate_state`. This is ignored if `skipped_tests` is set.
`eos_validate_state`. This is ignored if `skip_tests` is set.
save_catalog_name (str): When set, the generated catalog is saved to a file using this name.
custom_anta_catalogs (list[Path]): An optional list of custom ANTA catalog files to merge with the generated catalog.
yaml_dumper (Dumper): Dumper to use to dump the ANTA catalog. Default is NoAliasDumper to avoid anchors.
Expand All @@ -79,22 +79,22 @@ def get_anta_results(
# Setup ANTA logging
setup_logging(level=logging_level)

if skipped_tests:
LOGGER.warning("The variable 'skipped_tests' has been set. Ansible tags are ignored for filtering tests.")
if skip_tests:
LOGGER.warning("The variable 'skip_tests' has been set. Ansible tags are ignored for filtering tests.")
# Backward compatibility with legacy eos_validate_state Ansible tags
elif ansible_tags:
run_tags = ansible_tags.get("ansible_run_tags", ())
skip_tags = ansible_tags.get("ansible_skip_tags", ())
# Update the skipped_tests variable according to Ansible tags
skipped_tests = get_skipped_tests_from_tags(run_tags, skip_tags)
# Update the skip_tests variable according to Ansible tags
skip_tests = get_skip_tests_from_tags(run_tags, skip_tags)

device_name = anta_device.name

# Load and merge the custom catalogs for the device if any
custom_catalog = load_custom_catalogs(custom_anta_catalogs) if custom_anta_catalogs else None

# Create the ANTA Catalog object with the appropriate skipped tests if any
tests = generate_tests(config_manager, skipped_tests, custom_catalog)
tests = generate_tests(config_manager, skip_tests, custom_catalog)
anta_catalog = AntaCatalog.from_dict(data=tests) if tests else AntaCatalog()

if save_catalog_name is not None:
Expand Down Expand Up @@ -165,7 +165,7 @@ def dump_to_file(tests: dict, filename: Path, yaml_dumper: Dumper = NoAliasDumpe
dump(tests, fd, Dumper=yaml_dumper)


def get_skipped_tests_from_tags(run_tags: tuple, skip_tags: tuple) -> list[dict]:
def get_skip_tests_from_tags(run_tags: tuple, skip_tags: tuple) -> list[dict]:
"""Get the list of AVD test categories to skip from the Ansible tags.

Args:
Expand Down Expand Up @@ -197,18 +197,18 @@ def get_skipped_tests_from_tags(run_tags: tuple, skip_tags: tuple) -> list[dict]
return result


def generate_tests(config_manager: ConfigManager, skipped_tests: list[dict], custom_catalog: dict | None = None) -> RawCatalogInput:
def generate_tests(config_manager: ConfigManager, skip_tests: list[dict], custom_catalog: dict | None = None) -> RawCatalogInput:
"""Create the test catalog in a dictionary format generated from the AVD test classes.

Test definitions are generated from the AVD structured_config for each AVD test classes and are merged together
with an optional custom_catalog to create the final catalog.

Tests can be skipped from the catalog depending on `skipped_tests`.
Tests can be skipped from the catalog depending on `skip_tests`.

Args:
----
config_manager (ConfigManager): The device ConfigManager object containing data to be used by the tests.
skipped_tests (list[dict]): A list of dictionary containing the categories and/or tests to skip.
skip_tests (list[dict]): A list of dictionary containing the categories and/or tests to skip.
custom_catalog (dict): An optional custom catalog to merge with the generated catalog.

Returns:
Expand All @@ -220,9 +220,9 @@ def generate_tests(config_manager: ConfigManager, skipped_tests: list[dict], cus

for avd_test_class in AVD_TEST_CLASSES:
# Check if the whole class is to be skipped
class_skip_config = get_item(skipped_tests, "category", avd_test_class.__name__)
class_skip_config = get_item(skip_tests, "category", avd_test_class.__name__)
if class_skip_config is not None and not class_skip_config.get("tests"):
msg = f"Skipping all tests of {avd_test_class.__name__} per the `skipped_tests` input variable."
msg = f"Skipping all tests of {avd_test_class.__name__} per the `skip_tests` input variable."
LOGGER.info(msg)
continue

Expand All @@ -231,12 +231,12 @@ def generate_tests(config_manager: ConfigManager, skipped_tests: list[dict], cus
generated_tests = eos_validate_state_module.render()

# Remove the individual tests that are to be skipped
if class_skip_config is not None and (avd_test_class_skipped_tests := class_skip_config.get("tests")) is not None:
msg = f"Skipping the following tests of {avd_test_class.__name__} per the `skipped_tests` input variable: "
msg += ", ".join(avd_test_class_skipped_tests)
if class_skip_config is not None and (avd_test_class_skip_tests := class_skip_config.get("tests")) is not None:
msg = f"Skipping the following tests of {avd_test_class.__name__} per the `skip_tests` input variable: "
msg += ", ".join(avd_test_class_skip_tests)
LOGGER.info(msg)
for anta_tests in generated_tests.values():
anta_tests[:] = [test for test in anta_tests if next(iter(test.keys())) not in avd_test_class_skipped_tests]
anta_tests[:] = [test for test in anta_tests if next(iter(test.keys())) not in avd_test_class_skip_tests]

catalog = merge_catalogs(catalog, generated_tests)

Expand Down
3 changes: 1 addition & 2 deletions ansible_collections/arista/avd/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,5 @@ requests>=2.27.0
PyYAML>=6.0.0
deepmerge>=1.1.0
cryptography>=38.0.4
# No anta requirement until the eos_validate_state integration is out of preview.
# anta>=1.0.0
anta>=1.0.0
aristaproto>=0.1.1
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Please familiarize yourself with the Arista WAN terminology before proceeding:
- Internet-exit for Zscaler is in PREVIEW
- `flow_tracking_settings` is in PREVIEW as the model will change in the next release.
- `eos_validate_state` is being enriched to support new tests for WAN designs.
These new tests are added only in the [ANTA preview](../../../eos_validate_state/ANTA-Preview.md) mode.
These new tests are added only in the [ANTA integration](../../../eos_validate_state/anta_integration.md) mode.

### Known limitations

Expand Down
Loading
Loading