ANTA runner performance issues at high scale

[carl-baillargeon]

Tested ANTA on a fabric of 1040 devices and 2 606 033 tests. ANTA was run outside of the CLI, meaning no CLI tags were used. The catalog consisted of evenly distributed tests across devices, around 75% of the tests had filters (tags) for specific devices.

Used tqdm to monitor iterations per second (it/s) on various loops in the runner and cProfile.

Findings:

1. Creating AntaTestRunner tuples from the tags for device in inventory.devices: --> 12 it/s (3h24m total)
   a. selected_tests.update((t, device) for t in catalog.tests if t.inputs.filters is None or t.inputs.filters.tags is None) --> 860 000 it/s (3-4 seconds per device)
   b. selected_tests.update((t, device) for t in catalog.get_tests_by_tags(device.tags)) --> 320 000 it/s (7-8 seconds per device)

   • addressed by fix(anta): Optimize device to tests creation in ANTA runner #648

2. Creating coroutines for test_definition, device in selected_tests: --> 294 it/s (2h27m total)
   a. Looking at cProfile, most of time is spent in Pydantic, which makes sense since AntaCommand and AntaTemplate are Pydantic models.

   • to be addressed

   Here is a screenshot of the runner cProfile based on 45 000 tests:
   

Other findings:

1. await selected_inventory.connect_inventory() Here we connect to all devices of the inventory even though we might not run tests on these devices depending on the tags from the catalog/inventory.
   • to be addressed