Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize cv_facts execution. #147

Closed
titom73 opened this issue Mar 5, 2020 · 0 comments
Closed

Optimize cv_facts execution. #147

titom73 opened this issue Mar 5, 2020 · 0 comments
Assignees
Labels
module: cv_facts Issue related to cv_facts module state: accepted Issue is part of the development roadmap status: testing PR under testing type: enhancement New feature or request
Milestone

Comments

@titom73
Copy link
Contributor

titom73 commented Mar 5, 2020

Issue Type

Module Name

cv_facts

Summary

In a large environment, collecting facts from Cloudvision can take a long time increasing execution time of the overall playbook.

It has been seen in deployment with few hundreds of devices and thousands of configlets.

To reduce execution time, it might be useful to reduce number of API calls sent to Cloudvision.

2 options could be a good source of optimization:

  • One option suggesting by @ankudinov is to use get /configlet/getConfigletsAndAssociatedMappers.do call to collect configlet and association in just one API call instead of current implementation described below:

  • Second approach is to use get_configlets_by_device_id() in current cv_api201x wrapping provisioning/getConfigletsByNetElementId.do?netElementId API call.

def facts_configlets(module, facts):
    facts['configlets'] = []
    configlets = module.client.api.get_configlets()['data']
    # Reduce configlet data to required fields
    for configlet in configlets:
        MODULE_LOGGER.debug('  -> Working on %s', configlet['name'])
        # Get list of devices attached to configlet.
        configlet['devices'] = []
        applied_devices = module.client.api.get_devices_by_configlet(configlet['name'])
        for device in applied_devices['data']:
            configlet['devices'].append(device['hostName'])

        # Get list of containers attached to configlet.
        configlet['containers'] = []
        applied_containers = module.client.api.get_containers_by_configlet(configlet['name'])
        for container in applied_containers['data']:
            configlet['containers'].append(container['containerName'])

        # Add configlet to facts list
        facts['configlets'].append(configlet)

    return facts
@titom73 titom73 added type: enhancement New feature or request module: cv_facts Issue related to cv_facts module status: Triage New incoming issue labels Mar 5, 2020
@titom73 titom73 added this to the v1.1 milestone Mar 7, 2020
@titom73 titom73 added state: accepted Issue is part of the development roadmap and removed status: Triage New incoming issue labels Mar 10, 2020
titom73 added a commit that referenced this issue Mar 18, 2020
Rewrite def facts_configlets to use get_configlets_and_mappers function
and reduce number of API call.

No change in inputs and outputs.

Test cases:
-----------
- Manual cv_facts tasks
- AVD integration

Execution time enhacement:
--------------------------
- Original implementation: 6 sec
- New version: 0.3 sec

test setup:
------------
8 devices / 16 configlets / 10 containers

Execution logs:
---------------
2020-03-18 18:26:09,700 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:269) - Collecting facts v2
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:276) - Devices part of facts, using this version
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:283) - Containers part of facts, using this version
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet leaf02-baseline
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet RADIUS_LAB
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF2B
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet Builder_LEAFs
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet spine01-baseline
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-SPINE2
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF1B
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-SPINE1
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet GLOBAL-ALIASES
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-L2LEAF1A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet leaf01-baseline
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet veos-01
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-L2LEAF2A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF2A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet SYS_TelemetryBuilderV3
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF1A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet data.device.leaf01
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet template.interfaces
2020-03-18 18:26:10,020 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:317) - All configlets facts collected
@titom73 titom73 added the status: in-progress Currently under investigation or implementation label Mar 18, 2020
@titom73 titom73 self-assigned this Mar 18, 2020
titom73 added a commit that referenced this issue Mar 18, 2020
Rewrite def facts_configlets to use get_configlets_and_mappers function
and reduce number of API call.

No change in inputs and outputs.

Test cases:
-----------
- Manual cv_facts tasks
- AVD integration

Execution time enhacement:
--------------------------
- Original implementation: 6 sec
- New version: 0.3 sec

test setup:
------------
8 devices / 16 configlets / 10 containers

Execution logs:
---------------
2020-03-18 18:26:09,700 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:269) - Collecting facts v2
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:276) - Devices part of facts, using this version
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:283) - Containers part of facts, using this version
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet leaf02-baseline
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet RADIUS_LAB
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF2B
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet Builder_LEAFs
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet spine01-baseline
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-SPINE2
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF1B
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-SPINE1
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet GLOBAL-ALIASES
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-L2LEAF1A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet leaf01-baseline
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet veos-01
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-L2LEAF2A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF2A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet SYS_TelemetryBuilderV3
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF1A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet data.device.leaf01
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet template.interfaces
2020-03-18 18:26:10,020 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:317) - All configlets facts collected
titom73 added a commit that referenced this issue Mar 18, 2020
Rewrite def facts_configlets to use get_configlets_and_mappers function
and reduce number of API call.

No change in inputs and outputs.

Test cases:
-----------
- Manual cv_facts tasks
- AVD integration

Execution time enhacement:
--------------------------
- Original implementation: 6 sec
- New version: 0.3 sec

test setup:
------------
8 devices / 16 configlets / 10 containers

Execution logs:
---------------
2020-03-18 18:26:09,700 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:269) - Collecting facts v2
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:276) - Devices part of facts, using this version
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:283) - Containers part of facts, using this version
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet leaf02-baseline
2020-03-18 18:26:10,017 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet RADIUS_LAB
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF2B
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet Builder_LEAFs
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet spine01-baseline
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-SPINE2
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF1B
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-SPINE1
2020-03-18 18:26:10,018 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet GLOBAL-ALIASES
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-L2LEAF1A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet leaf01-baseline
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet veos-01
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-L2LEAF2A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF2A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet SYS_TelemetryBuilderV3
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet DC1-AVD_DC1-LEAF1A
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet data.device.leaf01
2020-03-18 18:26:10,019 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:296) - building list of mapping with devices and containers for configlet template.interfaces
2020-03-18 18:26:10,020 - arista.cvp.cv_facts: INFO - func: facts_configlets (L:317) - All configlets facts collected
@titom73 titom73 added status: To be merged Issue solved in a volatile branch and removed status: in-progress Currently under investigation or implementation labels Mar 30, 2020
titom73 added a commit that referenced this issue Apr 7, 2020
@titom73 titom73 added status: testing PR under testing and removed status: To be merged Issue solved in a volatile branch labels Apr 7, 2020
@titom73 titom73 closed this as completed Apr 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module: cv_facts Issue related to cv_facts module state: accepted Issue is part of the development roadmap status: testing PR under testing type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant