Fixed test issues for dualtor topologies. #11921

vivekverma-arista · 2024-03-07T05:55:05Z

Description of PR

Summary: Fixed test issue in dualtor topology.
Fixes # (issue): https://github.com/aristanetworks/sonic-qual.msft/issues/69

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

In some test cases the packets go to unselected ToR in case of active-active links or forwarded by the unselected ToR in case of active-standby links.
Some tests verify queue counters, but this check fails for UC1 in presence of active active links as UC1 always has gRPC traffic flowing through it.

How did you do it?

First issue can be addressed by using the following fixtures in case of active-standby (wherever missing, commit 1) :-
a) toggle_all_simulator_ports_to_rand_selected_tor
b) toggle_all_simulator_ports_to_enum_rand_one_per_hwsku_frontend_host_m

To handle this in case of active-active links new fixtures are introduced (commit 3 & 4)
a) setup_standby_ports_on_rand_unselected_tor
b) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m
c) setup_standby_ports_on_rand_unselected_tor_unconditionally
d) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m_unconditionally
Second issue can be addressed by simply skipping the check for UC1 in presence of active-active links. (commit 2)

How did you verify/test it?

Verified on Arista 7260 device using dualtor/dualtor-aa topology.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Used `toggle_all_simulator_ports_to_enum_rand_one_per_hwsku_frontend_host_m` to make all the links active of the randomly selected ToR.

mssonicbld · 2024-03-07T05:56:13Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/bgp/test_bgp_queue.py:4:1: F401 'tests.common.dualtor.dual_tor_common.active_active_ports' imported but unused tests/bgp/test_bgp_queue.py:32:84: F811 redefinition of unused 'active_active_ports' from line 4 tests/common/dualtor/dual_tor_utils.py:1733:1: E302 expected 2 blank lines, found 1 tests/common/dualtor/dual_tor_utils.py:1741:1: E302 expected 2 blank lines, found 1 tests/common/dualtor/dual_tor_utils.py:1749:1: E302 expected 2 blank lines, found 1 tests/common/dualtor/dual_tor_utils.py:1750:72: F811 redefinition of unused 'active_active_ports' from line 38 tests/common/dualtor/dual_tor_utils.py:1750:91: E231 missing whitespace after ',' tests/common/dualtor/dual_tor_utils.py:1750:121: E501 line too long (133 > 120 characters) tests/common/dualtor/dual_tor_utils.py:1761:1: E302 expected 2 blank lines, found 1 tests/common/dualtor/dual_tor_utils.py:1768:1: E302 expected 2 blank lines, found 1 tests/common/dualtor/dual_tor_utils.py:1769:88: F811 redefinition of unused 'active_active_ports' from line 38 ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

The testcase dualtor/test_tor_ecn.py::test_ecn_during_encap_on_standby is suppose to run on standby ToR however the test is not doing anything to make the randomly selected ToR as standby in case of active-active links as the fixture `toggle_all_simulator_ports_to_rand_unselected_tor` only works for active-standby links. The proposed solution is to move the fixture `setup_active_active_ports` from dualtor/test_tor_ecn.py, rename it to `setup_standby_ports_on_rand_selected_tor` and let both the tests use it.

UC1 will always have gRPC traffic flowing through if active-active links exist (dualtor-aa/dualtor-mixed), therefore, queue counters check for UC1 should be skipped in case of following tests if active-active links exist. 1. bgp/test_bgp_queue.py 2. dualtor/test_tor_ecn.py

The traffic from active-active mux ports is ECMPed twice:first time on the NiC to choose the ToR, second time on the ToR to choose the uplinks. The NiC ECMP is not within the test scope, and we also cannot guarantee that the traffic is evenly distributed among all the uplinks. The proposed fix is to configure active-active ports to work in active-standby mode in case of the following tests :- 1. dhcp_relay/test_dhcp_relay.py 2. dhcp_relay/test_dhcpv6_relay.py 3. drop_packets/test_configurable_drop_counters.py 4. pfcwd/test_pfcwd_function.py 5. route/test_route_flap.py 6. snmp/test_snmp_fdb.py 7. everflow/test_everflow_testbed.py 8. everflow/test_everflow_ipv6.py

tests/bgp/test_bgp_queue.py

tests/common/dualtor/dual_tor_utils.py

…nks." This change is not longer required as it was fixed by: sonic-net#11459 This reverts commit 45cb9ed.

mssonicbld · 2024-03-23T10:08:33Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/qos/test_tunnel_qos_remap.py:245:9: E128 continuation line under-indented for visual indent tests/qos/test_tunnel_qos_remap.py:354:9: E128 continuation line under-indented for visual indent tests/qos/test_tunnel_qos_remap.py:432:9: E128 continuation line under-indented for visual indent flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

lolyu

LGTM

StormLiangMS · 2024-03-29T04:44:22Z

hi @vivekverma-arista could I ask your help to validate these changes with T0/T1? to avoid regression. pls verify with 202305 image also.

mssonicbld · 2024-04-01T05:41:52Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing tests/dhcp_relay/test_dhcpv6_relay.py check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2024-04-01T06:23:14Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing tests/dhcp_relay/test_dhcpv6_relay.py check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

What is the motivation for this PR? In some test cases the packets go to unselected ToR in case of active-active links or forwarded by the unselected ToR in case of active-standby links. Some tests verify queue counters, but this check fails for UC1 in presence of active active links as UC1 always has gRPC traffic flowing through it. How did you do it? First issue can be addressed by using the following fixtures in case of active-standby (wherever missing, commit 1) :- a) toggle_all_simulator_ports_to_rand_selected_tor b) toggle_all_simulator_ports_to_enum_rand_one_per_hwsku_frontend_host_m To handle this in case of active-active links new fixtures are introduced (commit 3 & 4) a) setup_standby_ports_on_rand_unselected_tor b) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m c) setup_standby_ports_on_rand_unselected_tor_unconditionally d) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m_unconditionally Second issue can be addressed by simply skipping the check for UC1 in presence of active-active links. (commit 2) How did you verify/test it? Verified on Arista 7260 device using dualtor/dualtor-aa topology.

mssonicbld · 2024-04-17T12:06:32Z

Cherry-pick PR to 202305: #12488

What is the motivation for this PR? In some test cases the packets go to unselected ToR in case of active-active links or forwarded by the unselected ToR in case of active-standby links. Some tests verify queue counters, but this check fails for UC1 in presence of active active links as UC1 always has gRPC traffic flowing through it. How did you do it? First issue can be addressed by using the following fixtures in case of active-standby (wherever missing, commit 1) :- a) toggle_all_simulator_ports_to_rand_selected_tor b) toggle_all_simulator_ports_to_enum_rand_one_per_hwsku_frontend_host_m To handle this in case of active-active links new fixtures are introduced (commit 3 & 4) a) setup_standby_ports_on_rand_unselected_tor b) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m c) setup_standby_ports_on_rand_unselected_tor_unconditionally d) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m_unconditionally Second issue can be addressed by simply skipping the check for UC1 in presence of active-active links. (commit 2) How did you verify/test it? Verified on Arista 7260 device using dualtor/dualtor-aa topology.

mssonicbld · 2024-04-17T12:06:56Z

Cherry-pick PR to 202311: #12489

What is the motivation for this PR? In some test cases the packets go to unselected ToR in case of active-active links or forwarded by the unselected ToR in case of active-standby links. Some tests verify queue counters, but this check fails for UC1 in presence of active active links as UC1 always has gRPC traffic flowing through it. How did you do it? First issue can be addressed by using the following fixtures in case of active-standby (wherever missing, commit 1) :- a) toggle_all_simulator_ports_to_rand_selected_tor b) toggle_all_simulator_ports_to_enum_rand_one_per_hwsku_frontend_host_m To handle this in case of active-active links new fixtures are introduced (commit 3 & 4) a) setup_standby_ports_on_rand_unselected_tor b) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m c) setup_standby_ports_on_rand_unselected_tor_unconditionally d) setup_standby_ports_on_non_enum_rand_one_per_hwsku_frontend_host_m_unconditionally Second issue can be addressed by simply skipping the check for UC1 in presence of active-active links. (commit 2) How did you verify/test it? Verified on Arista 7260 device using dualtor/dualtor-aa topology.

Testcases fail in drop_packets/test_configurable_drop_counters.py fail in dualtor-aa because some of the traffic goes to the unselected ToR. How did you do it? Similar approach is taken as in #11921 How did you verify/test it? Verified on Arista 7260 and Arista 7050 with both dualtor and dualtor-aa topology.

Testcases fail in drop_packets/test_configurable_drop_counters.py fail in dualtor-aa because some of the traffic goes to the unselected ToR. How did you do it? Similar approach is taken as in sonic-net#11921 How did you verify/test it? Verified on Arista 7260 and Arista 7050 with both dualtor and dualtor-aa topology.

Testcases fail in drop_packets/test_configurable_drop_counters.py fail in dualtor-aa because some of the traffic goes to the unselected ToR. How did you do it? Similar approach is taken as in #11921 How did you verify/test it? Verified on Arista 7260 and Arista 7050 with both dualtor and dualtor-aa topology.

Fixed route/test_route_flap.py for dualtor

804c42d

Used `toggle_all_simulator_ports_to_enum_rand_one_per_hwsku_frontend_host_m` to make all the links active of the randomly selected ToR.

vivekverma-arista requested a review from StormLiangMS as a code owner March 7, 2024 05:55

vivekverma-arista added 3 commits March 6, 2024 22:35

vivekverma-arista force-pushed the dualtor-rand-dut-fix branch from 99975b4 to 3056620 Compare March 7, 2024 07:56

Merge branch 'sonic-net:master' into dualtor-rand-dut-fix

ee752e3

StormLiangMS requested a review from lolyu March 8, 2024 05:58

vivekverma-arista added 2 commits March 10, 2024 19:42

Merge branch 'sonic-net:master' into dualtor-rand-dut-fix

6e9c564

Merge branch 'sonic-net:master' into dualtor-rand-dut-fix

d15a600

lolyu reviewed Mar 14, 2024

View reviewed changes

tests/bgp/test_bgp_queue.py Outdated Show resolved Hide resolved

tests/common/dualtor/dual_tor_utils.py Show resolved Hide resolved

lolyu added Request for 202305 branch Request for 202311 branch labels Mar 14, 2024

vivekverma-arista and others added 5 commits March 23, 2024 14:41

Merge branch 'master' into dualtor-rand-dut-fix

c4505c1

Revert changes that crept in during merge conflict resolution

e35e91b

Revert "Skip queue counters check for UC1 in case of active-acitve li…

1702713

…nks." This change is not longer required as it was fixed by: sonic-net#11459 This reverts commit 45cb9ed.

Address review comments

f38aadc

Fix qos/test_tunnel_qos_remap.py for dualtor-aa

50c9367

vivekverma-arista added 2 commits March 23, 2024 03:13

Fix pylint errors

7d8930b

Removed unnecessary imports.

eb26a85

lolyu approved these changes Mar 26, 2024

View reviewed changes

vivekverma-arista mentioned this pull request Mar 26, 2024

Fix dhcp_relay/test_dhcp_relay.py::test_dhcp_relay_default for dualtor. #12074

Merged

8 tasks

Merge branch 'master' into dualtor-rand-dut-fix

709e2ad

Fix pylint errors

cbbab9e

Merge branch 'master' into dualtor-rand-dut-fix

484b1be

StormLiangMS added the Approved for 202311 branch label Apr 17, 2024

mssonicbld added the Created PR to 202305 branch label Apr 17, 2024

mssonicbld mentioned this pull request Apr 17, 2024

[action] [PR:11921] Fixed test issues for dualtor topologies. #12488

Merged

8 tasks

mssonicbld added the Created PR to 202311 branch label Apr 17, 2024

mssonicbld mentioned this pull request Apr 17, 2024

[action] [PR:11921] Fixed test issues for dualtor topologies. #12489

Merged

8 tasks

mssonicbld added Included in 202311 branch Included in 202305 branch and removed Created PR to 202311 branch Created PR to 202305 branch labels Apr 17, 2024

vivekverma-arista deleted the dualtor-rand-dut-fix branch April 18, 2024 05:59

This was referenced Apr 18, 2024

Fix drop_packets/test_configurable_drop_counters.py for active-active dualtor. #12500

Merged

Fix everflow tests for active-active dualtor. #12506

Merged

Fix qos/test_pfc_pause.py for dualtor-aa #12530

Merged

This was referenced Apr 26, 2024

[action] [PR:12506] Fix everflow tests for active-active dualtor. #12603

Merged

[action] [PR:12506] Fix everflow tests for active-active dualtor. #12604

Merged

mssonicbld mentioned this pull request May 16, 2024

[action] [PR:12500] Fix drop_packets/test_configurable_drop_counters.py for active-active dualtor. #12865

Merged

8 tasks

mssonicbld mentioned this pull request May 16, 2024

[action] [PR:12500] Fix drop_packets/test_configurable_drop_counters.py for active-active dualtor. #12866

Merged

8 tasks

mssonicbld mentioned this pull request May 28, 2024

[action] [PR:12530] Fix qos/test_pfc_pause.py for dualtor-aa #13018

Merged

8 tasks

mssonicbld mentioned this pull request Jun 13, 2024

[action] [PR:12530] Fix qos/test_pfc_pause.py for dualtor-aa #13257

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed test issues for dualtor topologies. #11921

Fixed test issues for dualtor topologies. #11921

vivekverma-arista commented Mar 7, 2024 •

edited

Loading

mssonicbld commented Mar 7, 2024

mssonicbld commented Mar 23, 2024

lolyu left a comment

StormLiangMS commented Mar 29, 2024

mssonicbld commented Apr 1, 2024

mssonicbld commented Apr 1, 2024

mssonicbld commented Apr 17, 2024

mssonicbld commented Apr 17, 2024

Fixed test issues for dualtor topologies. #11921

Fixed test issues for dualtor topologies. #11921

Conversation

vivekverma-arista commented Mar 7, 2024 • edited Loading

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented Mar 7, 2024

mssonicbld commented Mar 23, 2024

lolyu left a comment

Choose a reason for hiding this comment

StormLiangMS commented Mar 29, 2024

mssonicbld commented Apr 1, 2024

mssonicbld commented Apr 1, 2024

mssonicbld commented Apr 17, 2024

mssonicbld commented Apr 17, 2024

vivekverma-arista commented Mar 7, 2024 •

edited

Loading