[python3 migration] Added Python3 support for new test cases #4867

OleksandrKozodoi · 2021-12-22T09:20:37Z

Signed-off-by: Oleksandr Kozodoi [email protected]

Description of PR

This PR includes necessary changes from #9277 for the setup of the Python3 virtual environment in the sonic-mgmt docker container.

Summary:

Added Python3 support for lldp test case.
Added Python3 support for dhcp_relay test case.
Added Python3 support for decap test case.
Added Python3 support for fib test case.
Added Python3 support for bgp_bounce test case.

How to use Python3 environment for running test cases?

Connect to the sonic-mgmt container

$ docker exec -ti sonic-mgmt bash

Activate the virtual environment

$ source /var/user/env-python3/bin/activate

Deploy t0/t1 topology
Run test case

py.test --testbed=testbed-t0 --inventory=../ansible/lab --testbed_file=../ansible/testbed.csv --host-pattern=testbed-t0 -- module-path=../ansible/library lldp

py.test --testbed=testbed-t0 --inventory=../ansible/lab --testbed_file=../ansible/testbed.csv --host-pattern=testbed-t0 -- module-path=../ansible/library dhcp_relay

py.test --testbed=testbed-t0 --inventory=../ansible/lab --testbed_file=../ansible/testbed.csv --host-pattern=testbed-t0 -- module-path=../ansible/library decap

py.test --testbed=testbed-t0 --inventory=../ansible/lab --testbed_file=../ansible/testbed.csv --host-pattern=testbed-t0 -- module-path=../ansible/library fib

py.test --testbed=testbed-t0 --inventory=../ansible/lab --testbed_file=../ansible/testbed.csv --host-pattern=testbed-t0 -- module-path=../ansible/library bgp/test_bgp_bounce.py

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911

Approach

What is the motivation for this PR?

Migration of sonic-mgmt codebase from Python 2 to Python 3

How did you do it?

Updated codebase of sonic-mgmt for supporting Python3

How did you verify/test it?

Run test cases. Tests passed.

Any platform specific information?

SONiC Software Version: SONiC.master.60600-dirty-20211221.191419
Distribution: Debian 11.2
Kernel: 5.10.0-8-2-amd64
Build commit: 67e40b553
Build date: Tue Dec 21 19:22:34 UTC 2021
ASIC: barefoot

Supported testbed topology if it's a new test case?

t0, t1

Documentation

globaltrouble · 2021-12-22T09:23:59Z

ansible/library/lldp_facts.py

@@ -258,7 +258,7 @@ def main():

    lldp_data = dict()

-    for intf in lldp_rem_sys.viewkeys():
+    for intf in lldp_rem_sys.keys():


for loop will iter over keys by default. keys method call is redundant.

Done. Thanks

globaltrouble · 2021-12-22T09:27:54Z

tests/common/devices/sonic.py

-                nexthop_ip = ipaddress.ip_address(unicode(m.group(1)))
+                nexthop_ip = ipaddress.ip_address(m.group(1).encode().decode("utf-8"))


Here and later you can use six for porting code from py2 to py3 and wise verse. For example six.text_type to always work with unicode in both py versions and six.binary_type can be used to always work with bytes in both py versions.

Are you sure you need to do sequential call of opposite methods encode and decode? what is the type of m.group(1) and how does it change after conversion?

OleksandrKozodoi · 2022-01-11T08:44:47Z

@yxieca @wangxin Can you take a look? Thank you in advance.

saiarcot895 · 2022-01-11T18:59:31Z

tests/common/errors.py

@@ -26,8 +26,7 @@ def __init__(self, msg, results=None):
        self.results = results

    def _to_string(self):
-        return unicode(u"{}, Ansible Results =>\n{}".format(self.message, dump_ansible_results(self.results)))\
-            .encode('ascii', 'backslashreplace')
+        return "{}, Ansible Results =>\n{}".format(self.message, dump_ansible_results(self.results)).encode().decode("utf-8")


What's the reason for calling encode() followed by decode("utf-8") here?

This approach is used for backward compatibility of code between different versions of Python. Ansible returns different types of dump depending on Python versions.

wangxin · 2022-02-07T10:36:24Z

@OleksandrKozodoi Thanks for the efforts of making the scripts working for python3!
Have you tested the changes using python2? I believe there are more compatibilities issues in the repository. Before all the issues are resolved, we need to stick with python2 for some time. So, all the changes need to be both python2&3 compatible.

OleksandrKozodoi · 2022-02-07T10:51:57Z

@wangxin All scripts were tested using python2 and python3 versions. All necessary dependencies for testing python2&3 compatible will be available in python venv of the sonic-mgmt docker container after merging 9277. These changes allow running tests on different versions in a single sonic-mgmt container. Thanks!

wangxin · 2022-03-01T06:32:38Z

LGTM, @OleksandrKozodoi Can you help resolve the conflicts?

OleksandrKozodoi · 2022-03-03T17:26:20Z

@wangxin I've resolved conflicts and added some minor changes for supporting the original codebase. But should be merged 9277 PR of sonic-buildimage repo for checking compatibility of new changes with python3. Thanks.

Signed-off-by: Oleksandr Kozodoi <[email protected]>

ppikh · 2022-05-03T14:41:06Z

Hi @OleksandrKozodoi

Looks like this commit cause failures: TypeError: () takes exactly 2 arguments (1 given)

Issue happen when run tests from folder: qos/test_qos_sai.py

It's related to file: tests/common/fixtures/ptfhost_utils.py

Example of python traceback:

duthosts = [<MultiAsicSonicHost arc-switch1038>], rand_one_dut_hostname = 'arc-switch1038', ptfhost = <tests.common.devices.ptf.PTFHost object at 0x7f60d0924590>

    @pytest.fixture(scope='class')
    def ptf_portmap_file(duthosts, rand_one_dut_hostname, ptfhost):
        """
            Prepare and copys port map file to PTF host
    
            Args:
                request (Fixture): pytest request object
                duthost (AnsibleHost): Device Under Test (DUT)
                ptfhost (AnsibleHost): Packet Test Framework (PTF)
    
            Returns:
                filename (str): returns the filename copied to PTF host
        """
        duthost = duthosts[rand_one_dut_hostname]
        intfInfo = duthost.show_interface(command = "status")['ansible_facts']['int_status']
        portList = natsorted([port for port in intfInfo if port.startswith('Ethernet')])
        portMapFile = "/tmp/default_interface_to_front_map.ini"
        with open(portMapFile, 'w') as file:
            file.write("# ptf host interface @ switch front port name\n")
            file.writelines(
                map(
                        lambda index, port: "{0}@{1}\n".format(index, port),
>                       enumerate(portList)
                    )
                )
E           TypeError: <lambda>() takes exactly 2 arguments (1 given)

duthost    = <MultiAsicSonicHost arc-switch1038>
duthosts   = [<MultiAsicSonicHost arc-switch1038>]
file       = <closed file '/tmp/default_interface_to_front_map.ini', mode 'w' at 0x7f60d028cd20>
intfInfo   = {'Ethernet0': {'admin_state': u'up', 'alias': u'etp1', 'fec': u'rs', 'name': u'Ethernet0', ...}, 'Ethernet100': {'admi...et104', ...}, 'Ethernet108': {'admin_state': u'up', 'alias': u'etp28', 'fec': u'rs', 'name': u'Ethernet108', ...}, ...}
port       = 'Ethernet68'
portList   = ['Ethernet0', 'Ethernet4', 'Ethernet8', 'Ethernet12', 'Ethernet16', 'Ethernet20', ...]
portMapFile = '/tmp/default_interface_to_front_map.ini'
ptfhost    = <tests.common.devices.ptf.PTFHost object at 0x7f60d0924590>
rand_one_dut_hostname = 'arc-switch1038'

common/fixtures/ptfhost_utils.py:186: TypeError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /root/mars/workspace/sonic-mgmt/tests/common/fixtures/ptfhost_utils.py(186)ptf_portmap_file()
-> enumerate(portList)
(Pdb) portList
[u'Ethernet0', u'Ethernet4', u'Ethernet8', u'Ethernet12', u'Ethernet16', u'Ethernet20', u'Ethernet24', u'Ethernet28', u'Ethernet32', u'Ethernet36', u'Ethernet40', u'Ethernet44', u'Ethernet48', u'Ethernet52', u'Ethernet56', u'Ethernet60', u'Ethernet64', u'Ethernet65', u'Ethernet66', u'Ethernet67', u'Ethernet68', u'Ethernet72', u'Ethernet76', u'Ethernet80', u'Ethernet82', u'Ethernet84', u'Ethernet88', u'Ethernet92', u'Ethernet96', u'Ethernet100', u'Ethernet104', u'Ethernet108', u'Ethernet112', u'Ethernet116', u'Ethernet120', u'Ethernet124']
(Pdb) index
*** NameError: name 'index' is not defined
(Pdb) port
u'Ethernet68'

OleksandrKozodoi · 2022-05-11T10:01:32Z

Hi @OleksandrKozodoi

Looks like this commit cause failures: TypeError: () takes exactly 2 arguments (1 given)

Issue happen when run tests from folder: qos/test_qos_sai.py

It's related to file: tests/common/fixtures/ptfhost_utils.py

Example of python traceback:

duthosts = [<MultiAsicSonicHost arc-switch1038>], rand_one_dut_hostname = 'arc-switch1038', ptfhost = <tests.common.devices.ptf.PTFHost object at 0x7f60d0924590>

    @pytest.fixture(scope='class')
    def ptf_portmap_file(duthosts, rand_one_dut_hostname, ptfhost):
        """
            Prepare and copys port map file to PTF host
    
            Args:
                request (Fixture): pytest request object
                duthost (AnsibleHost): Device Under Test (DUT)
                ptfhost (AnsibleHost): Packet Test Framework (PTF)
    
            Returns:
                filename (str): returns the filename copied to PTF host
        """
        duthost = duthosts[rand_one_dut_hostname]
        intfInfo = duthost.show_interface(command = "status")['ansible_facts']['int_status']
        portList = natsorted([port for port in intfInfo if port.startswith('Ethernet')])
        portMapFile = "/tmp/default_interface_to_front_map.ini"
        with open(portMapFile, 'w') as file:
            file.write("# ptf host interface @ switch front port name\n")
            file.writelines(
                map(
                        lambda index, port: "{0}@{1}\n".format(index, port),
>                       enumerate(portList)
                    )
                )
E           TypeError: <lambda>() takes exactly 2 arguments (1 given)

duthost    = <MultiAsicSonicHost arc-switch1038>
duthosts   = [<MultiAsicSonicHost arc-switch1038>]
file       = <closed file '/tmp/default_interface_to_front_map.ini', mode 'w' at 0x7f60d028cd20>
intfInfo   = {'Ethernet0': {'admin_state': u'up', 'alias': u'etp1', 'fec': u'rs', 'name': u'Ethernet0', ...}, 'Ethernet100': {'admi...et104', ...}, 'Ethernet108': {'admin_state': u'up', 'alias': u'etp28', 'fec': u'rs', 'name': u'Ethernet108', ...}, ...}
port       = 'Ethernet68'
portList   = ['Ethernet0', 'Ethernet4', 'Ethernet8', 'Ethernet12', 'Ethernet16', 'Ethernet20', ...]
portMapFile = '/tmp/default_interface_to_front_map.ini'
ptfhost    = <tests.common.devices.ptf.PTFHost object at 0x7f60d0924590>
rand_one_dut_hostname = 'arc-switch1038'

common/fixtures/ptfhost_utils.py:186: TypeError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /root/mars/workspace/sonic-mgmt/tests/common/fixtures/ptfhost_utils.py(186)ptf_portmap_file()
-> enumerate(portList)
(Pdb) portList
[u'Ethernet0', u'Ethernet4', u'Ethernet8', u'Ethernet12', u'Ethernet16', u'Ethernet20', u'Ethernet24', u'Ethernet28', u'Ethernet32', u'Ethernet36', u'Ethernet40', u'Ethernet44', u'Ethernet48', u'Ethernet52', u'Ethernet56', u'Ethernet60', u'Ethernet64', u'Ethernet65', u'Ethernet66', u'Ethernet67', u'Ethernet68', u'Ethernet72', u'Ethernet76', u'Ethernet80', u'Ethernet82', u'Ethernet84', u'Ethernet88', u'Ethernet92', u'Ethernet96', u'Ethernet100', u'Ethernet104', u'Ethernet108', u'Ethernet112', u'Ethernet116', u'Ethernet120', u'Ethernet124']
(Pdb) index
*** NameError: name 'index' is not defined
(Pdb) port
u'Ethernet68'

Hi @ppikh. #5605 should fix this issue.

What is the motivation for this PR? Fix the issue reported in #4867, failures: TypeError: () takes exactly 2 arguments (1 given) The fix in #5605 still uses lambda which is not easy to read and may have compatibility issue. How did you do it? Use for loop and concentrate string together and append them to a list. How did you verify/test it? run tests/test_qos_sai.py Signed-off-by: Zhaohui Sun <[email protected]>

OleksandrKozodoi requested a review from a team as a code owner December 22, 2021 09:20

globaltrouble reviewed Dec 22, 2021

View reviewed changes

saiarcot895 reviewed Jan 11, 2022

View reviewed changes

wangxin previously approved these changes Mar 1, 2022

View reviewed changes

OleksandrKozodoi force-pushed the update_new_test_cases_for_supporting_python3 branch from f7f8676 to 48b39b8 Compare March 3, 2022 17:07

lizhijianrd dismissed wangxin’s stale review via 48b39b8 March 15, 2022 06:00

OleksandrKozodoi added 4 commits March 31, 2022 04:29

[python3 migration] Added Python3 support for new test cases

1a8a870

Signed-off-by: Oleksandr Kozodoi <[email protected]>

Added changes for lldp_facts module

b64a905

Signed-off-by: Oleksandr Kozodoi <[email protected]>

Added changes for sonic.py module

568c3db

Signed-off-by: Oleksandr Kozodoi <[email protected]>

Minor changes

9252d27

Signed-off-by: Oleksandr Kozodoi <[email protected]>

OleksandrKozodoi force-pushed the update_new_test_cases_for_supporting_python3 branch from 48b39b8 to 9252d27 Compare March 31, 2022 11:38

ZhaohuiS approved these changes Apr 25, 2022

View reviewed changes

ZhaohuiS merged commit 771f7d1 into sonic-net:master Apr 25, 2022

This was referenced Jun 21, 2022

fixed startup of QoS SAI tests #5605

Closed

Fix lambda TypeError in ptfhost_utils #5845

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python3 migration] Added Python3 support for new test cases #4867

[python3 migration] Added Python3 support for new test cases #4867

OleksandrKozodoi commented Dec 22, 2021

globaltrouble Dec 22, 2021

OleksandrKozodoi Dec 23, 2021

globaltrouble Dec 22, 2021 •

edited

Loading

OleksandrKozodoi Dec 23, 2021

OleksandrKozodoi commented Jan 11, 2022

saiarcot895 Jan 11, 2022

OleksandrKozodoi Jan 12, 2022

wangxin commented Feb 7, 2022

OleksandrKozodoi commented Feb 7, 2022

wangxin commented Mar 1, 2022

OleksandrKozodoi commented Mar 3, 2022

ppikh commented May 3, 2022

OleksandrKozodoi commented May 11, 2022

		nexthop_ip = ipaddress.ip_address(unicode(m.group(1)))
		nexthop_ip = ipaddress.ip_address(m.group(1).encode().decode("utf-8"))

[python3 migration] Added Python3 support for new test cases #4867

[python3 migration] Added Python3 support for new test cases #4867

Conversation

OleksandrKozodoi commented Dec 22, 2021

Description of PR

This PR includes necessary changes from #9277 for the setup of the Python3 virtual environment in the sonic-mgmt docker container.

How to use Python3 environment for running test cases?

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

globaltrouble Dec 22, 2021

Choose a reason for hiding this comment

OleksandrKozodoi Dec 23, 2021

Choose a reason for hiding this comment

globaltrouble Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

OleksandrKozodoi Dec 23, 2021

Choose a reason for hiding this comment

OleksandrKozodoi commented Jan 11, 2022

saiarcot895 Jan 11, 2022

Choose a reason for hiding this comment

OleksandrKozodoi Jan 12, 2022

Choose a reason for hiding this comment

wangxin commented Feb 7, 2022

OleksandrKozodoi commented Feb 7, 2022

wangxin commented Mar 1, 2022

OleksandrKozodoi commented Mar 3, 2022

ppikh commented May 3, 2022

OleksandrKozodoi commented May 11, 2022

globaltrouble Dec 22, 2021 •

edited

Loading