Update scripts in sonic-buildimage from py-swsssdk to swsscommon #11215

liuh-80 · 2022-06-22T06:31:49Z

Why I did it

Update scripts in sonic-buildimage from py-swsssdk to swsscommon

How I did it

Change code to use swsscommon.

How to verify it

Pass all E2E test case

Which release branch to backport (provide reason below if selected)

Description for the changelog

Update scripts in sonic-buildimage from py-swsssdk to swsscommon

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

liuh-80 · 2022-07-04T02:56:07Z

src/sonic-bgpcfgd/bgpmon/bgpmon.py

        self.db.connect(self.db.STATE_DB, False)
-        client = self.db.get_redis_client(self.db.STATE_DB)
-        self.pipe = client.pipeline()
+        self.pipe = self.db.get_redis_client(self.db.STATE_DB)


Remove pipe because in flush_pipe(self, data) has following code:
if value is None:
# delete case
self.pipe.delete(key)
else:
# Add or Modify case
self.db.hmset(self.pipe.STATE_DB. key, value)

This is because swsscommon not using pyredis, and self.db.get_redis_client(self.db.STATE_DB) will return StrictRedis. so the StrictRedis.pipeline() method does not exist in swss-common, in swss-common get_redis_client will return DBConnector.

Also another reason is the DBConnector only has following 1 method in swsscommon.py:
def hmset(self, multiHash):
return _swsscommon.DBConnector_hmset(self, multiHash)

Also DBConnector using redisContext from lib hredis to handle all redis operation, which doesn't not provide hmemset and delete operation.

liuh-80 · 2022-07-04T03:23:51Z

This pull request introduces 1 alert and fixes 3 when merging d557724 into 1568667 - view on LGTM.com

new alerts:

1 for Module is imported more than once

fixed alerts:

3 for Unused import

Fixed.

lgtm-com · 2022-07-04T03:56:45Z

This pull request fixes 3 alerts when merging ddf1c0b9b5f32b906d7be3bfda081c5a30fb24e6 into 1568667 - view on LGTM.com

fixed alerts:

3 for Unused import

lgtm-com · 2022-07-04T06:49:08Z

This pull request fixes 3 alerts when merging cde36331c6869fd5431fcf1895d6b6576bbc2004 into 8fb534e - view on LGTM.com

fixed alerts:

3 for Unused import

liuh-80 · 2022-07-05T07:22:59Z

src/sonic-py-common/sonic_py_common/util.py

+from getopt import getopt
+
+
+# TODO: move to dbsync project.


Can't move to dbsync, because currently snmp agent also using this:
https://github.com/Azure/sonic-snmpagent/blob/5f8f475b54fbe12b959258bb5567f71577aff0ea/src/sonic_ax_impl/__main__.py

qiluo-msft · 2022-07-06T00:41:22Z

led_control.py

We can move all the changes in platform/ and device/ paths into another PR. Those changes need to be tested with proper hardware, reviewed and approved by platform vendors.

Other platform-independent changes are relative easy to test.

Refers to: device/accton/x86_64-accton_minipack-r0/plugins/led_control.py:1 in 1bcf1fd. [](commit_id = 1bcf1fd7cd9aa83264e4c99182c0dc5287ce2f3d, deletion_comment = False)

qiluo-msft · 2022-07-06T00:43:53Z

src/sonic-bgpcfgd/bgpmon/bgpmon.py

@@ -106,11 +104,10 @@ def flush_pipe(self, data):
        for key, value in data.items():
            if value is None:
                # delete case
-                self.pipe.delete(key)
+                self.db.delete(self.db.STATE_DB, key)


db

Pipeline is more efficient on a batch of similar operations.

Not an exact replacement of redis-py pipe, but you may check if RedisPipeline satisfy the usage here.

The RedisPipeline not have delete and hmset methods, to use RedisPipeline we need build a new RedisCommand command and run the command with RedisPipeline. that basically rewrite the delete and hmset method in DBConnector:

https://github.com/Azure/sonic-swss-common/blob/15c0f729648ee9f26ac9aea767d5712db34ca298/common/dbconnector.cpp#L862

delete should be straightforward. We need more discussion on hmset

I will check if we can improve current hmset without using lua script.

I found the hmset here actually using following API which is not lua script based:

template
void DBConnector::hmset(const std::string &key, InputIterator start, InputIterator stop)
{
RedisCommand shmset;
shmset.formatHMSET(key, start, stop);
RedisReply r(this, shmset, REDIS_REPLY_STATUS);
}

So maybe we can finish this PR first, and improve the lua based hmset with another PR.

Could you postpone this change to another future PR? we can merge the other files in this PR first.

Fixed, change in this file reverted.

qiluo-msft · 2022-07-06T00:46:23Z

src/sonic-py-common/sonic_py_common/port_util.py

@@ -0,0 +1,196 @@
+"""
+Bridge/Port mapping utility library.


If these files are simply moved from another repo, let's separate them into a standalone PR, not to mix with other changes not of the same nature.

Fixed, create a new PR here: #11347
Also, will update this PR to only contains sonic script change, and split vender HW related change to different PRs.

liuh-80 · 2022-07-06T02:11:49Z

led_control.py

We can move all the changes in platform/ and device/ paths into another PR. Those changes need to be tested with proper hardware, reviewed and approved by platform vendors.

Other platform-independent changes are relative easy to test.

Refers to: device/accton/x86_64-accton_minipack-r0/plugins/led_control.py:1 in 1bcf1fd. [](commit_id = 1bcf1fd, deletion_comment = False)

All hardware related change move to following PRs for different vender:
[Accton] Remove unused swsssdk import from accton device code by liuh-80 · Pull Request #11348 · Azure/sonic-buildimage (github.com)
[AlphaNetworks] Replace swsssdk with swsscommon in alphanetworks devices by liuh-80 · Pull Request #11349 · Azure/sonic-buildimage (github.com)
[Centec] Replace swsssdk with swsscommon in centec devices. by liuh-80 · Pull Request #11350 · Azure/sonic-buildimage (github.com)

liuh-80 · 2022-07-06T04:16:15Z

dockers/docker-sflow/port_index_mapper.py

@@ -5,7 +5,7 @@
 import traceback
 from sonic_py_common.logger import Logger
 from socket import if_nametoindex
-from swsssdk import SonicV2Connector, port_util
+from sonic_py_common import port_util


The port_util change split to another PR, this PR need wait for another PR merge first:
#11347

Related work items: sonic-net#284, sonic-net#288, sonic-net#624, sonic-net#630, sonic-net#640, sonic-net#2164, sonic-net#2206, sonic-net#2207, sonic-net#2233, sonic-net#2234, sonic-net#2238, sonic-net#8857, sonic-net#10034, sonic-net#10519, sonic-net#10685, sonic-net#10711, sonic-net#10987, sonic-net#10990, sonic-net#11047, sonic-net#11070, sonic-net#11117, sonic-net#11186, sonic-net#11207, sonic-net#11213, sonic-net#11215, sonic-net#11220, sonic-net#11221, sonic-net#11257, sonic-net#11291, sonic-net#11298, sonic-net#11301, sonic-net#11326, sonic-net#11333, sonic-net#11335, sonic-net#11341, sonic-net#11344, sonic-net#11347, sonic-net#11359, sonic-net#11366, sonic-net#11368, sonic-net#11370, sonic-net#11372, sonic-net#11375, sonic-net#11385, sonic-net#11386, sonic-net#11394, sonic-net#11397, sonic-net#11401, sonic-net#11402, sonic-net#11403, sonic-net#11405, sonic-net#11414

This comment was marked as duplicate.

Sign in to view

This comment was marked as outdated.

Sign in to view

This comment was marked as duplicate.

Sign in to view

liuh-80 commented Jul 4, 2022

View reviewed changes

This comment was marked as duplicate.

Sign in to view

liuh-80 force-pushed the dev/liuh/replace-swsssdk-in-scripts branch from 4e8ea6a to cde3633 Compare July 4, 2022 06:26

liuh-80 changed the title ~~[WIP] update scripts in sonic-buildimage from py-swsssdk to swsscommon~~ Update scripts in sonic-buildimage from py-swsssdk to swsscommon Jul 5, 2022

liuh-80 marked this pull request as ready for review July 5, 2022 01:09

liuh-80 requested review from StormLiangMS, a team, qiluo-msft and lguohan as code owners July 5, 2022 01:09

This was referenced Jul 5, 2022

Replace pyswsssdk with swsscommon sonic-net/sonic-utilities#2251

Merged

Replace swsssdk with swsscommon sonic-net/sonic-snmpagent#263

Merged

Replace py-swsssdk with swsscommon sonic-net/sonic-mgmt#5926

Merged

liuh-80 commented Jul 5, 2022

View reviewed changes

This comment was marked as duplicate.

Sign in to view

qiluo-msft reviewed Jul 6, 2022

View reviewed changes

liuh-80 force-pushed the dev/liuh/replace-swsssdk-in-scripts branch from 1bcf1fd to 659e24f Compare July 6, 2022 02:14

liuh-80 commented Jul 6, 2022

View reviewed changes

Replace swsssdk with swsscommon

4089311

liuh-80 force-pushed the dev/liuh/replace-swsssdk-in-scripts branch from 659e24f to 4089311 Compare July 8, 2022 02:18

qiluo-msft approved these changes Jul 8, 2022

View reviewed changes

liuh-80 merged commit a9b7a1f into sonic-net:master Jul 11, 2022

vaibhavhd mentioned this pull request Aug 1, 2022

[master] Errors in docker database supervisord init - flush_unused_database failed #11597

Closed

anamehra mentioned this pull request Sep 14, 2022

swsscommon Error in docker database supervisord init - flush_unused_database failed #12050

Closed

liuh-80 deleted the dev/liuh/replace-swsssdk-in-scripts branch September 15, 2022 00:03

mhampasagar mentioned this pull request Jan 20, 2023

When configure route by frr mgmt-frmework, there is a error log output of frrcfgd.py when init switch #13109

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update scripts in sonic-buildimage from py-swsssdk to swsscommon #11215

Update scripts in sonic-buildimage from py-swsssdk to swsscommon #11215

liuh-80 commented Jun 22, 2022

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

liuh-80 Jul 4, 2022 •

edited

Loading

This comment was marked as duplicate.

liuh-80 commented Jul 4, 2022

lgtm-com bot commented Jul 4, 2022

lgtm-com bot commented Jul 4, 2022

liuh-80 Jul 5, 2022

This comment was marked as duplicate.

qiluo-msft commented Jul 6, 2022

led_control.py

qiluo-msft Jul 6, 2022

qiluo-msft Jul 6, 2022

liuh-80 Jul 6, 2022

qiluo-msft Jul 6, 2022

liuh-80 Jul 7, 2022

liuh-80 Jul 7, 2022

qiluo-msft Jul 7, 2022

liuh-80 Jul 8, 2022

qiluo-msft Jul 6, 2022

liuh-80 Jul 6, 2022

liuh-80 commented Jul 6, 2022

led_control.py

liuh-80 Jul 6, 2022

Update scripts in sonic-buildimage from py-swsssdk to swsscommon #11215

Update scripts in sonic-buildimage from py-swsssdk to swsscommon #11215

Conversation

liuh-80 commented Jun 22, 2022

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

liuh-80 Jul 4, 2022 • edited Loading

Choose a reason for hiding this comment

This comment was marked as duplicate.

liuh-80 commented Jul 4, 2022

lgtm-com bot commented Jul 4, 2022

lgtm-com bot commented Jul 4, 2022

Choose a reason for hiding this comment

This comment was marked as duplicate.

qiluo-msft commented Jul 6, 2022

led_control.py

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuh-80 commented Jul 6, 2022

led_control.py

Choose a reason for hiding this comment

liuh-80 Jul 4, 2022 •

edited

Loading