sonic-installer does not work when there is no database running #10434

stepanblyschak · 2022-04-01T09:13:24Z

Description

sonic-installer and show version commands don't work when there is no database running.
These utilities should be working regardless of database state.

If there is an issue in the image with the database it won't be possible to reinstall the image.

Steps to reproduce the issue:

docker kill docker ps -q
sudo sonic-installer list
show version

Describe the results you received:

admin@r-panther-13:~$ show version
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 559, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 615, in _connect
    raise err
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 603, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/show", line 5, in <module>
    from show.main import cli
  File "/usr/local/lib/python3.7/dist-packages/show/main.py", line 8, in <module>
    import utilities_common.cli as clicommon
  File "/usr/local/lib/python3.7/dist-packages/utilities_common/cli.py", line 183, in <module>
    iface_alias_converter = InterfaceAliasConverter()
  File "/usr/local/lib/python3.7/dist-packages/utilities_common/cli.py", line 121, in __init__
    self.port_dict = multi_asic.get_port_table()
  File "/usr/local/lib/python3.7/dist-packages/sonic_py_common/multi_asic.py", line 285, in get_port_table
    ports = get_port_table_for_asic(ns)
  File "/usr/local/lib/python3.7/dist-packages/sonic_py_common/multi_asic.py", line 299, in get_port_table_for_asic
    config_db = connect_config_db_for_ns(namespace)
  File "/usr/local/lib/python3.7/dist-packages/sonic_py_common/multi_asic.py", line 44, in connect_config_db_for_ns
    config_db.connect()
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 79, in connect
    self.db_connect('CONFIG_DB', wait_for_init, retry_on)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 74, in db_connect
    SonicV2Connector.connect(self, self.db_name, retry_on)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/dbconnector.py", line 268, in connect
    self.dbintf.connect(db_id, db_name, retry_on)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 175, in connect
    self._onetime_connect(db_id, db_name)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 192, in _onetime_connect
    client.config_set('notify-keyspace-events', self.KEYSPACE_EVENTS)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 1243, in config_set
    return self.execute_command('CONFIG SET', name, value)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.

admin@r-panther-13:~$ sudo sonic_installer list
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 559, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 615, in _connect
    raise err
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 603, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sonic_installer", line 5, in <module>
    from sonic_installer.main import sonic_installer
  File "/usr/local/lib/python3.7/dist-packages/sonic_installer/main.py", line 7, in <module>
    import utilities_common.cli as clicommon
  File "/usr/local/lib/python3.7/dist-packages/utilities_common/cli.py", line 183, in <module>
    iface_alias_converter = InterfaceAliasConverter()
  File "/usr/local/lib/python3.7/dist-packages/utilities_common/cli.py", line 121, in __init__
    self.port_dict = multi_asic.get_port_table()
  File "/usr/local/lib/python3.7/dist-packages/sonic_py_common/multi_asic.py", line 285, in get_port_table
    ports = get_port_table_for_asic(ns)
  File "/usr/local/lib/python3.7/dist-packages/sonic_py_common/multi_asic.py", line 299, in get_port_table_for_asic
    config_db = connect_config_db_for_ns(namespace)
  File "/usr/local/lib/python3.7/dist-packages/sonic_py_common/multi_asic.py", line 44, in connect_config_db_for_ns
    config_db.connect()
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 79, in connect
    self.db_connect('CONFIG_DB', wait_for_init, retry_on)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/configdb.py", line 74, in db_connect
    SonicV2Connector.connect(self, self.db_name, retry_on)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/dbconnector.py", line 268, in connect
    self.dbintf.connect(db_id, db_name, retry_on)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 175, in connect
    self._onetime_connect(db_id, db_name)
  File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 192, in _onetime_connect
    client.config_set('notify-keyspace-events', self.KEYSPACE_EVENTS)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 1243, in config_set
    return self.execute_command('CONFIG SET', name, value)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.

Describe the results you expected:

sonic-installer and show version working properly when there is no connection to the database.

Output of `show version`:

SONiC Software Version: SONiC.202012_rc.1-c3b679ee1_Internal
Distribution: Debian 10.12
Kernel: 4.19.0-12-2-amd64
Build commit: c3b679ee1
Build date: Wed Mar 30 16:00:49 UTC 2022
Built by: sw-r2d2-bot@r-build-sonic-ci03-241

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2020T04244
Uptime: 08:57:27 up  8:49,  1 user,  load average: 0.98, 1.31, 1.70

Docker images:
REPOSITORY                    TAG                              IMAGE ID            SIZE
docker-syncd-mlnx             202012_rc.1-c3b679ee1_Internal   3dea8a034a3d        851MB
docker-syncd-mlnx             latest                           3dea8a034a3d        851MB
docker-teamd                  202012_rc.1-c3b679ee1_Internal   74fab50fbffc        373MB
docker-teamd                  latest                           74fab50fbffc        373MB
docker-nat                    202012_rc.1-c3b679ee1_Internal   5f4d39b7fbad        376MB
docker-nat                    latest                           5f4d39b7fbad        376MB
docker-router-advertiser      202012_rc.1-c3b679ee1_Internal   84be2aa68462        362MB
docker-router-advertiser      latest                           84be2aa68462        362MB
docker-platform-monitor       202012_rc.1-c3b679ee1_Internal   5f9b4245e1da        663MB
docker-platform-monitor       latest                           5f9b4245e1da        663MB
docker-database               202012_rc.1-c3b679ee1_Internal   bc95fad661d5        361MB
docker-database               latest                           bc95fad661d5        361MB
docker-lldp                   202012_rc.1-c3b679ee1_Internal   60b9c78e2de2        402MB
docker-lldp                   latest                           60b9c78e2de2        402MB
docker-orchagent              202012_rc.1-c3b679ee1_Internal   d6f5ab0d0c1d        388MB
docker-orchagent              latest                           d6f5ab0d0c1d        388MB
docker-snmp                   202012_rc.1-c3b679ee1_Internal   be05cabd9f63        405MB
docker-snmp                   latest                           be05cabd9f63        405MB
docker-dhcp-relay             202012_rc.1-c3b679ee1_Internal   fc40828e7820        375MB
docker-dhcp-relay             latest                           fc40828e7820        375MB
docker-sonic-telemetry        202012_rc.1-c3b679ee1_Internal   5301401f2127        451MB
docker-sonic-telemetry        latest                           5301401f2127        451MB
docker-sonic-mgmt-framework   202012_rc.1-c3b679ee1_Internal   48ece549e690        687MB
docker-sonic-mgmt-framework   latest                           48ece549e690        687MB
docker-mux                    202012_rc.1-c3b679ee1_Internal   ff233d40c665        414MB
docker-mux                    latest                           ff233d40c665        414MB
docker-fpm-frr                202012_rc.1-c3b679ee1_Internal   7b985b0b3a61        391MB
docker-fpm-frr                latest                           7b985b0b3a61        391MB
docker-sflow                  202012_rc.1-c3b679ee1_Internal   5d6909ba07ff        374MB
docker-sflow                  latest                           5d6909ba07ff        374MB

Output of `show techsupport`:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

The text was updated successfully, but these errors were encountered:

prsunny · 2022-04-13T15:37:07Z

[Issue Triage] This issue is fixed by #10179.

@abdosi to validate and create another issue if required for multi-asic

stepanblyschak · 2022-04-15T08:02:41Z

@prsunny the above mentioned PR is not a fix for this issue

liuh-80 · 2022-05-24T00:26:53Z

sonic installer import utilities_common.cli because it need use following class:
MutuallyExclusiveOption

However the global variable iface_alias_converter in utilities_common.cli will connect to config db when initialize.

But actually this global variable is not necessary, because most place outside this file using this class will create a new instance.

liuh-80 · 2022-05-24T02:28:16Z

PR for this issue: sonic-net/sonic-utilities#2183
Also will be a submodule update PR after this.

…cker not running issue. (#2183) Fix sonic-installer and 'show version' command crash when database docker not running issue. #### Description Global variable utilies_common.cli.iface_alias_converter will connect to config DB when initialize itself. If database docker not running, the initialize code will crash. To fix this issue, change the variable to lazy initialize variable with lazy_object_proxy. #### Motivation and Context Fix this issue: sonic-net/sonic-buildimage#10434 #### How Has This Been Tested? Pass all UT and sonic-buildimage E2E test. #### Additional Information (Optional)

qiluo-msft · 2022-05-30T04:05:29Z

Fixed by sonic-net/sonic-utilities#2183

…cker not running issue. (#2183) Fix sonic-installer and 'show version' command crash when database docker not running issue. #### Description Global variable utilies_common.cli.iface_alias_converter will connect to config DB when initialize itself. If database docker not running, the initialize code will crash. To fix this issue, change the variable to lazy initialize variable with lazy_object_proxy. #### Motivation and Context Fix this issue: sonic-net/sonic-buildimage#10434 #### How Has This Been Tested? Pass all UT and sonic-buildimage E2E test. #### Additional Information (Optional)

stepanblyschak mentioned this issue Apr 1, 2022

[sonic_installer] Use use_unix_socket_path=True in DB connector sonic-net/sonic-utilities#2096

Closed

prsunny added the Triaged this issue has been triaged label Apr 13, 2022

qiluo-msft assigned liuh-80 May 23, 2022

qiluo-msft mentioned this issue May 30, 2022

Fix sonic-installer and 'show version' command crash when database docker not running issue. sonic-net/sonic-utilities#2183

Merged

qiluo-msft closed this as completed May 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sonic-installer does not work when there is no database running #10434

sonic-installer does not work when there is no database running #10434

stepanblyschak commented Apr 1, 2022

prsunny commented Apr 13, 2022

stepanblyschak commented Apr 15, 2022 •

edited

Loading

liuh-80 commented May 24, 2022 •

edited

Loading

liuh-80 commented May 24, 2022

qiluo-msft commented May 30, 2022

sonic-installer does not work when there is no database running #10434

sonic-installer does not work when there is no database running #10434

Comments

stepanblyschak commented Apr 1, 2022

Description

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

prsunny commented Apr 13, 2022

stepanblyschak commented Apr 15, 2022 • edited Loading

liuh-80 commented May 24, 2022 • edited Loading

liuh-80 commented May 24, 2022

qiluo-msft commented May 30, 2022

Output of `show version`:

Output of `show techsupport`:

stepanblyschak commented Apr 15, 2022 •

edited

Loading

liuh-80 commented May 24, 2022 •

edited

Loading