Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Smartswitch] Orchagent is crashing when the MGMT_VRF is enabled #19638

Closed
vivekrnv opened this issue Jul 19, 2024 · 6 comments
Closed

[Smartswitch] Orchagent is crashing when the MGMT_VRF is enabled #19638

vivekrnv opened this issue Jul 19, 2024 · 6 comments
Assignees
@liuh-80
Labels
MSFT Triaged this issue has been triaged

Comments

@vivekrnv
Copy link
Contributor

Description

When the MGMT_VRF_CONFIG is present, zmq_bind fails in orchagent and aborts. Seen on smartswitch platforms

Steps to reproduce the issue:

  1. Add the MGMT_VRF config
"MGMT_VRF_CONFIG": {"vrf_global": {
       "mgmtVrfEnabled": "true"}}
  1. Do config reload or reboot

or run the following test test_syslog_config_work_after_reboot

Describe the results you received:

Jul  8 16:09:52.748231 r-smartswitch-03 ERR swss#orchagent: :- mqPollThread: zmq_bind failed on endpoint: tcp://10.210.25.4:8100, zmqerrno: 99

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@vivekrnv
Copy link
Contributor Author

@liuh-80 @Pterosaur @prsunny Please check

@vivekrnv
Copy link
Contributor Author

https://github.com/sonic-net/sonic-buildimage/blob/master/dockers/docker-orchagent/orchagent.sh#L87

@judyjoseph
Copy link
Contributor

@liuh-80 could you take a look at this issue

@judyjoseph judyjoseph added Triaged this issue has been triaged MSFT labels Jul 31, 2024
@liuh-80
Copy link
Contributor

liuh-80 commented Aug 12, 2024

will check and fix the issue soon

@JibinBao JibinBao mentioned this issue Sep 10, 2024
Merged
8 tasks
@liuh-80
Copy link
Contributor

liuh-80 commented Sep 24, 2024 •
edited
Loading

The root cause of this issue is because currently ZMQ not bind to VRF by ZMQ_BINDTODEVICE when VRF enabled:
https://libzmq.readthedocs.io/en/latest/zmq_setsockopt.html
This need code change in sonic-swss-common/sonic-gnmi/sonic-buildimage.

Need 4 PRs to fix this issue:

Repo PR
sonic-swss-common sonic-net/sonic-swss-common#920
sonic-swss sonic-net/sonic-swss#3299
sonic-gnmi sonic-net/sonic-gnmi#290
sonic-buildimage #20345

This was referenced Sep 24, 2024
Merged
Merged
Merged
Merged
qiluo-msft pushed a commit to sonic-net/sonic-swss that referenced this issue Oct 24, 2024
@liuh-80
Add mgmt VRF support. (#3299)
bd945f6 
Add mgmt VRF support.

#### Why I did it
Fix following issue:
sonic-net/sonic-buildimage#19638

### How I did it
Add VRF parameter and bind ZMQ to VRF.

#### How to verify it
Pass all UT.
Manually verify issue fixed.


### Description for the changelog
Add mgmt VRF support.
qiluo-msft pushed a commit that referenced this issue Nov 4, 2024
@liuh-80
Add VRF parameter to orchagent and GNMI start script. (#20345)
0601007 
Add VRF parameter to orchagent and GNMI start script.

Why I did it
Orchagent failed bind to ZMQ when mgmt VRF enabled: #19638

How I did it
Add VRF parameter to orchagent and GNMI start script when mgmt-vrf feature enabled.

How to verify it
Pass all UT.
@liuh-80
Copy link
Contributor

liuh-80 commented Nov 5, 2024

All PR merged, manually verified orchagent not crash after enable vrf, close this bug.

@liuh-80 liuh-80 closed this as completed Nov 5, 2024
rkavitha-hcl pushed a commit to rkavitha-hcl/sonic-buildimage that referenced this issue Nov 5, 2024
@liuh-80 @rkavitha-hcl
Add VRF parameter to orchagent and GNMI start script. (sonic-net#20345)
b92e350 
Add VRF parameter to orchagent and GNMI start script.

Why I did it
Orchagent failed bind to ZMQ when mgmt VRF enabled: sonic-net#19638

How I did it
Add VRF parameter to orchagent and GNMI start script when mgmt-vrf feature enabled.

How to verify it
Pass all UT.
rkavitha-hcl pushed a commit to rkavitha-hcl/sonic-buildimage that referenced this issue Nov 5, 2024
@liuh-80 @rkavitha-hcl
Add VRF parameter to orchagent and GNMI start script. (sonic-net#20345)
d14c4fa 
Add VRF parameter to orchagent and GNMI start script.

Why I did it
Orchagent failed bind to ZMQ when mgmt VRF enabled: sonic-net#19638

How I did it
Add VRF parameter to orchagent and GNMI start script when mgmt-vrf feature enabled.

How to verify it
Pass all UT.
stepanblyschak pushed a commit to stepanblyschak/sonic-swss that referenced this issue Nov 13, 2024
@liuh-80 @stepanblyschak
Add mgmt VRF support. (sonic-net#3299)
008b8dd 
Add mgmt VRF support.

#### Why I did it
Fix following issue:
sonic-net/sonic-buildimage#19638

### How I did it
Add VRF parameter and bind ZMQ to VRF.

#### How to verify it
Pass all UT.
Manually verify issue fixed.


### Description for the changelog
Add mgmt VRF support.
rkavitha-hcl pushed a commit to rkavitha-hcl/sonic-buildimage that referenced this issue Nov 15, 2024
@liuh-80
Add VRF parameter to orchagent and GNMI start script. (sonic-net#20345)
a661032 
Add VRF parameter to orchagent and GNMI start script.

Why I did it
Orchagent failed bind to ZMQ when mgmt VRF enabled: sonic-net#19638

How I did it
Add VRF parameter to orchagent and GNMI start script when mgmt-vrf feature enabled.

How to verify it
Pass all UT.
aidan-gallagher pushed a commit to aidan-gallagher/sonic-buildimage that referenced this issue Nov 16, 2024
@liuh-80
Add VRF parameter to orchagent and GNMI start script. (sonic-net#20345)
afc8156 
Add VRF parameter to orchagent and GNMI start script.

Why I did it
Orchagent failed bind to ZMQ when mgmt VRF enabled: sonic-net#19638

How I did it
Add VRF parameter to orchagent and GNMI start script when mgmt-vrf feature enabled.

How to verify it
Pass all UT.
divyachandralekha pushed a commit to divyachandralekha/sonic-swss that referenced this issue Dec 12, 2024
@liuh-80 @divyachandralekha
Add mgmt VRF support. (sonic-net#3299)
bbdb02e 
Add mgmt VRF support.

#### Why I did it
Fix following issue:
sonic-net/sonic-buildimage#19638

### How I did it
Add VRF parameter and bind ZMQ to VRF.

#### How to verify it
Pass all UT.
Manually verify issue fixed.


### Description for the changelog
Add mgmt VRF support.
divyachandralekha pushed a commit to divyachandralekha/sonic-swss that referenced this issue Dec 12, 2024
@liuh-80 @divyachandralekha
Add mgmt VRF support. (sonic-net#3299)
89393fb 
Add mgmt VRF support.

#### Why I did it
Fix following issue:
sonic-net/sonic-buildimage#19638

### How I did it
Add VRF parameter and bind ZMQ to VRF.

#### How to verify it
Pass all UT.
Manually verify issue fixed.


### Description for the changelog
Add mgmt VRF support.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuh-80 liuh-80

Labels
MSFT Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vivekrnv @judyjoseph @liuh-80