Traffic forwarding faliure in voq chassis after config reload

[vganesan-nokia]

Description

In VOQ chassis systems, traffic forwarding across different asics is disrupted permanently after config reload. All required routes and and neighbors are present in kernel.

Steps to reproduce the issue:

1. Configure two eBGP neighbors Nbr1 and Nbr2 in ports from two different asics in different line cards.
2. Redistribute routes of two destination networks Nw1 in Nbr1 and Nw2 in Nbr2 so that hosts from these networks attached to these eBGP neighbors have two way reachability
3. Send/receive traffic to/from a hosts in network Nw1 to/from Nw2.
4. Make sure that there is no traffic loss.
5. Do config reload in one of the line cards.

Describe the results you received:

• After config reload is complete and after the system is stable (swss and syncd dockers are up and running, bgp sessions are established, and routes are learned and programmed in hw), the traffic between the hosts from Nw1 and Nw2 have not recovered.

Describe the results you expected:

• After config reload and after system became stable, the traffic between the hosts from Nw1 and Nw2 should fully recover and there should not be traffic loss.

Additional information you deem important (e.g. issue happens only occasionally):

• The problem is due to mismatch of encap index of the neighbor records of eBGP neighbors in different asics. In voq systems, for successful traffic forwarding, the encap index of neighbor record in all the asics must be same as the encap index allocated by the asic to which the neighbor belongs (Refer VOQ HLD: https://github.com/Azure/SONiC/blob/master/doc/voq/voq_hld.md)
• This problem is fixed by sonic-swss PR [neighorch] VOQ encap index change handling sonic-swss#1729

[vganesan-nokia]

Fixed by PR sonic-net/sonic-swss#1729.