[Neighbor]When two different Vlans in different VRFs have the same neighbor, one of them is not learnt

[dgsudharsan]

Description

When two different Vlans in different VRFs have the same neighbor, one of them is not learnt since the other one has route pointing to it.

2024-05-01.15:54:05.589894|ROUTE_TABLE:Vrf_0002:2000:2:2::/64|SET|nexthop:2000:3:2::2|ifname:Vlan3002
2024-05-01.15:54:05.592280|NEIGH_TABLE:Vlan3002:2000:3:2::2|SET|neigh:50:6b:4b:94:fd:00|family:IPv6
2024-05-01.15:54:09.645207|ROUTE_TABLE:Vrf_0003:2000:2:2::/64|SET|nexthop:2000:3:2::2|ifname:Vlan3003
2024-05-01.15:54:09.647685|NEIGH_TABLE:Vlan3003:2000:3:2::2|SET|neigh:50:6b:4b:94:fd:00|family:IPv6

This is introduced by sonic-net/sonic-swss#3049

The PR doesn't account different VRFs and also references and thus this scenario fails. The below error logs will also be seen

May  1 15:54:05.594736 sonic NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 2000:3:2::2, 50:6b:4b:94:fd:00 on Vlan3002
May  1 15:54:05.600840 sonic NOTICE swss#orchagent: :- addNextHop: Created next hop 2000:3:2::2 on Vlan3002
May  1 15:54:09.511537 sonic DEBUG bgp#staticroutebfd: Received message : '('Vrf_0003|2000:2:2::/64', 'SET', (('blackhole', 'false'), ('distance', '0'), ('ifname', ''), ('nexthop', '2000:3:2::2'), ('nexthop-vrf', 'Vrf_0003')))'
May  1 15:54:09.511743 sonic DEBUG bgp#bgpcfgd: Received message : '('Vrf_0003|2000:2:2::/64', 'SET', (('blackhole', 'false'), ('distance', '0'), ('ifname', ''), ('nexthop', '2000:3:2::2'), ('nexthop-vrf', 'Vrf_0003')))'
May  1 15:54:09.511967 sonic DEBUG bgp#bgpcfgd: CONFIG_DB Static route Vrf_0003|2000:2:2::/64 is scheduled for updates. ['ipv6 route 2000:2:2::/64 2000:3:2::2 nexthop-vrf Vrf_0003 vrf Vrf_0003 tag 1']
May  1 15:54:09.648154 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:09.648527 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002
May  1 15:54:09.648839 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:09.649090 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002
May  1 15:54:10.010851 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:10.011143 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002
May  1 15:54:10.513528 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:10.513528 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002
May  1 15:54:11.010894 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:11.011218 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002
May  1 15:54:11.017480 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:11.017757 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002
May  1 15:54:11.018134 sonic NOTICE swss#orchagent: :- addNeighbor: Neighbor 2000:3:2::2 on Vlan3002 already exists, removing before adding new neighbor
May  1 15:54:11.018609 sonic ERR swss#orchagent: :- addNeighbor: Failed to remove neighbor 2000:3:2::2 on Vlan3002

2024-05-01.15:54:05.592563|c|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"2000:3:2::2","rif":"oid:0x600000000095f","switch_id":"oid:0x21000000000000"}|SAI_NEIGHBOR_ENTRY_ATTR_DST_MAC_ADDRESS=50:6B:4B:94:FD:00
2024-05-01.15:54:05.594875|c|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x4000000000967|SAI_NEXT_HOP_ATTR_TYPE=SAI_NEXT_HOP_TYPE_IP|SAI_NEXT_HOP_ATTR_IP=2000:3:2::2|SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID=oid:0x600000000095f
2024-05-01.15:54:05.601535|C|SAI_OBJECT_TYPE_ROUTE_ENTRY||{"dest":"2000:2:2::/64","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000951"}|SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID=oid:0x4000000000967

Steps to reproduce the issue:

1. Create two Vlans in two different VRFs
2. Learn the neighbor on first Vlan. Add a route.
3. Try to add a static route on the second Vrf which will trigger learn.
4. The last one will not get learnt.

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[dgsudharsan]

@theasianpianist @prsunny Can you please prioritize this?

[dgsudharsan]

@theasianpianist Can you please provide ETA for fix?

[theasianpianist]

@dgsudharsan can you provide the specific CLI commands or APPL_DB entries you are using to reproduce this?