Fix port up/bfd sessions bringup notification delay issue.

[liuh-80]

Fix port up/bfd sessions bringup notification delay issue.

Why I did it

Fix following issue:
sonic-net/sonic-buildimage#19569

Work item tracking

• Microsoft ADO: 29192284

How I did it

Revert change in Consumer::execute(), which introduced by this commit:
9258978#diff-96451cb89f907afccbd39ddadb6d30aa21fe6fbd01b1cbaf6362078b926f1f08

The change in this commit add a while loop:
do
{
std::deque entries;
table->pops(entries);
update_size = addToSync(entries);
} while (update_size != 0);

The addToSync sync method will return the size of entries
Which means, if there are massive routes notification, other high priority notification for example port up notification will blocked until all routes notification been handled.

How to verify it

Pass all UT.
Manually verify issue fixed.

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106
• 202111
• 202205
• 202211
• 202305

Tested branch (Please provide the tested image version)

• SONiC.master-20030.629638-f370e2fa8

Description for the changelog

Fix port up/bfd sessions bringup notification delay issue.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[mlok-nokia]

@liuh-80 I built an image with this change and tested. For the first time boot up after installation on a single linecard, all ports come up in 8 minutes and all 34k routes are also installed. For subsequent reboot a single linecard, it takes about 7 minutes for all linkup and 34k routes installed. It seems this change addresses the issue. We need to do more testing to verify that, includes the OC testing.

[liuh-80]

@bocon13 , we found a performance issue which cause by your PR, can you review this fix?

Performance issue: sonic-net/sonic-buildimage#19569
PR cause performance issue: #1992

[lguohan]

@liuh-80 , we must need some UT to prevent such regression.

[wenyiz2021]

thanks @liuh-80 ! just curious what is the result of Comsumer pop notification once VS pop until size of entry is 0?

[liuh-80]

Comsumer

Will add sonic-swss test case to prevent this issue happen again,

[liuh-80]

thanks @liuh-80 ! just curious what is the result of Comsumer pop notification once VS pop until size of entry is 0?

This will make Consumer pop all notifications belong to current consumer, so higher priority notification will be blocked.

[mssonicbld]

/AzurePipelines run Azure.sonic-swss

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/AzurePipelines run Azure.sonic-swss

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[liuh-80]

thanks @liuh-80 ! just curious what is the result of Comsumer pop notification once VS pop until size of entry is 0?

This will make Consumer pop all notifications belong to current consumer, so higher priority notification will be blocked.

thanks! then does current design in this PR ensure the priority task notify, or it only notify the first come notification?

With this PR, each Consumer will pop less than 128 notifications every time. which means orchagent will check and handle high priority notification more frequently.

[prsunny]

thanks @liuh-80 ! just curious what is the result of Comsumer pop notification once VS pop until size of entry is 0?

This will make Consumer pop all notifications belong to current consumer, so higher priority notification will be blocked.

thanks! then does current design in this PR ensure the priority task notify, or it only notify the first come notification?

With this PR, each Consumer will pop less than 128 notifications every time. which means orchagent will check and handle high priority notification more frequently.

does this mean, bulk-api's (SAI) that orchagent invoke for routes etc will now have a limit of 128 entries

[liuh-80]

thanks @liuh-80 ! just curious what is the result of Comsumer pop notification once VS pop until size of entry is 0?

This will make Consumer pop all notifications belong to current consumer, so higher priority notification will be blocked.

thanks! then does current design in this PR ensure the priority task notify, or it only notify the first come notification?

With this PR, each Consumer will pop less than 128 notifications every time. which means orchagent will check and handle high priority notification more frequently.

does this mean, bulk-api's (SAI) that orchagent invoke for routes etc will now have a limit of 128 entries

Yes, it will have a 128 entries limit.
This is an orchagent design issue.

[liuh-80]

The following tests in pc suite seems to be triggering the crashes. orchagent tries to remove the neighbor and nexthop, either meta layer or SAI is not in sync with orchagent and returns error which causes orchagent to exit. test_po_update_io_no_loss test_po_update

I think it's a timing issue, for example the validation of this PR has lots of test case failed, but after I increase the wait_for_n_keys timeout, many test case passed.

However this change do impact performance, because after this change every doTask() call can only handle 128 entries, so some scenario take longer time.

I'm trying to only change RouteOrch to improve performance.

[saksarav-nokia]

We tried the fix in the PR in the Voq chassis and seeing the following issue
The IMM has two asics and has 2 pot channels in each asic and 2 port members in each port channel.
The ip address is configured on each port channel and bgp is eanbled. The neighbor and routes are learned on these port channel.
In sonic-mgmt pc suite, the test case po-update removes the port members from one of the port channel, removes the ip address configured on that port channel, creates new port channel, adds the same port members to the new port channel, adds the same ip address to the new port channel.
In the remote asic, before all the routes learned on the old port channel are removed by routeOrch, the neighbor and nexthop for the old portchannel are being attempted to be removed. But since the routes are pending, the old nexthop and neighbor are not removed. Then the neighbor and nexthop for the new port channel are being added. If the neighbor is learned on remote system port in remote asic, the nexthop is added with alias as inband port's alias, so the key (ip,alias) is same for both old nexthop and new nexthop. When the new nexthop is added , it calls hasNextHop function to check if the nexthop with (ip-address, alias) as key and since the old nexthop is not removed yet, the hasNextHop returns true, however the assert(!hasNextHop) does n't trigger the crash. So addNextHop function replace the old nexthop with old rif-id with new nexthop with new old rif-id in the nexthop map. Then after all the routes learned on old port channel is removed, the old neighbor and old nexthop are being removed. Sine the old nexthop was replaced with new nexthop, when orchagent tries to delete the old nexthop, it actually deletes the new nexthop from SAI. Then when it tries to remove the old neighbor, SAI returns error since orchagent removed the new nexthop from SAI instead of old nexthop and old neighbor is still referenced by the old nexthop in SAI. So orchagent crashes when SAI returns error.

[liuh-80]

We tried the fix in the PR in the Voq chassis and seeing the following issue The IMM has two asics and has 2 pot channels in each asic and 2 port members in each port channel. The ip address is configured on each port channel and bgp is eanbled. The neighbor and routes are learned on these port channel. In sonic-mgmt pc suite, the test case po-update removes the port members from one of the port channel, removes the ip address configured on that port channel, creates new port channel, adds the same port members to the new port channel, adds the same ip address to the new port channel. In the remote asic, before all the routes learned on the old port channel are removed by routeOrch, the neighbor and nexthop for the old portchannel are being attempted to be removed. But since the routes are pending, the old nexthop and neighbor are not removed. Then the neighbor and nexthop for the new port channel are being added. If the neighbor is learned on remote system port in remote asic, the nexthop is added with alias as inband port's alias, so the key (ip,alias) is same for both old nexthop and new nexthop. When the new nexthop is added , it calls hasNextHop function to check if the nexthop with (ip-address, alias) as key and since the old nexthop is not removed yet, the hasNextHop returns true, however the assert(!hasNextHop) does n't trigger the crash. So addNextHop function replace the old nexthop with old rif-id with new nexthop with new old rif-id in the nexthop map. Then after all the routes learned on old port channel is removed, the old neighbor and old nexthop are being removed. Sine the old nexthop was replaced with new nexthop, when orchagent tries to delete the old nexthop, it actually deletes the new nexthop from SAI. Then when it tries to remove the old neighbor, SAI returns error since orchagent removed the new nexthop from SAI instead of old nexthop and old neighbor is still referenced by the old nexthop in SAI. So orchagent crashes when SAI returns error.

Hi @saksarav-nokia , the code change in this PR does not change any orch logic, so the as my understand this issue is a bug in orchagent and it's already there. is there a plan to fix the issue?

[liuh-80]

If this PR need cherry-pick, following PR also need cherry-pick first:
#3304
#3305

[yejianquan]

@bingwang-ms for viz

[mssonicbld]

Cherry-pick PR to 202405: #3328