Dynamic BGP_BBR is not working as expected #5929

wangxin · 2020-11-16T08:15:04Z

Description

Issue1: When BBR is enabled in /etc/sonic/constants.yml, BBR cannot be disabled by following the instructions in #5626

Issue2: When BBR is disabled in /etc/sonic/constants.yml (manually updated it). BGP configuration lines “neighbor PEER_V4 allowas-in 1” and “ neighbor PEER_V6 allowas-in 1” are not added after enabled BBR by following the instructions in #5626

Steps to reproduce the issue:

Details of Issue1:

BBR is enabled by default.

admin@vlab-03:~$ cat /etc/sonic/constants.yml | grep bbr -A 1
    bbr:
      enabled: true
--
        bbr:
          PEER_V4:

admin@vlab-03:~$ redis-cli -n 4 HGET "BGP_BBR|all" "status"
(nil)
admin@vlab-03:~$ vtysh -c 'show run' | grep allow
  neighbor PEER_V4 allowas-in 1
  neighbor PEER_V6 allowas-in 1

Create file /tmp/disable_bbr.json:

admin@vlab-03:~$ cat /tmp/disable_bbr.json 
{
    "BGP_BBR": {
        "all": {
            "status": "disabled"
        }
    }
}

Disable BBR:

admin@vlab-03:~$ sonic-cfggen -j /tmp/disable_bbr.json -w
admin@vlab-03:~$ redis-cli -n 4 HGET "BGP_BBR|all" "status"
"disabled"
admin@vlab-03:~$ vtysh -c 'show run' | grep allow
admin@vlab-03:~$

Announce route to the first T0 VM. The route has DUT asn (65100) once in aspath. ASN 64101 is a dummy ASN.

johnar@86171a92b545:~/code/sonic-mgmt/tests$ python
Python 2.7.12 (default, Apr 15 2020, 17:07:12)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> url='http://10.250.0.106:5008'
>>> announce={'commands': 'announce route 172.16.10.0/24 next-hop 10.0.0.32 as-path [ 65100 64101 ]'}
>>> requests.post(url, announce)
<Response [200]>

ARISTA01T0#show ip bgp | grep 172.16.10.0/24
 * >     172.16.10.0/24         10.0.0.32             0       100     0       65100 64101 i

Check the route on DUT. The route is accepted by DUT. This is unexpected because BBR is disabled.

admin@vlab-03:~$ vtysh -c 'show bgp ipv4' | grep 172.16.10.0/24
*> 172.16.10.0/24   10.0.0.33                              0 64001 65100 64101 i

Withdraw the announced routes

>>> withdraw={'commands': 'withdraw route 172.16.10.0/24 next-hop 10.0.0.32'}
>>> requests.post(url, withdraw)

ARISTA01T0#show ip bgp | grep 172.16.10.0/24
ARISTA01T0#

Details of issue2:

Set bbr to disabled in /etc/sonic/constants.yml.

admin@vlab-03:~$ cat /etc/sonic/constants.yml | grep bbr -A 1
    bbr:
      enabled: false
--
        bbr:
          PEER_V4:

Run ‘sudo config reload -y’.
Check BBR configuration. BBR is disabled.

admin@vlab-03:~$ redis-cli -n 4 HGET "BGP_BBR|all" "status"
"disabled"
admin@vlab-03:~$ vtysh -c 'show run' | grep allow
admin@vlab-03:~$

Try to enable BBR:

admin@vlab-03:~$ cat /tmp/enable_bbr.json 
{
    "BGP_BBR": {
        "all": {
            "status": "enabled"
        }
    }
} 
admin@vlab-03:~$ sonic-cfggen -j /tmp/enable_bbr.json -w
admin@vlab-03:~$ redis-cli -n 4 HGET "BGP_BBR|all" "status"
"enabled"
admin@vlab-03:~$ vtysh -c 'show run' | grep allow
admin@vlab-03:~$

Configuration lines neighbor PEER_V4 allowas-in 1 and neighbor PEER_V4 allowas-in 1 are not observed in vtysh -c ‘show run’. This is unexpected.
Announce route to the first T0 VM. The route has DUT asn (65100) once in aspath. ASN 64101 is a dummy ASN.

johnar@86171a92b545:~/code/sonic-mgmt/tests$ python
Python 2.7.12 (default, Apr 15 2020, 17:07:12)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> url='http://10.250.0.106:5008'
>>> announce={'commands': 'announce route 172.16.10.0/24 next-hop 10.0.0.32 as-path [ 65100 64101 ]'}
>>> requests.post(url, announce)
<Response [200]>

ARISTA01T0#show ip bgp | grep 172.16.10.0/24
 * >     172.16.10.0/24         10.0.0.32             0       100     0       65100 64101 i

Check the route on DUT. The route is NOT accepted by DUT. This is OK because there is no BGP configuration lines neighbor PEER_V4 allowas-in 1 and neighbor PEER_V6 allowas-in 1

admin@vlab-03:~$ vtysh -c 'show bgp ipv4' | grep 172.16.10.0/24
admin@vlab-03:~$

Withdraw the announced routes

>>> withdraw={'commands': 'withdraw route 172.16.10.0/24 next-hop 10.0.0.32'}
>>> requests.post(url, withdraw)

ARISTA01T0#show ip bgp | grep 172.16.10.0/24
ARISTA01T0#

Manually add BGP configuration allowas-in

admin@vlab-03:~$ vtysh

Hello, this is FRRouting (version 7.2.1-sonic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

vlab-03# conf t
vlab-03(config)# router bgp 65100
vlab-03(config-router)# address-family ipv4 unicast 
vlab-03(config-router-af)# neighbor PEER_V4 allowas-in 1
vlab-03(config-router-af)# exit
vlab-03(config-router)# address-family ipv6 unicast 
vlab-03(config-router-af)# neighbor PEER_V6 allowas-in 1
vlab-03(config-router-af)# end
vlab-03# exit
admin@vlab-03:~$ vtysh -c 'show run' | grep allow
  neighbor PEER_V4 allowas-in 1
  neighbor PEER_V6 allowas-in 1
admin@vlab-03:~$ redis-cli -n 4 HGET "BGP_BBR|all" "status"
"enabled"

Announce route to the first T0 VM. The route has DUT asn (65100) once in aspath. ASN 64101 is a dummy ASN.

>>> announce
{'commands': 'announce route 172.16.10.0/24 next-hop 10.0.0.32 as-path [ 65100 64101 ]'}
>>> url
'http://10.250.0.106:5008'
>>> requests.post(url, announce)
<Response [200]>

ARISTA01T0#show ip bgp | grep 172.16.10.0/24
 * >     172.16.10.0/24         10.0.0.32             0       100     0       65100 64101 i

Check the route on DUT. The route is accepted by DUT. This time, BBR works after manually enabled it in step 9.

admin@vlab-03:~$ vtysh -c 'show bgp ipv4' | grep 172.16.10.0/24
*> 172.16.10.0/24   10.0.0.33                              0 64001 65100 64101 i

Describe the results you received:

When BBR is enabled in /etc/sonic/constants.yml, BBR is still working after dynamically disabled it by:

admin@vlab-03:~$ cat /tmp/disable_bbr.json 
{
    "BGP_BBR": {
        "all": {
            "status": "disabled"
        }
    }
}admin@vlab-03:~$ sonic-cfggen -j /tmp/disable_bbr.json -w

When BBR is disabled in /etc/sonic/constants.yml, BBR configuration is not added after dynamically enabled it by:

admin@vlab-03:~$ cat /tmp/enable_bbr.json 
{
    "BGP_BBR": {
        "all": {
            "status": "enabled"
        }
    }
}admin@vlab-03:~$ sonic-cfggen -j /tmp/enable_bbr.json -w

Describe the results you expected:

BBR should not work when it is disabled
BBR configuration should can be added dynamically when /etc/sonic/constants.yml has BBR disabled.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

admin@vlab-03:~$ show version

SONiC Software Version: SONiC.master.552-a704ded6
Distribution: Debian 10.6
Kernel: 4.19.0-9-2-amd64
Build commit: a704ded6
Build date: Thu Nov 12 11:58:51 UTC 2020
Built by: johnar@jenkins-worker-11

Platform: x86_64-kvm_x86_64-r0
HwSKU: Force10-S6000
ASIC: vs
Serial Number: 000000
Uptime: 08:11:01 up  1:25,  1 user,  load average: 0.70, 1.02, 1.39

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-gbsyncd-vs             latest                8a15967a93b7        427MB
docker-gbsyncd-vs             master.552-a704ded6   8a15967a93b7        427MB
docker-syncd-vs               latest                51a7f15e4b00        427MB
docker-syncd-vs               master.552-a704ded6   51a7f15e4b00        427MB
docker-teamd                  latest                6e104b7df0be        454MB
docker-teamd                  master.552-a704ded6   6e104b7df0be        454MB
docker-nat                    latest                fb9b379b1698        457MB
docker-nat                    master.552-a704ded6   fb9b379b1698        457MB
docker-router-advertiser      latest                07794278c37c        421MB
docker-router-advertiser      master.552-a704ded6   07794278c37c        421MB
docker-platform-monitor       latest                6a0d418031a0        532MB
docker-platform-monitor       master.552-a704ded6   6a0d418031a0        532MB
docker-lldp                   latest                9af034b30d4d        485MB
docker-lldp                   master.552-a704ded6   9af034b30d4d        485MB
docker-database               latest                d07cf865ca79        421MB
docker-database               master.552-a704ded6   d07cf865ca79        421MB
docker-orchagent              latest                7af4e393376e        468MB
docker-orchagent              master.552-a704ded6   7af4e393376e        468MB
docker-snmp                   latest                e9f90983ecbf        458MB
docker-snmp                   master.552-a704ded6   e9f90983ecbf        458MB
docker-dhcp-relay             latest                30670ee9bf59        428MB
docker-dhcp-relay             master.552-a704ded6   30670ee9bf59        428MB
docker-sonic-telemetry        latest                19f21756bc5c        491MB
docker-sonic-telemetry        master.552-a704ded6   19f21756bc5c        491MB
docker-sonic-mgmt-framework   latest                d9448086a28b        582MB
docker-sonic-mgmt-framework   master.552-a704ded6   d9448086a28b        582MB
docker-fpm-frr                latest                edcd18dc5a32        471MB
docker-fpm-frr                master.552-a704ded6   edcd18dc5a32        471MB
docker-sflow                  latest                d9d07e70a0a3        455MB
docker-sflow                  master.552-a704ded6   d9d07e70a0a3        455MB

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

The text was updated successfully, but these errors were encountered:

pavel-shirshov · 2020-11-23T20:14:40Z

Should be fixed in #5956

daall · 2021-01-27T00:54:56Z

This behavior is still seen in master branch. Consistently repro'd w/ test_bpg_bbr.

lguohan · 2021-02-01T07:32:23Z

i checked 201911 branch and the issue is not there.

lguohan · 2021-02-01T08:40:58Z

it is the same root cause. I found this pr is not in master or 202012 branch.

9e0ea83 2020-11-26 | [bgpcfgd]: Use peer commands for BBR, not peer-group ([bgpcfgd]: Use peer commands for BBR, not peer-group #6048) [pavel-shirshov]

lguohan · 2021-02-01T08:58:08Z

looks like there is a new fix for this in upstream FRRouting/frr#7977

wangxin mentioned this issue Nov 16, 2020

Implement the BGP Bounce Back Routing test script sonic-net/sonic-mgmt#2538

Merged

3 tasks

pavel-shirshov linked a pull request Nov 23, 2020 that will close this issue

[bgpcfgd]: Fixes for BBR #5956

Merged

3 tasks

pavel-shirshov self-assigned this Nov 23, 2020

daall assigned abdosi and unassigned pavel-shirshov Jan 27, 2021

daall added Master Branch Quality Issue for 202012 labels Jan 27, 2021

lguohan assigned shi-su and unassigned abdosi Feb 1, 2021

lguohan linked a pull request Feb 1, 2021 that will close this issue

[frr]: upgrade frr to frr-7.5-s2 tag #6628

Merged

4 tasks

lguohan mentioned this issue Feb 1, 2021

[frr]: upgrade frr to frr-7.5-s2 tag #6628

Merged

4 tasks

lguohan closed this as completed in #6628 Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic BGP_BBR is not working as expected #5929

Dynamic BGP_BBR is not working as expected #5929

wangxin commented Nov 16, 2020

pavel-shirshov commented Nov 23, 2020

daall commented Jan 27, 2021 •

edited

Loading

lguohan commented Feb 1, 2021 •

edited

Loading

lguohan commented Feb 1, 2021

lguohan commented Feb 1, 2021

Dynamic BGP_BBR is not working as expected #5929

Dynamic BGP_BBR is not working as expected #5929

Comments

wangxin commented Nov 16, 2020

pavel-shirshov commented Nov 23, 2020

daall commented Jan 27, 2021 • edited Loading

lguohan commented Feb 1, 2021 • edited Loading

lguohan commented Feb 1, 2021

lguohan commented Feb 1, 2021

daall commented Jan 27, 2021 •

edited

Loading

lguohan commented Feb 1, 2021 •

edited

Loading