[bgpcfgd]: Dynamic BBR support #5626

pavel-shirshov · 2020-10-14T18:44:43Z

- Why I did it
To introduce dynamic support of BBR functionality into bgpcfgd.
BBR is adding `neighbor PEER_GROUP allowas-in 1' for all BGP peer-groups which points to T0
Now we can add and remove this configuration based on CONFIG_DB entry

- How I did it
I introduced a new CONFIG_DB entry:

table name: "BGP_BBR"
key value: "all". Currently only "all" is supported, which means that all peer-groups which points to T0s will be updated
data value: a dictionary: {"status": "status_value"}, where status_value could be either "enabled" or "disabled"

Initially, when bgpcfgd starts, it reads initial BBR status values from the constants.yml. Then you can control BBR status by changing "BGP_BBR" table in the CONFIG_DB (see examples below).

bgpcfgd knows what peer-groups to change fron constants.yml. The dictionary contains peer-group names as keys, and a list of address-families as values. So when bgpcfgd got a request to change the BBR state, it changes the state only for peer-groups listed in the constants.yml dictionary (and only for address families from the peer-group value).

- How to verify it
Initially, when we start SONiC FRR has BBR enabled for PEER_V4 and PEER_V6:

admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas'
  neighbor PEER_V4 allowas-in 1
  neighbor PEER_V6 allowas-in 1

Then we apply following configuration to the db:

admin@str-s6100-acs-1:~$ cat disable.json                
{
        "BGP_BBR": {
            "all": {
                "status": "disabled"
            }
        }
}


admin@str-s6100-acs-1:~$ sonic-cfggen -j disable.json -w

The log output are:

Oct 14 18:40:22.450322 str-s6100-acs-1 DEBUG bgp#bgpcfgd: Received message : '('all', 'SET', (('status', 'disabled'),))'
Oct 14 18:40:22.450620 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-f', '/tmp/tmpmWTiuq']'.
Oct 14 18:40:22.681084 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V4 soft in']'.
Oct 14 18:40:22.904626 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V6 soft in']'.

Check FRR configuraiton and see that no allowas parameters are there:

admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' 
admin@str-s6100-acs-1:~$

Then we apply enabling configuration back:

admin@str-s6100-acs-1:~$ cat enable.json 
{
        "BGP_BBR": {
            "all": {
                "status": "enabled"
            }
        }
}

admin@str-s6100-acs-1:~$ sonic-cfggen -j enable.json -w

The log output:

Oct 14 18:40:41.074720 str-s6100-acs-1 DEBUG bgp#bgpcfgd: Received message : '('all', 'SET', (('status', 'enabled'),))'
Oct 14 18:40:41.074720 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-f', '/tmp/tmpDD6SKv']'.
Oct 14 18:40:41.587257 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V4 soft in']'.
Oct 14 18:40:42.042967 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V6 soft in']'.

Check FRR configuraiton and see that the BBR configuration is back:

admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas'
  neighbor PEER_V4 allowas-in 1
  neighbor PEER_V6 allowas-in 1

*** The test coverage ***
Below is the test coverage

---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name                             Stmts   Miss  Cover
----------------------------------------------------
bgpcfgd/__init__.py                  0      0   100%
bgpcfgd/__main__.py                  3      3     0%
bgpcfgd/config.py                   78     41    47%
bgpcfgd/directory.py                63     34    46%
bgpcfgd/log.py                      15      3    80%
bgpcfgd/main.py                     51     51     0%
bgpcfgd/manager.py                  41     23    44%
bgpcfgd/managers_allow_list.py     385     21    95%
bgpcfgd/managers_bbr.py             76      0   100%
bgpcfgd/managers_bgp.py            193    193     0%
bgpcfgd/managers_db.py               9      9     0%
bgpcfgd/managers_intf.py            33     33     0%
bgpcfgd/managers_setsrc.py          45     45     0%
bgpcfgd/runner.py                   39     39     0%
bgpcfgd/template.py                 64     11    83%
bgpcfgd/utils.py                    32     24    25%
bgpcfgd/vars.py                      1      0   100%
----------------------------------------------------
TOTAL                             1128    530    53%

- Which release branch to backport (provide reason below if selected)

201811
201911
202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

pavel-shirshov · 2020-10-14T21:15:03Z

I see the test issue. I'll fix it ASAP.

lguohan · 2020-10-14T22:00:50Z

can you fix the test?

lguohan · 2020-10-14T22:50:14Z

have you check the bgp update timer? are those prefixes withdrawn immediately?

lguohan · 2020-10-20T07:49:04Z

does enable/disable bbr have traffic impact?

lguohan · 2020-10-20T07:51:07Z

src/sonic-bgpcfgd/bgpcfgd/managers_bbr.py

+        rv = self.cfg_mgr.push_list(cmds)
+        if not rv:
+            log_crit("BBRMgr::can't apply configuration")
+            return True


always return True?

Yes. False means save it in the queue, and give it back to me next time,
True means - remove it completely.
In this case I don't see any reason the change could be applied without any issue in the future

src/sonic-bgpcfgd/bgpcfgd/managers_bbr.py

lguohan · 2020-10-20T07:53:27Z

src/sonic-bgpcfgd/bgpcfgd/managers_bbr.py

+
+    def __restart_peers(self):
+        """ Restart peer-groups which support BBR """
+        for peer_group in sorted(self.bbr_enabled_pgs.keys()):


why sort here? any reason?

Otherwise my test doesn't know what to expect (or I can use conversion to sets) and compare sets. I'm going to remove it after the batching change

lguohan · 2020-10-20T07:56:36Z

src/sonic-bgpcfgd/tests/test_bbr.py

+    m.enabled = is_enabled
+    prepare_config_return_value = [
+                               ["vtysh", "-c", "clear bgp peer-group PEER_V4 soft in"],
+                               ["vtysh", "-c", "clear bgp peer-group PEER_V6 soft in"]


in the description, when you enable/disable bbr, you only modify the peer group configuration for the peer group for T0. However, here you seems restart PEER_V4/V6 which is the peer group for T0 as well. Do we need to restart all peer groups, or only T0 peer groups?

In the public we have PEER_V4 and PEER_V6, They both work for T2, T1, and T0 peers. So here I'll restart them as T0. There is no way to distinct T1 from T0 (no neighbor meta in the public enabled by requests from community). But internal configs has PEERs which a designed to be used only for T0. In our internal case we restart only required peer groups.

lguohan · 2020-10-20T07:59:42Z

src/sonic-bgpcfgd/bgpcfgd/managers_bbr.py

+        if 'bbr' in self.constants['bgp'] and \
+                'enabled' in self.constants['bgp']['bbr'] and \
+                self.constants['bgp']['bbr']['enabled']:
+            self.bbr_enabled_pgs = self.__read_pgs()


"BBR is adding `neighbor PEER_GROUP allowas-in 1' for all BGP peer-groups which points to T0"

can you point me code where you identify if a peer group points to T0 or not?

Sure. I use file constants

lguohan · 2020-10-20T08:01:37Z

src/sonic-bgpcfgd/tests/test_bbr.py

+        ' address-family ipv4',
+        '  no neighbor PEER_V4 allowas-in 1',
+        ' address-family ipv6',
+        '  no neighbor PEER_V4 allowas-in 1',


I am a little bit confused here, why do we have PEER_V4 in address-family ipv6?

Sorry for misleading here, I added this just for a test:
As you can see from the source data, PEER_V4 works for both ipv4 and ipv6

"PEER_V4": ["ipv4", "ipv6"], "PEER_V6": ["ipv6"],

lguohan · 2020-10-20T08:03:00Z

files/image_config/constants/constants.yml

+          PEER_V4:
+            - ipv4
+          PEER_V6:
+            - ipv6


why not use device metadata role to identify if a peer is a tor or not?

I don't have neighbor metadata in public. That was disabled by multiple requests from the community. Without this change I can implement the change in internal, but not in the public. But we want to test the feature on public too.

I need a way to define, what address families are used by each peer-group for BBR.

I think this adds additional maintaining cost. it is better to check if neighbor metadata is available or not, if it is available, then use it. otherwise, look for bbr definition. do you agree?

I don't agree here, sorry

We need to define mapping between address families and peer-groups. We can't extract this information from the minigraph, it is available only in templates.

Currently bgpcfgd doesn't know anything about the device neighbor metadata, also as T0, T1, T2 and so on. That was requested explicitly by community. So I add an option to depend on the neighbor metadata presence, but I enable that option only for internal image. All other operations with T0, T1 and so on is done inside of the internal templates.

We need to test this functionality using the public image, so we need a way to define this without neighbor meta

Current approach allows us to have T0 peer-groups, which doesn't support BBR.

My goal is to make templates downloadable from external source, like we download acl rules. I think of having a tar archive with templates + metadata file in the archive describing the metadata of the templates.

pavel-shirshov · 2020-10-20T16:07:53Z

@lguohan No traffic impact, I use 'soft' reset to update RIB

**- Why I did it** To introduce dynamic support of BBR functionality into bgpcfgd. BBR is adding `neighbor PEER_GROUP allowas-in 1' for all BGP peer-groups which points to T0 Now we can add and remove this configuration based on CONFIG_DB entry **- How I did it** I introduced a new CONFIG_DB entry: - table name: "BGP_BBR" - key value: "all". Currently only "all" is supported, which means that all peer-groups which points to T0s will be updated - data value: a dictionary: {"status": "status_value"}, where status_value could be either "enabled" or "disabled" Initially, when bgpcfgd starts, it reads initial BBR status values from the [constants.yml](https://github.com/Azure/sonic-buildimage/pull/5626/files#diff-e6f2fe13a6c276dc2f3b27a5bef79886f9c103194be4fcb28ce57375edf2c23cR34). Then you can control BBR status by changing "BGP_BBR" table in the CONFIG_DB (see examples below). bgpcfgd knows what peer-groups to change fron [constants.yml](https://github.com/Azure/sonic-buildimage/pull/5626/files#diff-e6f2fe13a6c276dc2f3b27a5bef79886f9c103194be4fcb28ce57375edf2c23cR39). The dictionary contains peer-group names as keys, and a list of address-families as values. So when bgpcfgd got a request to change the BBR state, it changes the state only for peer-groups listed in the constants.yml dictionary (and only for address families from the peer-group value). **- How to verify it** Initially, when we start SONiC FRR has BBR enabled for PEER_V4 and PEER_V6: ``` admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' neighbor PEER_V4 allowas-in 1 neighbor PEER_V6 allowas-in 1 ``` Then we apply following configuration to the db: ``` admin@str-s6100-acs-1:~$ cat disable.json { "BGP_BBR": { "all": { "status": "disabled" } } } admin@str-s6100-acs-1:~$ sonic-cfggen -j disable.json -w ``` The log output are: ``` Oct 14 18:40:22.450322 str-s6100-acs-1 DEBUG bgp#bgpcfgd: Received message : '('all', 'SET', (('status', 'disabled'),))' Oct 14 18:40:22.450620 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-f', '/tmp/tmpmWTiuq']'. Oct 14 18:40:22.681084 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V4 soft in']'. Oct 14 18:40:22.904626 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V6 soft in']'. ``` Check FRR configuraiton and see that no allowas parameters are there: ``` admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' admin@str-s6100-acs-1:~$ ``` Then we apply enabling configuration back: ``` admin@str-s6100-acs-1:~$ cat enable.json { "BGP_BBR": { "all": { "status": "enabled" } } } admin@str-s6100-acs-1:~$ sonic-cfggen -j enable.json -w ``` The log output: ``` Oct 14 18:40:41.074720 str-s6100-acs-1 DEBUG bgp#bgpcfgd: Received message : '('all', 'SET', (('status', 'enabled'),))' Oct 14 18:40:41.074720 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-f', '/tmp/tmpDD6SKv']'. Oct 14 18:40:41.587257 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V4 soft in']'. Oct 14 18:40:42.042967 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V6 soft in']'. ``` Check FRR configuraiton and see that the BBR configuration is back: ``` admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' neighbor PEER_V4 allowas-in 1 neighbor PEER_V6 allowas-in 1 ``` *** The test coverage *** Below is the test coverage ``` ---------- coverage: platform linux2, python 2.7.12-final-0 ---------- Name Stmts Miss Cover ---------------------------------------------------- bgpcfgd/__init__.py 0 0 100% bgpcfgd/__main__.py 3 3 0% bgpcfgd/config.py 78 41 47% bgpcfgd/directory.py 63 34 46% bgpcfgd/log.py 15 3 80% bgpcfgd/main.py 51 51 0% bgpcfgd/manager.py 41 23 44% bgpcfgd/managers_allow_list.py 385 21 95% bgpcfgd/managers_bbr.py 76 0 100% bgpcfgd/managers_bgp.py 193 193 0% bgpcfgd/managers_db.py 9 9 0% bgpcfgd/managers_intf.py 33 33 0% bgpcfgd/managers_setsrc.py 45 45 0% bgpcfgd/runner.py 39 39 0% bgpcfgd/template.py 64 11 83% bgpcfgd/utils.py 32 24 25% bgpcfgd/vars.py 1 0 100% ---------------------------------------------------- TOTAL 1128 530 53% ``` **- Which release branch to backport (provide reason below if selected)** - [ ] 201811 - [x] 201911 - [x] 202006

What is the motivation for this PR? The BGP Bounce Back Routing (BBR) feature has been introduced into SONiC. This change is to implement a test script for testing this BGP BBR feature. How did you do it? This test requires T1 type topology. Added 3 test cases: Test Case1 * Enable BBR, announce a route to the first T0 VM. The route has DUT asn in aspath. * Verify that the route is accepted by DUT and propagated to other VMs. Test Case2 * Enable BBR, announce a route to the first T0 VM. The route has two DUT asn in aspath. * Verify that the route is not accepted by DUT and not propagated to other VMs. Test Case3 * Disable BBR, announce a route to the first T0 VM. The route has DUT asn in aspath. * Verify that the route is not accepted by DUT and not propagated to other VMs. * Enable BBR again after test. How did you verify/test it? Test run the script on vs setup running t1-lag topology and 201911 image. Any platform specific information? No Supported testbed topology if it's a new test case? t1 type topology Documentation sonic-net/sonic-buildimage#5626 Signed-off-by: Xin Wang <[email protected]>

**- Why I did it** To introduce dynamic support of BBR functionality into bgpcfgd. BBR is adding `neighbor PEER_GROUP allowas-in 1' for all BGP peer-groups which points to T0 Now we can add and remove this configuration based on CONFIG_DB entry **- How I did it** I introduced a new CONFIG_DB entry: - table name: "BGP_BBR" - key value: "all". Currently only "all" is supported, which means that all peer-groups which points to T0s will be updated - data value: a dictionary: {"status": "status_value"}, where status_value could be either "enabled" or "disabled" Initially, when bgpcfgd starts, it reads initial BBR status values from the [constants.yml](https://github.com/Azure/sonic-buildimage/pull/5626/files#diff-e6f2fe13a6c276dc2f3b27a5bef79886f9c103194be4fcb28ce57375edf2c23cR34). Then you can control BBR status by changing "BGP_BBR" table in the CONFIG_DB (see examples below). bgpcfgd knows what peer-groups to change fron [constants.yml](https://github.com/Azure/sonic-buildimage/pull/5626/files#diff-e6f2fe13a6c276dc2f3b27a5bef79886f9c103194be4fcb28ce57375edf2c23cR39). The dictionary contains peer-group names as keys, and a list of address-families as values. So when bgpcfgd got a request to change the BBR state, it changes the state only for peer-groups listed in the constants.yml dictionary (and only for address families from the peer-group value). **- How to verify it** Initially, when we start SONiC FRR has BBR enabled for PEER_V4 and PEER_V6: ``` admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' neighbor PEER_V4 allowas-in 1 neighbor PEER_V6 allowas-in 1 ``` Then we apply following configuration to the db: ``` admin@str-s6100-acs-1:~$ cat disable.json { "BGP_BBR": { "all": { "status": "disabled" } } } admin@str-s6100-acs-1:~$ sonic-cfggen -j disable.json -w ``` The log output are: ``` Oct 14 18:40:22.450322 str-s6100-acs-1 DEBUG bgp#bgpcfgd: Received message : '('all', 'SET', (('status', 'disabled'),))' Oct 14 18:40:22.450620 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-f', '/tmp/tmpmWTiuq']'. Oct 14 18:40:22.681084 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V4 soft in']'. Oct 14 18:40:22.904626 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V6 soft in']'. ``` Check FRR configuraiton and see that no allowas parameters are there: ``` admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' admin@str-s6100-acs-1:~$ ``` Then we apply enabling configuration back: ``` admin@str-s6100-acs-1:~$ cat enable.json { "BGP_BBR": { "all": { "status": "enabled" } } } admin@str-s6100-acs-1:~$ sonic-cfggen -j enable.json -w ``` The log output: ``` Oct 14 18:40:41.074720 str-s6100-acs-1 DEBUG bgp#bgpcfgd: Received message : '('all', 'SET', (('status', 'enabled'),))' Oct 14 18:40:41.074720 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-f', '/tmp/tmpDD6SKv']'. Oct 14 18:40:41.587257 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V4 soft in']'. Oct 14 18:40:42.042967 str-s6100-acs-1 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'clear bgp peer-group PEER_V6 soft in']'. ``` Check FRR configuraiton and see that the BBR configuration is back: ``` admin@str-s6100-acs-1:~$ vtysh -c 'show run' | egrep 'PEER_V.? allowas' neighbor PEER_V4 allowas-in 1 neighbor PEER_V6 allowas-in 1 ``` *** The test coverage *** Below is the test coverage ``` ---------- coverage: platform linux2, python 2.7.12-final-0 ---------- Name Stmts Miss Cover ---------------------------------------------------- bgpcfgd/__init__.py 0 0 100% bgpcfgd/__main__.py 3 3 0% bgpcfgd/config.py 78 41 47% bgpcfgd/directory.py 63 34 46% bgpcfgd/log.py 15 3 80% bgpcfgd/main.py 51 51 0% bgpcfgd/manager.py 41 23 44% bgpcfgd/managers_allow_list.py 385 21 95% bgpcfgd/managers_bbr.py 76 0 100% bgpcfgd/managers_bgp.py 193 193 0% bgpcfgd/managers_db.py 9 9 0% bgpcfgd/managers_intf.py 33 33 0% bgpcfgd/managers_setsrc.py 45 45 0% bgpcfgd/runner.py 39 39 0% bgpcfgd/template.py 64 11 83% bgpcfgd/utils.py 32 24 25% bgpcfgd/vars.py 1 0 100% ---------------------------------------------------- TOTAL 1128 530 53% ``` **- Which release branch to backport (provide reason below if selected)** - [ ] 201811 - [x] 201911 - [x] 202006

Introduce dynamic BBR change into bgpcfgd

e6a663e

pavel-shirshov requested a review from lguohan October 14, 2020 18:46

pavel-shirshov self-assigned this Oct 14, 2020

pavel-shirshov added bgp Enhancement ➕ FRR 🚥 Request for 201911 Branch Request for 202006 Branch Unit Tests labels Oct 14, 2020

Fix tests

671c915

lguohan reviewed Oct 20, 2020

View reviewed changes

src/sonic-bgpcfgd/bgpcfgd/managers_bbr.py Outdated Show resolved Hide resolved

lguohan reviewed Oct 20, 2020

View reviewed changes

Pavel Shirshov and others added 2 commits October 20, 2020 08:14

Fix a typo

db19e62

Merge branch 'master' into pavelsh/bbr_new

b3a1cc3

lguohan approved these changes Oct 22, 2020

View reviewed changes

pavel-shirshov merged commit c94f93f into sonic-net:master Oct 22, 2020

rlhui added the Approved for 201911 Branch label Oct 30, 2020

abdosi added the Included in 201911 Branch label Oct 30, 2020

wangxin mentioned this pull request Nov 16, 2020

Implement the BGP Bounce Back Routing test script sonic-net/sonic-mgmt#2538

Merged

3 tasks

wangxin mentioned this pull request Nov 16, 2020

Dynamic BGP_BBR is not working as expected #5929

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bgpcfgd]: Dynamic BBR support #5626

[bgpcfgd]: Dynamic BBR support #5626

pavel-shirshov commented Oct 14, 2020 •

edited

Loading

pavel-shirshov commented Oct 14, 2020

lguohan commented Oct 14, 2020

lguohan commented Oct 14, 2020

lguohan commented Oct 20, 2020

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020 •

edited

Loading

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020

lguohan Oct 20, 2020

pavel-shirshov Oct 20, 2020

pavel-shirshov commented Oct 20, 2020

[bgpcfgd]: Dynamic BBR support #5626

[bgpcfgd]: Dynamic BBR support #5626

Conversation

pavel-shirshov commented Oct 14, 2020 • edited Loading

pavel-shirshov commented Oct 14, 2020

lguohan commented Oct 14, 2020

lguohan commented Oct 14, 2020

lguohan commented Oct 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavel-shirshov Oct 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavel-shirshov commented Oct 20, 2020

pavel-shirshov commented Oct 14, 2020 •

edited

Loading

pavel-shirshov Oct 20, 2020 •

edited

Loading