[Generic Config Updater] Failed to replace interface speed via GCU when the interface is up

[nazariig]

Description

The issue is caused by Buffer Configuration: once speed is updated the new profile is being generated automatically.
When port is admin down no automatic profile generation takes place.
Basically, the Config DB is being modified while Generic Config Updater expects data consistency.

Simple reproduce:

1. Port is Admin UP

root@sonic:/home/admin# show interfaces status Ethernet12
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
 Ethernet12    24,25      50G   9100    N/A    etp4a   trunk      up       up  QSFP28 or later         off

root@sonic:/home/admin# sonic-cfggen --from-db --print-data > ex_old.json
root@sonic:/home/admin# config interface speed Ethernet12 25000

root@sonic:/home/admin# show interfaces status Ethernet12
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
 Ethernet12    24,25      25G   9100    N/A    etp4a   trunk    down       up  QSFP28 or later         off

root@sonic:/home/admin# sonic-cfggen --from-db --print-data > ex_new.json

root@sonic:/home/admin# diff ex_new.json ex_old.json
411c411
<             "profile": "pg_lossless_25000_5m_profile"
---
>             "profile": "pg_lossless_50000_5m_profile"
414c414
<             "profile": "pg_lossless_25000_5m_profile"
---
>             "profile": "pg_lossless_50000_5m_profile"
1731,1737d1730
<         "pg_lossless_25000_5m_profile": {
<             "dynamic_th": "0",
<             "pool": "ingress_lossless_pool",
<             "size": "19456",
<             "xoff": "26624",
<             "xon": "19456"
<         },
3689c3682
<             "speed": "25000",
---
>             "speed": "50000",

2. Port is Admin DOWN

root@sonic:/home/admin# sonic-cfggen --from-db --print-data > ex_old.json
root@sonic:/home/admin# show interfaces status Ethernet12
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
 Ethernet12    24,25      50G   9100    N/A    etp4a   trunk      up       up  QSFP28 or later         off

root@sonic:/home/admin# config interface shutdown Ethernet12
root@sonic:/home/admin# config interface speed Ethernet12 25000
root@sonic:/home/admin# sonic-cfggen --from-db --print-data > ex_new.json

root@sonic:/home/admin# show interfaces status Ethernet12
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
 Ethernet12    24,25      25G   9100    N/A    etp4a   trunk    down     down  QSFP28 or later         off

root@sonic:/home/admin# diff ex_new.json ex_old.json
409a410,415
>         "Ethernet12|3": {
>             "profile": "pg_lossless_50000_5m_profile"
>         },
>         "Ethernet12|4": {
>             "profile": "pg_lossless_50000_5m_profile"
>         },
3669c3675
<             "admin_status": "down",
---
>             "admin_status": "up",
3676c3682
<             "speed": "25000",
---
>             "speed": "50000",

Extra info:
/usr/local/lib/python3.9/dist-packages/generic_config_updater/generic_updater.py

    # Validate config updated successfully
    self.logger.log_notice("Verifying patch updates are reflected on ConfigDB.")
    new_config = self.config_wrapper.get_config_db_as_json()
    # DEBUG
    if not(self.patch_wrapper.verify_same_json(target_config, new_config)):
        p = self.patch_wrapper.generate_patch(target_config, new_config)
        print("Patch Applier: Unexpected diff:\n" + json.dumps(json.loads(str(p)),indent=4))
        raise GenericConfigUpdaterError(f"After applying patch to config, there are still some parts not updated")
    # DEBUG
    #if not(self.patch_wrapper.verify_same_json(target_config, new_config)):
    #    raise GenericConfigUpdaterError(f"After applying patch to config, there are still some parts not updated")

    self.logger.log_notice("Patch application completed.")

Steps to reproduce the issue:

1. Check port state

root@sonic:/home/admin# show interfaces status Ethernet12
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
 Ethernet12    24,25      50G   9100    N/A    etp4a   trunk      up       up  QSFP28 or later         off

2. Prepare patch

root@sonic:/home/admin# cat test.json | jq --indent 4 .
[
    {
        "op": "replace",
        "path": "/PORT/Ethernet12/speed",
        "value": "25000"
    }
]

3. Apply patch

root@sonic:/home/admin# config apply-patch test.json
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "replace", "path": "/PORT/Ethernet12/speed", "value": "25000"}]
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
Patch Applier: Sorting patch updates.
Patch Applier: The patch was sorted into 1 change:
Patch Applier:   * [{"op": "replace", "path": "/PORT/Ethernet12/speed", "value": "25000"}]
Patch Applier: Applying 1 change in order:
Patch Applier:   * [{"op": "replace", "path": "/PORT/Ethernet12/speed", "value": "25000"}]
Patch Applier: Verifying patch updates are reflected on ConfigDB.
Patch Applier: Unexpected diff:
[
    {
        "op": "add",
        "path": "/BUFFER_PROFILE/pg_lossless_25000_5m_profile",
        "value": {
            "dynamic_th": "0",
            "pool": "ingress_lossless_pool",
            "size": "19456",
            "xoff": "26624",
            "xon": "19456"
        }
    },
    {
        "op": "replace",
        "path": "/BUFFER_PG/Ethernet12|4/profile",
        "value": "pg_lossless_25000_5m_profile"
    },
    {
        "op": "replace",
        "path": "/BUFFER_PG/Ethernet12|3/profile",
        "value": "pg_lossless_25000_5m_profile"
    }
]
Failed to apply patch
Usage: config apply-patch [OPTIONS] PATCH_FILE_PATH
Try "config apply-patch -h" for help.

Error: After applying patch to config, there are still some parts not updated

4. Check port state

root@sonic:/home/admin# show interfaces status Ethernet12
  Interface    Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  -------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
 Ethernet12    24,25      25G   9100    N/A    etp4a   trunk    down       up  QSFP28 or later         off

Describe the results you received:

Port speed update with GCU config apply-patch has failed

Describe the results you expected:

Port speed update with GCU config apply-patch should pass

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

• GCU PR: [config][generic-update] Adding apply-patch, rollback, checkpoints commands sonic-utilities#1536

[nazariig]

@ghooo FYI

[zhangyanzhao]

@qiluo-msft the issue still exists, please help to take a look. Thanks.

[qiluo-msft]

Could you check syslog messages? I guess it is duplicated issue of #11576 and there are someone assigned there.

[stephenxs]

Could you check syslog messages? I guess it is duplicated issue of #11576 and there are someone assigned there.

@qiluo-msft Yes. You are correct.

[qiluo-msft]

It is duplicated issue of #11576. Close this one.