Bullseye: DHCP relay test case fails with missing packets and zebra crash

[saiarcot895]

Description

On a Bullseye image, when running the dhcp_relay/test_dhcp_relay.py testcase on a kvm t0 topology, zebra crashes, with the following backtrace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ff499a54535 in __GI_abort () at abort.c:79
#2  0x00007ff499a5440f in __assert_fail_base (fmt=0x7ff499bb6ee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ff4995624f1 "data_len", file=0x7ff499562074 "zebra/zebra_fpm.c", line=998, function=<optimized out>) at assert.c:92
#3  0x00007ff499a62102 in __GI___assert_fail (assertion=assertion@entry=0x7ff4995624f1 "data_len", file=file@entry=0x7ff499562074 "zebra/zebra_fpm.c", line=line@entry=998,
    function=function@entry=0x7ff499562c70 <__PRETTY_FUNCTION__.21844> "zfpm_build_route_updates") at assert.c:101
#4  0x00007ff499560dd2 in zfpm_build_route_updates () at zebra/zebra_fpm.c:998
#5  zfpm_build_updates () at zebra/zebra_fpm.c:1154
#6  zfpm_write_cb (thread=<optimized out>) at zebra/zebra_fpm.c:1191
#7  zfpm_write_cb (thread=0x7ffc1dcd2c10) at zebra/zebra_fpm.c:1162
#8  0x00007ff499dac516 in thread_call (thread=thread@entry=0x7ffc1dcd2c10) at lib/thread.c:1628
#9  0x00007ff499d74cf0 in frr_run (master=0x562209e7da10) at lib/libfrr.c:1099
#10 0x000056220880ed0d in main (argc=9, argv=0x7ffc1dcd3018) at zebra/main.c:457

In addition, the DHCP test case fails because it's unable to find the relay/offer packets.

Describe the results you received:

Snippet of test output:

        "WARNING: No route found for IPv6 destination :: (no default route?)",                                                                                                                                                                                      "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.",                                                                                                                                                                                                                                                                 "  from cryptography.hazmat.backends import default_backend",                                                                                                                                                                                               "dhcp_relay_test.DHCPTest ... FAIL",                                                                                                                                                                                                                        "",                                                                                                                                                                                                                                                         "======================================================================",                                                                                                                                                                                   "FAIL: dhcp_relay_test.DHCPTest",                                                                                                                                                                                                                           "----------------------------------------------------------------------",                                                                                                                                                                                   "Traceback (most recent call last):",                                                                                                                                                                                                                       "  File \"ptftests/dhcp_relay_test.py\", line 565, in runTest",                                                                                                                                                                                             "    self.verify_offer_received()",                                                                                                                                                                                                                         "  File \"ptftests/dhcp_relay_test.py\", line 483, in verify_offer_received",                                                                                                                                                                               "    testutils.verify_packet(self, masked_offer, self.client_port_index)",                                                                                                                                                                                  "  File \"/usr/lib/python2.7/dist-packages/ptf/testutils.py\", line 2405, in verify_packet",                                                                                                                                                                "    % (device, port, result.format()))",                                                                                                                                                                                                                   "AssertionError: Expected packet was not received on device 0, port 1.",                                                                                                                                                                                    "========== EXPECTED ==========",                                                                                                                                                                                                                           "Mask:",                                                                                                                                                                                                                                                    "0000   FF FF FF FF FF FF 52 54  00 C6 07 77 08 00 45 00   ......RT...w..E.",                                                                                                                                                                               "0010   01 22 00 01 00 00 40 11  B9 21 C0 A8 00 01 FF FF   .\"....@..!......",                                                                                                                                                                              "0020   FF FF 00 43 00 44 01 06  0C 1F 02 01 06 00 00 00   ...C.D..........",                                                                                                                                                                               "0030   00 00 00 00 80 00 00 00  00 00 C0 A8 00 02 C0 00   ................",                                                                                                                                                                               "0040   00 01 C0 A8 00 01 FE 54  00 83 E9 01 00 00 00 00   .......T........",                                                                                                                                                                               "0050   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "0060   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "0070   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "0080   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "0090   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "00a0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "00b0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "00c0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "00d0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "00e0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "00f0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "0100   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................",
        "0110   00 00 00 00 00 00 63 82  53 63 35 01 02 36 04 C0   ......c.Sc5..6..",
        "0120   00 00 01 33 04 00 01 51  80 01 04 FF FF F8 00 FF   ...3...Q........",
        "mask = ff ff ff ff ff ff ff ff  ff ff ff ff ff ff 00 00",
        "0010   00 00 00 00 00 00 00 00  00 00 ff ff ff ff ff ff",
        "0020   ff ff 00 00 ff ff 00 00  00 00 ff ff ff ff ff ff",
        "0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff",
        "0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff",
        "0050   ff ff ff ff ff ff 00 00  00 00 00 00 00 00 00 00",
        "0060   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "0070   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "0080   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "0090   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "00a0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "00b0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "00c0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "00d0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "00e0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "00f0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "0100   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00",
        "0110   00 00 00 00 00 00 ff ff  ff ff ff ff ff ff ff ff",
        "0120   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff",
        "",

Describe the results you expected:

Output of show version:

SONiC Software Version: SONiC.bullseye-upgrade-base.0-dirty-20210813.131217
Distribution: Debian 11.0
Kernel: 5.10.0-7-2-amd64
Build commit: 0487f6f66
Build date: Fri Aug 13 20:24:14 UTC 2021
Built by: sarcot@sarcot-dev-vm-01

Platform: x86_64-kvm_x86_64-r0
HwSKU: Force10-S6000
ASIC: vs
ASIC Count: 1
Serial Number: None
Model Number: None
Hardware Revision: None
Uptime: 22:50:24 up  1:55,  1 user,  load average: 0.51, 0.39, 0.32

Output of show techsupport:

sonic_dump_vlab-01_20210813_222231.tar.gz

Additional information you deem important (e.g. issue happens only occasionally):

Since this is a Bullseye image, python on the switch points to Python 3. Within the containers, python still points to Python 2.

Related PR: #8191

[lguohan]

the bug is fixed in #8443 , can you rebase your pr?

[saiarcot895]

Zebra isn't crashing anymore, so the switch is in a good state now, but DHCP packets are still not being received.