Working/2.0/patch set 161218a #15

louberger · 2016-12-19T01:09:09Z

This patch set includes changes resulting from initial regression runs with stable/2.0
Issues found include:
valgrind reported memory loss (this set doesn't fix them all)
missing vpn&encap commands (see #14)
some RFAPI?VNC patches were missing
- fix for issue reported in #9 (mislabeled as #30)
- Other changes made in November

- "redist foo" parsing modified to check for foo==vnc and foo==vnc-direct instead of just leading 'v' character - string designating ZEBRA_ROUTE_VNC_DIRECT changed from "vpn" to "vnc-direct" - route_types.pl parser recognizes 7th field to restrict availability of a route type in the redist command to specific daemons - restrict "vnc-direct" to bgpd only (doesn't make sense elsewhere) - vnc documentation updated to match

expose bgp_rfapi_get_group_by_lni_label for use by rfp add EVPN Ethernet Tag (VID) RT ensure as is init'ed fix spelling of information

donaldsharp

shouldn't the first hunk be:
&& (mpath_cfg && CHECK_FLAG(....)

Why would we want to compare cluster length if they have not configured it from the cli?

donaldsharp · 2016-12-19T18:43:03Z

bgpd/bgp_route.c

-       "Address family\n"
-       "Address Family modifier\n"
-       "Address Family modifier\n"
+       "Address family\n" 


White space changes, need to be removed.

@donaldsharp WRT && (mpath_cfg && CHECK_FLAG(....) - no as this is a passed parameter which may be NULL, i.e. not provide any config info.

donaldsharp · 2016-12-19T18:43:12Z

bgpd/bgp_route.c

@@ -8161,28 +8160,26 @@ DEFUN (show_ip_bgp_ipv4,
       SHOW_STR
       IP_STR
       BGP_STR
-       "Address family\n"
-       "Address Family modifier\n"
+       "Address family\n" 


white space changes need to be removed

@donaldsharp okay

eqvinox · 2016-12-20T14:47:58Z

@louberger can you add Signed-off-bys on these?
Also, the commit messages on ddf0364 and 0254f45 are mangled (everything was condensed into the subject line)

louberger · 2016-12-20T18:03:19Z

@eqvinox I'll add signoffs, but not sure what you mean on the commits as they show fine for me in git log...

topotest diagnostics

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > FRRouting#1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > FRRouting#2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > FRRouting#3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > FRRouting#4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > FRRouting#5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > FRRouting#6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > FRRouting#7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > FRRouting#8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > FRRouting#9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > FRRouting#10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > FRRouting#11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > FRRouting#12 0x7f26f275bae4 in route_table_free lib/table.c:111 > FRRouting#13 0x7f26f275b749 in route_table_finish lib/table.c:46 > FRRouting#14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > FRRouting#16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > FRRouting#17 0x7f26f2777108 in event_call lib/event.c:2011 > FRRouting#18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > FRRouting#19 0x55910c4f49cb in main zebra/main.c:531 > FRRouting#20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]>

Fix a heap-after-free that causes zebra to crash even without address-sanitizer. To reproduce: > echo "100 my_table" | tee -a /etc/iproute2/rt_tables > ip route add blackhole default table 100 > ip route show table 100 > ip l add red type vrf table 100 > ip l del red > ip route del blackhole default table 100 Zebra manages routing tables for all existing Linux RT tables, regardless of whether they are assigned to a VRF interface. When a table is not assigned to any VRF, zebra arbitrarily assigns it to the default VRF, even though this is not strictly accurate (the code expects this behavior). When an RT table is created after a VRF, zebra correctly assigns the table to the VRF. However, if a VRF interface is assigned to an existing RT table, zebra does not update the table owner, which remains as the default VRF. As a result, existing routing entries remain under the default VRF, while new entries are correctly assigned to the VRF. The VRF mismatch is unexpected in the code and creates crashes and memory related issues. Furthermore, Linux does not automatically delete RT tables when they are unassigned from a VRF. It is incorrect to delete these tables from zebra. Instead, at VRF disabling, do not release the table but reassign it to the default VRF. At VRF enabling, change the table owner back to the appropriate VRF. > ==2866266==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000154f54 at pc 0x7fa32474b83f bp 0x7ffe94f67d90 sp 0x7ffe94f67d88 > READ of size 1 at 0x606000154f54 thread T0 > #0 0x7fa32474b83e in rn_hash_node_const_find lib/table.c:28 > #1 0x7fa32474bab1 in rn_hash_node_find lib/table.c:28 > #2 0x7fa32474d783 in route_node_get lib/table.c:283 > #3 0x7fa3247328dd in srcdest_rnode_get lib/srcdest_table.c:231 > FRRouting#4 0x55b0e4fa8da4 in rib_find_rn_from_ctx zebra/zebra_rib.c:1957 > FRRouting#5 0x55b0e4fa8e31 in rib_process_result zebra/zebra_rib.c:1988 > FRRouting#6 0x55b0e4fb9d64 in rib_process_dplane_results zebra/zebra_rib.c:4894 > FRRouting#7 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#8 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#9 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#10 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > FRRouting#11 0x55b0e4e2d649 in _start (/usr/lib/frr/zebra+0x1a1649) > > 0x606000154f54 is located 20 bytes inside of 56-byte region [0x606000154f40,0x606000154f78) > freed by thread T0 here: > #0 0x7fa324ca9b6f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:123 > #1 0x7fa324668d8f in qfree lib/memory.c:130 > #2 0x7fa32474c421 in route_table_free lib/table.c:126 > #3 0x7fa32474bf96 in route_table_finish lib/table.c:46 > FRRouting#4 0x55b0e4fbca3a in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#5 0x55b0e4fbccea in zebra_router_release_table zebra/zebra_router.c:214 > FRRouting#6 0x55b0e4fd428e in zebra_vrf_disable zebra/zebra_vrf.c:219 > FRRouting#7 0x7fa32476fabf in vrf_disable lib/vrf.c:326 > FRRouting#8 0x7fa32476f5d4 in vrf_delete lib/vrf.c:231 > FRRouting#9 0x55b0e4e4ad36 in interface_vrf_change zebra/interface.c:1478 > FRRouting#10 0x55b0e4e4d5d2 in zebra_if_dplane_ifp_handling zebra/interface.c:1949 > FRRouting#11 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#12 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#13 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#14 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#15 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#16 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > > previously allocated by thread T0 here: > #0 0x7fa324caa037 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > #1 0x7fa324668c4d in qcalloc lib/memory.c:105 > #2 0x7fa32474bf33 in route_table_init_with_delegate lib/table.c:38 > #3 0x7fa32474e73c in route_table_init lib/table.c:512 > FRRouting#4 0x55b0e4fbc353 in zebra_router_get_table zebra/zebra_router.c:137 > FRRouting#5 0x55b0e4fd4da0 in zebra_vrf_table_create zebra/zebra_vrf.c:358 > FRRouting#6 0x55b0e4fd3d30 in zebra_vrf_enable zebra/zebra_vrf.c:140 > FRRouting#7 0x7fa32476f9b2 in vrf_enable lib/vrf.c:286 > FRRouting#8 0x55b0e4e4af76 in interface_vrf_change zebra/interface.c:1533 > FRRouting#9 0x55b0e4e4d612 in zebra_if_dplane_ifp_handling zebra/interface.c:1968 > FRRouting#10 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#11 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#12 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#13 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#14 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#15 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 Fixes: d8612e6 ("zebra: Track tables allocated by vrf and cleanup") Signed-off-by: Louis Scalbert <[email protected]>

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > FRRouting#4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > FRRouting#5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > FRRouting#6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > FRRouting#7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > FRRouting#8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > FRRouting#9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > FRRouting#10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > FRRouting#11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > FRRouting#12 0x7f26f275bae4 in route_table_free lib/table.c:111 > FRRouting#13 0x7f26f275b749 in route_table_finish lib/table.c:46 > FRRouting#14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > FRRouting#16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > FRRouting#17 0x7f26f2777108 in event_call lib/event.c:2011 > FRRouting#18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > FRRouting#19 0x55910c4f49cb in main zebra/main.c:531 > FRRouting#20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]>

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > FRRouting#1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > FRRouting#2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > FRRouting#3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > FRRouting#4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > FRRouting#5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > FRRouting#6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > FRRouting#7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > FRRouting#8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > FRRouting#9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > FRRouting#10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > FRRouting#11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > FRRouting#12 0x7f26f275bae4 in route_table_free lib/table.c:111 > FRRouting#13 0x7f26f275b749 in route_table_finish lib/table.c:46 > FRRouting#14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > FRRouting#16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > FRRouting#17 0x7f26f2777108 in event_call lib/event.c:2011 > FRRouting#18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > FRRouting#19 0x55910c4f49cb in main zebra/main.c:531 > FRRouting#20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]>

…d: fix show bgp all with evpn Merge in HARDWARE/frr from psuchy/fix_show_bgp_all to akamai/debian/frr-8.4.2 Squashed commit of the following: commit 094f403d1c900e232ac009f3ac0047dfd652c58e Author: Louis Scalbert <[email protected]> Date: Thu Dec 29 16:50:54 2022 +0100 bgpd: fix show bgp all with evpn Fix crash on "show bgp all" when BGP EVPN is set. > #0 raise (sig=11) at ../sysdeps/unix/sysv/linux/raise.c:50 > #1 0x00007fdfe03cf53c in core_handler (signo=11, siginfo=0x7ffdebbffe30, context=0x7ffdebbffd00) at lib/sigevent.c:261 > FRRouting#2 <signal handler called> > FRRouting#3 0x00000000004d4fec in bgp_attr_get_community (attr=0x41) at bgpd/bgp_attr.h:553 > FRRouting#4 0x00000000004eee84 in bgp_show_table (vty=0x1a790d0, bgp=0x19d0a00, safi=SAFI_EVPN, table=0x19f6010, type=bgp_show_type_normal, output_arg=0x0, rd=0x0, is_last=1, output_cum=0x0, > total_cum=0x0, json_header_depth=0x7ffdebc00bf8, show_flags=4, rpki_target_state=RPKI_NOT_BEING_USED) at bgpd/bgp_route.c:11329 > FRRouting#5 0x00000000004f7765 in bgp_show (vty=0x1a790d0, bgp=0x19d0a00, afi=AFI_L2VPN, safi=SAFI_EVPN, type=bgp_show_type_normal, output_arg=0x0, show_flags=4, > rpki_target_state=RPKI_NOT_BEING_USED) at bgpd/bgp_route.c:11814 > FRRouting#6 0x00000000004fb53b in show_ip_bgp_magic (self=0x6752b0 <show_ip_bgp_cmd>, vty=0x1a790d0, argc=3, argv=0x19cb050, viewvrfname=0x0, all=0x1395390 "all", aa_nn=0x0, community_list=0, > community_list_str=0x0, community_list_name=0x0, as_path_filter_name=0x0, prefix_list=0x0, accesslist_name=0x0, rmap_name=0x0, version=0, version_str=0x0, alias_name=0x0, > orr_group_name=0x0, detail_routes=0x0, uj=0x0, detail_json=0x0, wide=0x0) at bgpd/bgp_route.c:13040 > FRRouting#7 0x00000000004fa322 in show_ip_bgp (self=0x6752b0 <show_ip_bgp_cmd>, vty=0x1a790d0, argc=3, argv=0x19cb050) at ./bgpd/bgp_route_clippy.c:519 > FRRouting#8 0x00007fdfe033ccc8 in cmd_execute_command_real (vline=0x19c9300, filter=FILTER_RELAXED, vty=0x1a790d0, cmd=0x0, up_level=0) at lib/command.c:996 > FRRouting#9 0x00007fdfe033c739 in cmd_execute_command (vline=0x19c9300, vty=0x1a790d0, cmd=0x0, vtysh=0) at lib/command.c:1056 > FRRouting#10 0x00007fdfe033cdf5 in cmd_execute (vty=0x1a790d0, cmd=0x19c9eb0 "show bgp all", matched=0x0, vtysh=0) at lib/command.c:1223 > FRRouting#11 0x00007fdfe03f65c6 in vty_command (vty=0x1a790d0, buf=0x19c9eb0 "show bgp all") at lib/vty.c:486 > FRRouting#12 0x00007fdfe03f603b in vty_execute (vty=0x1a790d0) at lib/vty.c:1249 > FRRouting#13 0x00007fdfe03f533b in vtysh_read (thread=0x7ffdebc03838) at lib/vty.c:2148 > FRRouting#14 0x00007fdfe03e815d in thread_call (thread=0x7ffdebc03838) at lib/thread.c:2006 > FRRouting#15 0x00007fdfe0379b54 in frr_run (master=0x1246880) at lib/libfrr.c:1198 > FRRouting#16 0x000000000042b2a8 in main (argc=7, argv=0x7ffdebc03af8) at bgpd/bgp_main.c:520 Link: FRRouting#12576 Signed-off-by: Louis Scalbert <[email protected]>

When a failover happens on ECMP paths that use the same nexthop which is recursively resolved, ZEBRA replaces the old NHG with a new one, and updates the pointer of all routes using that nexthop. Actually, if only the recursive nexthop changed, there is no need to replace the old NHG. Modify the zebra_nhg_proto_add() function, by updating the recursive nexthop on the original NHG. Using this change replaces the old method that was consisting in allocating a new nhe. This change triggers an ASAN in the bgp_nhg_zapi_scalability test, function test_bgp_ipv4_simulate_r5_machine_going_down(). > ==1195107==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000de580 at pc 0x55b6b7d55d8e bp 0x7fffd81977a0 sp 0x7fffd8197790 > READ of size 4 at 0x60e0000de580 thread T0 > #0 0x55b6b7d55d8d in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1858 > FRRouting#1 0x55b6b7d55fee in zebra_nhg_free_members zebra/zebra_nhg.c:1752 > FRRouting#2 0x55b6b7d55fee in zebra_nhg_free zebra/zebra_nhg.c:1772 > FRRouting#3 0x55b6b7d59215 in zebra_nhg_proto_add zebra/zebra_nhg.c:3883 > FRRouting#4 0x55b6b7d83615 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#5 0x55b6b7d83615 in process_subq zebra/zebra_rib.c:3344 > FRRouting#6 0x55b6b7d83615 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#7 0x7fe57a916fef in work_queue_run lib/workqueue.c:282 > FRRouting#8 0x7fe57a8f863b in event_call lib/event.c:1996 > FRRouting#9 0x7fe57a81e527 in frr_run lib/libfrr.c:1237 > FRRouting#10 0x55b6b7c40c74 in main zebra/main.c:526 > FRRouting#11 0x7fe57a229d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#12 0x7fe57a229e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#13 0x55b6b7c43b84 in _start (/usr/lib/frr/zebra+0x1adb84) > > 0x60e0000de580 is located 96 bytes inside of 160-byte region [0x60e0000de520,0x60e0000de5c0) > freed by thread T0 here: > #0 0x7fe57acb4537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127 > FRRouting#1 0x55b6b7d59628 in zebra_nhg_proto_add zebra/zebra_nhg.c:3876 > FRRouting#2 0x55b6b7d83615 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#3 0x55b6b7d83615 in process_subq zebra/zebra_rib.c:3344 > FRRouting#4 0x55b6b7d83615 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#5 0x7fe57a916fef in work_queue_run lib/workqueue.c:282 > FRRouting#6 0x7fe57a8f863b in event_call lib/event.c:1996 > FRRouting#7 0x7fe57a81e527 in frr_run lib/libfrr.c:1237 > FRRouting#8 0x55b6b7c40c74 in main zebra/main.c:526 > FRRouting#9 0x7fe57a229d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > previously allocated by thread T0 here: > #0 0x7fe57acb4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > FRRouting#1 0x7fe57a83e98e in qcalloc lib/memory.c:106 > FRRouting#2 0x55b6b7d5149e in zebra_nhg_alloc zebra/zebra_nhg.c:392 > FRRouting#3 0x55b6b7d5149e in zebra_nhe_copy zebra/zebra_nhg.c:499 > FRRouting#4 0x55b6b7d5181f in zebra_nhg_hash_alloc zebra/zebra_nhg.c:538 > FRRouting#5 0x7fe57a7fbf0d in hash_get lib/hash.c:147 > FRRouting#6 0x55b6b7d542ea in zebra_nhe_find zebra/zebra_nhg.c:832 > FRRouting#7 0x55b6b7d5495f in zebra_nhg_find zebra/zebra_nhg.c:1014 > FRRouting#8 0x55b6b7d54dcd in zebra_nhg_find_nexthop zebra/zebra_nhg.c:1031 > FRRouting#9 0x55b6b7d535e8 in depends_find_recursive zebra/zebra_nhg.c:1514 > FRRouting#10 0x55b6b7d535e8 in depends_find zebra/zebra_nhg.c:1563 > FRRouting#11 0x55b6b7d535e8 in depends_find_add zebra/zebra_nhg.c:1602 > FRRouting#12 0x55b6b7d59884 in zebra_nhg_update_nhe zebra/zebra_nhg.c:3738 > FRRouting#13 0x55b6b7d59884 in zebra_nhg_proto_add zebra/zebra_nhg.c:3844 > FRRouting#14 0x55b6b7d83615 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#15 0x55b6b7d83615 in process_subq zebra/zebra_rib.c:3344 > FRRouting#16 0x55b6b7d83615 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#17 0x7fe57a916fef in work_queue_run lib/workqueue.c:282 > FRRouting#18 0x7fe57a8f863b in event_call lib/event.c:1996 > FRRouting#19 0x7fe57a81e527 in frr_run lib/libfrr.c:1237 > FRRouting#20 0x55b6b7c40c74 in main zebra/main.c:526 > FRRouting#21 0x7fe57a229d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > SUMMARY: AddressSanitizer: heap-use-after-free zebra/zebra_nhg.c:1858 in zebra_nhg_decrement_ref > Shadow bytes around the buggy address: > 0x0c1c80013c60: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013c70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013c80: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > 0x0c1c80013c90: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa > 0x0c1c80013ca0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd > =>0x0c1c80013cb0:[fd]fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > 0x0c1c80013cc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c80013cd0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013ce0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013cf0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 > 0x0c1c80013d00: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > Shadow gap: cc > ==1195107==ABORTING > Signed-off-by: Philippe Guibert <[email protected]>

A general flush is done on the nhg depend of the protocol nexthop group. Actually, the NHG should not be removed, if there are routes attached to it. In the same time, it seems the route count does not propagate to the nhg_depends. The con of this method is that there is still ASAN, and by comparing the refcount value of the old way (allocation), the count is less than expectd, for nexthop group with route count only: Allocation method in proto_add(): > 2024/10/14 10:57:24.915401 ZEBRA: [VB8P9-5F2GE] zebra_nhg_proto_add: BEFORE NHE 71428576, (71428576[39/49/59]) cnt 2002 > 2024/10/14 10:57:24.915510 ZEBRA: [HCTBK-W37K2] zebra_nhg_proto_add: NHE 71428576, (71428576[49/59/65]) cnt 1 > 2024/10/14 10:57:24.915513 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 49, (49[50]) cnt 2012 > 2024/10/14 10:57:24.915515 ZEBRA: [VP9H1-EV2BN] (71428573) > 2024/10/14 10:57:24.915515 ZEBRA: [VP9H1-EV2BN] (71428574) > 2024/10/14 10:57:24.915516 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:57:24.915517 ZEBRA: [VP9H1-EV2BN] (71428578) > 2024/10/14 10:57:24.915517 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 59, (59[60]) cnt 2007 > 2024/10/14 10:57:24.915519 ZEBRA: [VP9H1-EV2BN] (71428575) > 2024/10/14 10:57:24.915519 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:57:24.915520 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 65, (65[42]) cnt 4 > 2024/10/14 10:57:24.915521 ZEBRA: [VP9H1-EV2BN] (71428571) > 2024/10/14 10:57:24.915522 ZEBRA: [VP9H1-EV2BN] (71428576) Method using general flush, but keep old pointer: > 2024/10/14 10:51:17.229799 ZEBRA: [VB8P9-5F2GE] zebra_nhg_proto_add: BEFORE NHE 71428576, (71428576[39/49/59]) cnt 2002 > 2024/10/14 10:51:17.229909 ZEBRA: [HCTBK-W37K2] zebra_nhg_proto_add: NHE 71428576, (71428576[49/59/65]) cnt 2002 > 2024/10/14 10:51:17.229912 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 49, (49[50]) cnt 2011 > 2024/10/14 10:51:17.229914 ZEBRA: [VP9H1-EV2BN] (71428573) > 2024/10/14 10:51:17.229915 ZEBRA: [VP9H1-EV2BN] (71428574) > 2024/10/14 10:51:17.229915 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:51:17.229916 ZEBRA: [VP9H1-EV2BN] (71428578) > 2024/10/14 10:51:17.229916 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 59, (59[60]) cnt 2006 > 2024/10/14 10:51:17.229918 ZEBRA: [VP9H1-EV2BN] (71428575) > 2024/10/14 10:51:17.229918 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:51:17.229919 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 65, (65[42]) cnt 4 > 2024/10/14 10:51:17.229920 ZEBRA: [VP9H1-EV2BN] (71428571) > 2024/10/14 10:51:17.229921 ZEBRA: [VP9H1-EV2BN] (71428576) Resulting ASAN error when running bgp_nhg_zapi_notification, on the test_bgp_ipv4_simulate_r5_machine_going_down() function: > r1: zebra triggered an exception by AddressSanitizer > AddressSanitizer error in topotest `test_bgp_nhg_zapi_scalability.py`, test `teardown_module`, router `r1` > > ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000de580 at pc 0x558a7d98cd8e bp 0x7fff4915a6e0 sp 0x7fff4915a6d0 > READ of size 4 at 0x60e0000de580 thread T0 > #0 0x558a7d98cd8d in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1858 > FRRouting#1 0x558a7d98cfee in zebra_nhg_free_members zebra/zebra_nhg.c:1752 > FRRouting#2 0x558a7d98cfee in zebra_nhg_free zebra/zebra_nhg.c:1772 > FRRouting#3 0x558a7d9901ff in zebra_nhg_proto_add zebra/zebra_nhg.c:3861 > FRRouting#4 0x558a7d9ba365 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#5 0x558a7d9ba365 in process_subq zebra/zebra_rib.c:3344 > FRRouting#6 0x558a7d9ba365 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#7 0x7fa262f16fef in work_queue_run lib/workqueue.c:282 > FRRouting#8 0x7fa262ef863b in event_call lib/event.c:1996 > FRRouting#9 0x7fa262e1e527 in frr_run lib/libfrr.c:1237 > FRRouting#10 0x558a7d877c74 in main zebra/main.c:526 > FRRouting#11 0x7fa262829d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#12 0x7fa262829e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#13 0x558a7d87ab84 in _start (/usr/lib/frr/zebra+0x1acb84) > > 0x60e0000de580 is located 96 bytes inside of 160-byte region [0x60e0000de520,0x60e0000de5c0) > freed by thread T0 here: > #0 0x7fa2632b4537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127 > FRRouting#1 0x558a7d9908a1 in zebra_nhg_proto_add zebra/zebra_nhg.c:3854 > FRRouting#2 0x558a7d9ba365 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#3 0x558a7d9ba365 in process_subq zebra/zebra_rib.c:3344 > FRRouting#4 0x558a7d9ba365 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#5 0x7fa262f16fef in work_queue_run lib/workqueue.c:282 > FRRouting#6 0x7fa262ef863b in event_call lib/event.c:1996 > FRRouting#7 0x7fa262e1e527 in frr_run lib/libfrr.c:1237 > FRRouting#8 0x558a7d877c74 in main zebra/main.c:526 > FRRouting#9 0x7fa262829d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > previously allocated by thread T0 here: > #0 0x7fa2632b4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > FRRouting#1 0x7fa262e3e98e in qcalloc lib/memory.c:106 > FRRouting#2 0x558a7d98849e in zebra_nhg_alloc zebra/zebra_nhg.c:392 > FRRouting#3 0x558a7d98849e in zebra_nhe_copy zebra/zebra_nhg.c:499 > FRRouting#4 0x558a7d98881f in zebra_nhg_hash_alloc zebra/zebra_nhg.c:538 > FRRouting#5 0x7fa262dfbf0d in hash_get lib/hash.c:147 > FRRouting#6 0x558a7d98b2ea in zebra_nhe_find zebra/zebra_nhg.c:832 > FRRouting#7 0x558a7d98b95f in zebra_nhg_find zebra/zebra_nhg.c:1014 > FRRouting#8 0x558a7d98bdcd in zebra_nhg_find_nexthop zebra/zebra_nhg.c:1031 > FRRouting#9 0x558a7d98a5e8 in depends_find_recursive zebra/zebra_nhg.c:1514 > FRRouting#10 0x558a7d98a5e8 in depends_find zebra/zebra_nhg.c:1563 > FRRouting#11 0x558a7d98a5e8 in depends_find_add zebra/zebra_nhg.c:1602 > FRRouting#12 0x558a7d990378 in zebra_nhg_update_nhe zebra/zebra_nhg.c:3739 > FRRouting#13 0x558a7d990378 in zebra_nhg_proto_add zebra/zebra_nhg.c:3822 > FRRouting#14 0x558a7d9ba365 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#15 0x558a7d9ba365 in process_subq zebra/zebra_rib.c:3344 > FRRouting#16 0x558a7d9ba365 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#17 0x7fa262f16fef in work_queue_run lib/workqueue.c:282 > FRRouting#18 0x7fa262ef863b in event_call lib/event.c:1996 > FRRouting#19 0x7fa262e1e527 in frr_run lib/libfrr.c:1237 > FRRouting#20 0x558a7d877c74 in main zebra/main.c:526 > FRRouting#21 0x7fa262829d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > SUMMARY: AddressSanitizer: heap-use-after-free zebra/zebra_nhg.c:1858 in zebra_nhg_decrement_ref > Shadow bytes around the buggy address: > 0x0c1c80013c60: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013c70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013c80: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > 0x0c1c80013c90: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa > 0x0c1c80013ca0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd > =>0x0c1c80013cb0:[fd]fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > 0x0c1c80013cc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c80013cd0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013ce0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013cf0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 > 0x0c1c80013d00: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > Shadow gap: cc > Signed-off-by: Philippe Guibert <[email protected]>

When a failover happens on ECMP paths that use the same nexthop which is recursively resolved, ZEBRA replaces the old NHG with a new one, and updates the pointer of all routes using that nexthop. Actually, if only the recursive nexthop changed, there is no need to replace the old NHG. Modify the zebra_nhg_proto_add() function, by updating the recursive nexthop on the original NHG. Using this change replaces the old method that was consisting in allocating a new nhe. This change triggers an ASAN in the bgp_nhg_zapi_scalability test, function test_bgp_ipv4_simulate_r5_machine_going_down(). > ==1195107==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000de580 at pc 0x55b6b7d55d8e bp 0x7fffd81977a0 sp 0x7fffd8197790 > READ of size 4 at 0x60e0000de580 thread T0 > #0 0x55b6b7d55d8d in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1858 > FRRouting#1 0x55b6b7d55fee in zebra_nhg_free_members zebra/zebra_nhg.c:1752 > FRRouting#2 0x55b6b7d55fee in zebra_nhg_free zebra/zebra_nhg.c:1772 > FRRouting#3 0x55b6b7d59215 in zebra_nhg_proto_add zebra/zebra_nhg.c:3883 > FRRouting#4 0x55b6b7d83615 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#5 0x55b6b7d83615 in process_subq zebra/zebra_rib.c:3344 > FRRouting#6 0x55b6b7d83615 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#7 0x7fe57a916fef in work_queue_run lib/workqueue.c:282 > FRRouting#8 0x7fe57a8f863b in event_call lib/event.c:1996 > FRRouting#9 0x7fe57a81e527 in frr_run lib/libfrr.c:1237 > FRRouting#10 0x55b6b7c40c74 in main zebra/main.c:526 > FRRouting#11 0x7fe57a229d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#12 0x7fe57a229e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#13 0x55b6b7c43b84 in _start (/usr/lib/frr/zebra+0x1adb84) > > 0x60e0000de580 is located 96 bytes inside of 160-byte region [0x60e0000de520,0x60e0000de5c0) > freed by thread T0 here: > #0 0x7fe57acb4537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127 > FRRouting#1 0x55b6b7d59628 in zebra_nhg_proto_add zebra/zebra_nhg.c:3876 > FRRouting#2 0x55b6b7d83615 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#3 0x55b6b7d83615 in process_subq zebra/zebra_rib.c:3344 > FRRouting#4 0x55b6b7d83615 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#5 0x7fe57a916fef in work_queue_run lib/workqueue.c:282 > FRRouting#6 0x7fe57a8f863b in event_call lib/event.c:1996 > FRRouting#7 0x7fe57a81e527 in frr_run lib/libfrr.c:1237 > FRRouting#8 0x55b6b7c40c74 in main zebra/main.c:526 > FRRouting#9 0x7fe57a229d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > previously allocated by thread T0 here: > #0 0x7fe57acb4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > FRRouting#1 0x7fe57a83e98e in qcalloc lib/memory.c:106 > FRRouting#2 0x55b6b7d5149e in zebra_nhg_alloc zebra/zebra_nhg.c:392 > FRRouting#3 0x55b6b7d5149e in zebra_nhe_copy zebra/zebra_nhg.c:499 > FRRouting#4 0x55b6b7d5181f in zebra_nhg_hash_alloc zebra/zebra_nhg.c:538 > FRRouting#5 0x7fe57a7fbf0d in hash_get lib/hash.c:147 > FRRouting#6 0x55b6b7d542ea in zebra_nhe_find zebra/zebra_nhg.c:832 > FRRouting#7 0x55b6b7d5495f in zebra_nhg_find zebra/zebra_nhg.c:1014 > FRRouting#8 0x55b6b7d54dcd in zebra_nhg_find_nexthop zebra/zebra_nhg.c:1031 > FRRouting#9 0x55b6b7d535e8 in depends_find_recursive zebra/zebra_nhg.c:1514 > FRRouting#10 0x55b6b7d535e8 in depends_find zebra/zebra_nhg.c:1563 > FRRouting#11 0x55b6b7d535e8 in depends_find_add zebra/zebra_nhg.c:1602 > FRRouting#12 0x55b6b7d59884 in zebra_nhg_update_nhe zebra/zebra_nhg.c:3738 > FRRouting#13 0x55b6b7d59884 in zebra_nhg_proto_add zebra/zebra_nhg.c:3844 > FRRouting#14 0x55b6b7d83615 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#15 0x55b6b7d83615 in process_subq zebra/zebra_rib.c:3344 > FRRouting#16 0x55b6b7d83615 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#17 0x7fe57a916fef in work_queue_run lib/workqueue.c:282 > FRRouting#18 0x7fe57a8f863b in event_call lib/event.c:1996 > FRRouting#19 0x7fe57a81e527 in frr_run lib/libfrr.c:1237 > FRRouting#20 0x55b6b7c40c74 in main zebra/main.c:526 > FRRouting#21 0x7fe57a229d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > SUMMARY: AddressSanitizer: heap-use-after-free zebra/zebra_nhg.c:1858 in zebra_nhg_decrement_ref > Shadow bytes around the buggy address: > 0x0c1c80013c60: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013c70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013c80: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > 0x0c1c80013c90: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa > 0x0c1c80013ca0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd > =>0x0c1c80013cb0:[fd]fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > 0x0c1c80013cc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c80013cd0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013ce0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013cf0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 > 0x0c1c80013d00: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > Shadow gap: cc > ==1195107==ABORTING > Signed-off-by: Philippe Guibert <[email protected]>

A general flush is done on the nhg depend of the protocol nexthop group. Actually, the NHG should not be removed, if there are routes attached to it. In the same time, it seems the route count does not propagate to the nhg_depends. The con of this method is that there is still ASAN, and by comparing the refcount value of the old way (allocation), the count is less than expectd, for nexthop group with route count only: Allocation method in proto_add(): > 2024/10/14 10:57:24.915401 ZEBRA: [VB8P9-5F2GE] zebra_nhg_proto_add: BEFORE NHE 71428576, (71428576[39/49/59]) cnt 2002 > 2024/10/14 10:57:24.915510 ZEBRA: [HCTBK-W37K2] zebra_nhg_proto_add: NHE 71428576, (71428576[49/59/65]) cnt 1 > 2024/10/14 10:57:24.915513 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 49, (49[50]) cnt 2012 > 2024/10/14 10:57:24.915515 ZEBRA: [VP9H1-EV2BN] (71428573) > 2024/10/14 10:57:24.915515 ZEBRA: [VP9H1-EV2BN] (71428574) > 2024/10/14 10:57:24.915516 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:57:24.915517 ZEBRA: [VP9H1-EV2BN] (71428578) > 2024/10/14 10:57:24.915517 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 59, (59[60]) cnt 2007 > 2024/10/14 10:57:24.915519 ZEBRA: [VP9H1-EV2BN] (71428575) > 2024/10/14 10:57:24.915519 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:57:24.915520 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 65, (65[42]) cnt 4 > 2024/10/14 10:57:24.915521 ZEBRA: [VP9H1-EV2BN] (71428571) > 2024/10/14 10:57:24.915522 ZEBRA: [VP9H1-EV2BN] (71428576) Method using general flush, but keep old pointer: > 2024/10/14 10:51:17.229799 ZEBRA: [VB8P9-5F2GE] zebra_nhg_proto_add: BEFORE NHE 71428576, (71428576[39/49/59]) cnt 2002 > 2024/10/14 10:51:17.229909 ZEBRA: [HCTBK-W37K2] zebra_nhg_proto_add: NHE 71428576, (71428576[49/59/65]) cnt 2002 > 2024/10/14 10:51:17.229912 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 49, (49[50]) cnt 2011 > 2024/10/14 10:51:17.229914 ZEBRA: [VP9H1-EV2BN] (71428573) > 2024/10/14 10:51:17.229915 ZEBRA: [VP9H1-EV2BN] (71428574) > 2024/10/14 10:51:17.229915 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:51:17.229916 ZEBRA: [VP9H1-EV2BN] (71428578) > 2024/10/14 10:51:17.229916 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 59, (59[60]) cnt 2006 > 2024/10/14 10:51:17.229918 ZEBRA: [VP9H1-EV2BN] (71428575) > 2024/10/14 10:51:17.229918 ZEBRA: [VP9H1-EV2BN] (71428576) > 2024/10/14 10:51:17.229919 ZEBRA: [RM3ZQ-V7JN5] zebra_nhg_proto_add: NHE 65, (65[42]) cnt 4 > 2024/10/14 10:51:17.229920 ZEBRA: [VP9H1-EV2BN] (71428571) > 2024/10/14 10:51:17.229921 ZEBRA: [VP9H1-EV2BN] (71428576) Resulting ASAN error when running bgp_nhg_zapi_notification, on the test_bgp_ipv4_simulate_r5_machine_going_down() function: > r1: zebra triggered an exception by AddressSanitizer > AddressSanitizer error in topotest `test_bgp_nhg_zapi_scalability.py`, test `teardown_module`, router `r1` > > ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000de580 at pc 0x558a7d98cd8e bp 0x7fff4915a6e0 sp 0x7fff4915a6d0 > READ of size 4 at 0x60e0000de580 thread T0 > #0 0x558a7d98cd8d in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1858 > FRRouting#1 0x558a7d98cfee in zebra_nhg_free_members zebra/zebra_nhg.c:1752 > FRRouting#2 0x558a7d98cfee in zebra_nhg_free zebra/zebra_nhg.c:1772 > FRRouting#3 0x558a7d9901ff in zebra_nhg_proto_add zebra/zebra_nhg.c:3861 > FRRouting#4 0x558a7d9ba365 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#5 0x558a7d9ba365 in process_subq zebra/zebra_rib.c:3344 > FRRouting#6 0x558a7d9ba365 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#7 0x7fa262f16fef in work_queue_run lib/workqueue.c:282 > FRRouting#8 0x7fa262ef863b in event_call lib/event.c:1996 > FRRouting#9 0x7fa262e1e527 in frr_run lib/libfrr.c:1237 > FRRouting#10 0x558a7d877c74 in main zebra/main.c:526 > FRRouting#11 0x7fa262829d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#12 0x7fa262829e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#13 0x558a7d87ab84 in _start (/usr/lib/frr/zebra+0x1acb84) > > 0x60e0000de580 is located 96 bytes inside of 160-byte region [0x60e0000de520,0x60e0000de5c0) > freed by thread T0 here: > #0 0x7fa2632b4537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127 > FRRouting#1 0x558a7d9908a1 in zebra_nhg_proto_add zebra/zebra_nhg.c:3854 > FRRouting#2 0x558a7d9ba365 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#3 0x558a7d9ba365 in process_subq zebra/zebra_rib.c:3344 > FRRouting#4 0x558a7d9ba365 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#5 0x7fa262f16fef in work_queue_run lib/workqueue.c:282 > FRRouting#6 0x7fa262ef863b in event_call lib/event.c:1996 > FRRouting#7 0x7fa262e1e527 in frr_run lib/libfrr.c:1237 > FRRouting#8 0x558a7d877c74 in main zebra/main.c:526 > FRRouting#9 0x7fa262829d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > previously allocated by thread T0 here: > #0 0x7fa2632b4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > FRRouting#1 0x7fa262e3e98e in qcalloc lib/memory.c:106 > FRRouting#2 0x558a7d98849e in zebra_nhg_alloc zebra/zebra_nhg.c:392 > FRRouting#3 0x558a7d98849e in zebra_nhe_copy zebra/zebra_nhg.c:499 > FRRouting#4 0x558a7d98881f in zebra_nhg_hash_alloc zebra/zebra_nhg.c:538 > FRRouting#5 0x7fa262dfbf0d in hash_get lib/hash.c:147 > FRRouting#6 0x558a7d98b2ea in zebra_nhe_find zebra/zebra_nhg.c:832 > FRRouting#7 0x558a7d98b95f in zebra_nhg_find zebra/zebra_nhg.c:1014 > FRRouting#8 0x558a7d98bdcd in zebra_nhg_find_nexthop zebra/zebra_nhg.c:1031 > FRRouting#9 0x558a7d98a5e8 in depends_find_recursive zebra/zebra_nhg.c:1514 > FRRouting#10 0x558a7d98a5e8 in depends_find zebra/zebra_nhg.c:1563 > FRRouting#11 0x558a7d98a5e8 in depends_find_add zebra/zebra_nhg.c:1602 > FRRouting#12 0x558a7d990378 in zebra_nhg_update_nhe zebra/zebra_nhg.c:3739 > FRRouting#13 0x558a7d990378 in zebra_nhg_proto_add zebra/zebra_nhg.c:3822 > FRRouting#14 0x558a7d9ba365 in process_subq_nhg zebra/zebra_rib.c:2738 > FRRouting#15 0x558a7d9ba365 in process_subq zebra/zebra_rib.c:3344 > FRRouting#16 0x558a7d9ba365 in meta_queue_process zebra/zebra_rib.c:3397 > FRRouting#17 0x7fa262f16fef in work_queue_run lib/workqueue.c:282 > FRRouting#18 0x7fa262ef863b in event_call lib/event.c:1996 > FRRouting#19 0x7fa262e1e527 in frr_run lib/libfrr.c:1237 > FRRouting#20 0x558a7d877c74 in main zebra/main.c:526 > FRRouting#21 0x7fa262829d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > SUMMARY: AddressSanitizer: heap-use-after-free zebra/zebra_nhg.c:1858 in zebra_nhg_decrement_ref > Shadow bytes around the buggy address: > 0x0c1c80013c60: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013c70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013c80: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > 0x0c1c80013c90: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa > 0x0c1c80013ca0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd > =>0x0c1c80013cb0:[fd]fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > 0x0c1c80013cc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c80013cd0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c80013ce0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c80013cf0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 > 0x0c1c80013d00: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > Shadow gap: cc > Signed-off-by: Philippe Guibert <[email protected]>

Fix a heap-after-free that causes zebra to crash even without address-sanitizer. To reproduce: > echo "100 my_table" | tee -a /etc/iproute2/rt_tables > ip route add blackhole default table 100 > ip route show table 100 > ip l add red type vrf table 100 > ip l del red > ip route del blackhole default table 100 Zebra manages routing tables for all existing Linux RT tables, regardless of whether they are assigned to a VRF interface. When a table is not assigned to any VRF, zebra arbitrarily assigns it to the default VRF, even though this is not strictly accurate (the code expects this behavior). When an RT table is created after a VRF, zebra correctly assigns the table to the VRF. However, if a VRF interface is assigned to an existing RT table, zebra does not update the table owner, which remains as the default VRF. As a result, existing routing entries remain under the default VRF, while new entries are correctly assigned to the VRF. The VRF mismatch is unexpected in the code and creates crashes and memory related issues. Furthermore, Linux does not automatically delete RT tables when they are unassigned from a VRF. It is incorrect to delete these tables from zebra. Instead, at VRF disabling, do not release the table but reassign it to the default VRF. At VRF enabling, change the table owner back to the appropriate VRF. > ==2866266==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000154f54 at pc 0x7fa32474b83f bp 0x7ffe94f67d90 sp 0x7ffe94f67d88 > READ of size 1 at 0x606000154f54 thread T0 > #0 0x7fa32474b83e in rn_hash_node_const_find lib/table.c:28 > #1 0x7fa32474bab1 in rn_hash_node_find lib/table.c:28 > #2 0x7fa32474d783 in route_node_get lib/table.c:283 > #3 0x7fa3247328dd in srcdest_rnode_get lib/srcdest_table.c:231 > FRRouting#4 0x55b0e4fa8da4 in rib_find_rn_from_ctx zebra/zebra_rib.c:1957 > FRRouting#5 0x55b0e4fa8e31 in rib_process_result zebra/zebra_rib.c:1988 > FRRouting#6 0x55b0e4fb9d64 in rib_process_dplane_results zebra/zebra_rib.c:4894 > FRRouting#7 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#8 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#9 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#10 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > FRRouting#11 0x55b0e4e2d649 in _start (/usr/lib/frr/zebra+0x1a1649) > > 0x606000154f54 is located 20 bytes inside of 56-byte region [0x606000154f40,0x606000154f78) > freed by thread T0 here: > #0 0x7fa324ca9b6f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:123 > #1 0x7fa324668d8f in qfree lib/memory.c:130 > #2 0x7fa32474c421 in route_table_free lib/table.c:126 > #3 0x7fa32474bf96 in route_table_finish lib/table.c:46 > FRRouting#4 0x55b0e4fbca3a in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#5 0x55b0e4fbccea in zebra_router_release_table zebra/zebra_router.c:214 > FRRouting#6 0x55b0e4fd428e in zebra_vrf_disable zebra/zebra_vrf.c:219 > FRRouting#7 0x7fa32476fabf in vrf_disable lib/vrf.c:326 > FRRouting#8 0x7fa32476f5d4 in vrf_delete lib/vrf.c:231 > FRRouting#9 0x55b0e4e4ad36 in interface_vrf_change zebra/interface.c:1478 > FRRouting#10 0x55b0e4e4d5d2 in zebra_if_dplane_ifp_handling zebra/interface.c:1949 > FRRouting#11 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#12 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#13 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#14 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#15 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#16 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > > previously allocated by thread T0 here: > #0 0x7fa324caa037 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > #1 0x7fa324668c4d in qcalloc lib/memory.c:105 > #2 0x7fa32474bf33 in route_table_init_with_delegate lib/table.c:38 > #3 0x7fa32474e73c in route_table_init lib/table.c:512 > FRRouting#4 0x55b0e4fbc353 in zebra_router_get_table zebra/zebra_router.c:137 > FRRouting#5 0x55b0e4fd4da0 in zebra_vrf_table_create zebra/zebra_vrf.c:358 > FRRouting#6 0x55b0e4fd3d30 in zebra_vrf_enable zebra/zebra_vrf.c:140 > FRRouting#7 0x7fa32476f9b2 in vrf_enable lib/vrf.c:286 > FRRouting#8 0x55b0e4e4af76 in interface_vrf_change zebra/interface.c:1533 > FRRouting#9 0x55b0e4e4d612 in zebra_if_dplane_ifp_handling zebra/interface.c:1968 > FRRouting#10 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#11 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#12 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#13 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#14 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#15 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 Fixes: d8612e6 ("zebra: Track tables allocated by vrf and cleanup") Signed-off-by: Louis Scalbert <[email protected]>

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > FRRouting#4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > FRRouting#5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > FRRouting#6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > FRRouting#7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > FRRouting#8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > FRRouting#9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > FRRouting#10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > FRRouting#11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > FRRouting#12 0x7f26f275bae4 in route_table_free lib/table.c:111 > FRRouting#13 0x7f26f275b749 in route_table_finish lib/table.c:46 > FRRouting#14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > FRRouting#16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > FRRouting#17 0x7f26f2777108 in event_call lib/event.c:2011 > FRRouting#18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > FRRouting#19 0x55910c4f49cb in main zebra/main.c:531 > FRRouting#20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]>

Fix a heap-after-free that causes zebra to crash even without address-sanitizer. To reproduce: > echo "100 my_table" | tee -a /etc/iproute2/rt_tables > ip route add blackhole default table 100 > ip route show table 100 > ip l add red type vrf table 100 > ip l del red > ip route del blackhole default table 100 Zebra manages routing tables for all existing Linux RT tables, regardless of whether they are assigned to a VRF interface. When a table is not assigned to any VRF, zebra arbitrarily assigns it to the default VRF, even though this is not strictly accurate (the code expects this behavior). When an RT table is created after a VRF, zebra correctly assigns the table to the VRF. However, if a VRF interface is assigned to an existing RT table, zebra does not update the table owner, which remains as the default VRF. As a result, existing routing entries remain under the default VRF, while new entries are correctly assigned to the VRF. The VRF mismatch is unexpected in the code and creates crashes and memory related issues. Furthermore, Linux does not automatically delete RT tables when they are unassigned from a VRF. It is incorrect to delete these tables from zebra. Instead, at VRF disabling, do not release the table but reassign it to the default VRF. At VRF enabling, change the table owner back to the appropriate VRF. > ==2866266==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000154f54 at pc 0x7fa32474b83f bp 0x7ffe94f67d90 sp 0x7ffe94f67d88 > READ of size 1 at 0x606000154f54 thread T0 > #0 0x7fa32474b83e in rn_hash_node_const_find lib/table.c:28 > #1 0x7fa32474bab1 in rn_hash_node_find lib/table.c:28 > #2 0x7fa32474d783 in route_node_get lib/table.c:283 > #3 0x7fa3247328dd in srcdest_rnode_get lib/srcdest_table.c:231 > FRRouting#4 0x55b0e4fa8da4 in rib_find_rn_from_ctx zebra/zebra_rib.c:1957 > FRRouting#5 0x55b0e4fa8e31 in rib_process_result zebra/zebra_rib.c:1988 > FRRouting#6 0x55b0e4fb9d64 in rib_process_dplane_results zebra/zebra_rib.c:4894 > FRRouting#7 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#8 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#9 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#10 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > FRRouting#11 0x55b0e4e2d649 in _start (/usr/lib/frr/zebra+0x1a1649) > > 0x606000154f54 is located 20 bytes inside of 56-byte region [0x606000154f40,0x606000154f78) > freed by thread T0 here: > #0 0x7fa324ca9b6f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:123 > #1 0x7fa324668d8f in qfree lib/memory.c:130 > #2 0x7fa32474c421 in route_table_free lib/table.c:126 > #3 0x7fa32474bf96 in route_table_finish lib/table.c:46 > FRRouting#4 0x55b0e4fbca3a in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#5 0x55b0e4fbccea in zebra_router_release_table zebra/zebra_router.c:214 > FRRouting#6 0x55b0e4fd428e in zebra_vrf_disable zebra/zebra_vrf.c:219 > FRRouting#7 0x7fa32476fabf in vrf_disable lib/vrf.c:326 > FRRouting#8 0x7fa32476f5d4 in vrf_delete lib/vrf.c:231 > FRRouting#9 0x55b0e4e4ad36 in interface_vrf_change zebra/interface.c:1478 > FRRouting#10 0x55b0e4e4d5d2 in zebra_if_dplane_ifp_handling zebra/interface.c:1949 > FRRouting#11 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#12 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#13 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#14 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#15 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#16 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > > previously allocated by thread T0 here: > #0 0x7fa324caa037 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > #1 0x7fa324668c4d in qcalloc lib/memory.c:105 > #2 0x7fa32474bf33 in route_table_init_with_delegate lib/table.c:38 > #3 0x7fa32474e73c in route_table_init lib/table.c:512 > FRRouting#4 0x55b0e4fbc353 in zebra_router_get_table zebra/zebra_router.c:137 > FRRouting#5 0x55b0e4fd4da0 in zebra_vrf_table_create zebra/zebra_vrf.c:358 > FRRouting#6 0x55b0e4fd3d30 in zebra_vrf_enable zebra/zebra_vrf.c:140 > FRRouting#7 0x7fa32476f9b2 in vrf_enable lib/vrf.c:286 > FRRouting#8 0x55b0e4e4af76 in interface_vrf_change zebra/interface.c:1533 > FRRouting#9 0x55b0e4e4d612 in zebra_if_dplane_ifp_handling zebra/interface.c:1968 > FRRouting#10 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#11 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#12 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#13 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#14 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#15 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 Fixes: d8612e6 ("zebra: Track tables allocated by vrf and cleanup") Signed-off-by: Louis Scalbert <[email protected]>

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > FRRouting#4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > FRRouting#5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > FRRouting#6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > FRRouting#7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > FRRouting#8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > FRRouting#9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > FRRouting#10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > FRRouting#11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > FRRouting#12 0x7f26f275bae4 in route_table_free lib/table.c:111 > FRRouting#13 0x7f26f275b749 in route_table_finish lib/table.c:46 > FRRouting#14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > FRRouting#16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > FRRouting#17 0x7f26f2777108 in event_call lib/event.c:2011 > FRRouting#18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > FRRouting#19 0x55910c4f49cb in main zebra/main.c:531 > FRRouting#20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]>

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > #4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > #5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > #6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > #7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > #8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > #9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > #10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > #11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > #12 0x7f26f275bae4 in route_table_free lib/table.c:111 > #13 0x7f26f275b749 in route_table_finish lib/table.c:46 > #14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > #15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > #16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > #17 0x7f26f2777108 in event_call lib/event.c:2011 > #18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > #19 0x55910c4f49cb in main zebra/main.c:531 > #20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > #21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > #22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]> (cherry picked from commit 7ae70eb)

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > #4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > #5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > #6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > #7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > #8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > #9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > #10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > #11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > #12 0x7f26f275bae4 in route_table_free lib/table.c:111 > #13 0x7f26f275b749 in route_table_finish lib/table.c:46 > #14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > #15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > #16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > #17 0x7f26f2777108 in event_call lib/event.c:2011 > #18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > #19 0x55910c4f49cb in main zebra/main.c:531 > #20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > #21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > #22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]> (cherry picked from commit 7ae70eb) # Conflicts: # zebra/main.c # zebra/zebra_ns.h

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > #4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > #5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > #6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > #7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > #8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > #9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > #10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > #11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > #12 0x7f26f275bae4 in route_table_free lib/table.c:111 > #13 0x7f26f275b749 in route_table_finish lib/table.c:46 > #14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > #15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > #16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > #17 0x7f26f2777108 in event_call lib/event.c:2011 > #18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > #19 0x55910c4f49cb in main zebra/main.c:531 > #20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > #21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > #22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]> (cherry picked from commit 7ae70eb)

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > #4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > #5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > #6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > #7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > #8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > #9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > #10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > #11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > #12 0x7f26f275bae4 in route_table_free lib/table.c:111 > #13 0x7f26f275b749 in route_table_finish lib/table.c:46 > #14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > #15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > #16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > #17 0x7f26f2777108 in event_call lib/event.c:2011 > #18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > #19 0x55910c4f49cb in main zebra/main.c:531 > #20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > #21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > #22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]> (cherry picked from commit 7ae70eb) # Conflicts: # zebra/main.c # zebra/zebra_ns.h

Fix a heap-after-free that causes zebra to crash even without address-sanitizer. To reproduce: > echo "100 my_table" | tee -a /etc/iproute2/rt_tables > ip route add blackhole default table 100 > ip route show table 100 > ip l add red type vrf table 100 > ip l del red > ip route del blackhole default table 100 Zebra manages routing tables for all existing Linux RT tables, regardless of whether they are assigned to a VRF interface. When a table is not assigned to any VRF, zebra arbitrarily assigns it to the default VRF, even though this is not strictly accurate (the code expects this behavior). When an RT table is created after a VRF, zebra correctly assigns the table to the VRF. However, if a VRF interface is assigned to an existing RT table, zebra does not update the table owner, which remains as the default VRF. As a result, existing routing entries remain under the default VRF, while new entries are correctly assigned to the VRF. The VRF mismatch is unexpected in the code and creates crashes and memory related issues. Furthermore, Linux does not automatically delete RT tables when they are unassigned from a VRF. It is incorrect to delete these tables from zebra. Instead, at VRF disabling, do not release the table but reassign it to the default VRF. At VRF enabling, change the table owner back to the appropriate VRF. > ==2866266==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000154f54 at pc 0x7fa32474b83f bp 0x7ffe94f67d90 sp 0x7ffe94f67d88 > READ of size 1 at 0x606000154f54 thread T0 > #0 0x7fa32474b83e in rn_hash_node_const_find lib/table.c:28 > #1 0x7fa32474bab1 in rn_hash_node_find lib/table.c:28 > #2 0x7fa32474d783 in route_node_get lib/table.c:283 > #3 0x7fa3247328dd in srcdest_rnode_get lib/srcdest_table.c:231 > FRRouting#4 0x55b0e4fa8da4 in rib_find_rn_from_ctx zebra/zebra_rib.c:1957 > FRRouting#5 0x55b0e4fa8e31 in rib_process_result zebra/zebra_rib.c:1988 > FRRouting#6 0x55b0e4fb9d64 in rib_process_dplane_results zebra/zebra_rib.c:4894 > FRRouting#7 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#8 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#9 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#10 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > FRRouting#11 0x55b0e4e2d649 in _start (/usr/lib/frr/zebra+0x1a1649) > > 0x606000154f54 is located 20 bytes inside of 56-byte region [0x606000154f40,0x606000154f78) > freed by thread T0 here: > #0 0x7fa324ca9b6f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:123 > #1 0x7fa324668d8f in qfree lib/memory.c:130 > #2 0x7fa32474c421 in route_table_free lib/table.c:126 > #3 0x7fa32474bf96 in route_table_finish lib/table.c:46 > FRRouting#4 0x55b0e4fbca3a in zebra_router_free_table zebra/zebra_router.c:191 > FRRouting#5 0x55b0e4fbccea in zebra_router_release_table zebra/zebra_router.c:214 > FRRouting#6 0x55b0e4fd428e in zebra_vrf_disable zebra/zebra_vrf.c:219 > FRRouting#7 0x7fa32476fabf in vrf_disable lib/vrf.c:326 > FRRouting#8 0x7fa32476f5d4 in vrf_delete lib/vrf.c:231 > FRRouting#9 0x55b0e4e4ad36 in interface_vrf_change zebra/interface.c:1478 > FRRouting#10 0x55b0e4e4d5d2 in zebra_if_dplane_ifp_handling zebra/interface.c:1949 > FRRouting#11 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#12 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#13 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#14 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#15 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#16 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 > > previously allocated by thread T0 here: > #0 0x7fa324caa037 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > #1 0x7fa324668c4d in qcalloc lib/memory.c:105 > #2 0x7fa32474bf33 in route_table_init_with_delegate lib/table.c:38 > #3 0x7fa32474e73c in route_table_init lib/table.c:512 > FRRouting#4 0x55b0e4fbc353 in zebra_router_get_table zebra/zebra_router.c:137 > FRRouting#5 0x55b0e4fd4da0 in zebra_vrf_table_create zebra/zebra_vrf.c:358 > FRRouting#6 0x55b0e4fd3d30 in zebra_vrf_enable zebra/zebra_vrf.c:140 > FRRouting#7 0x7fa32476f9b2 in vrf_enable lib/vrf.c:286 > FRRouting#8 0x55b0e4e4af76 in interface_vrf_change zebra/interface.c:1533 > FRRouting#9 0x55b0e4e4d612 in zebra_if_dplane_ifp_handling zebra/interface.c:1968 > FRRouting#10 0x55b0e4e4fb89 in zebra_if_dplane_result zebra/interface.c:2268 > FRRouting#11 0x55b0e4fb9f26 in rib_process_dplane_results zebra/zebra_rib.c:4954 > FRRouting#12 0x7fa32476689c in event_call lib/event.c:1996 > FRRouting#13 0x7fa32463b7b2 in frr_run lib/libfrr.c:1232 > FRRouting#14 0x55b0e4e6c32a in main zebra/main.c:526 > FRRouting#15 0x7fa32424fd09 in __libc_start_main ../csu/libc-start.c:308 Fixes: d8612e6 ("zebra: Track tables allocated by vrf and cleanup") Signed-off-by: Louis Scalbert <[email protected]>

The following ASAN issue has been observed: > ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000acba4 at pc 0x55910c5694d0 bp 0x7ffe3a8ac850 sp 0x7ffe3a8ac840 > READ of size 4 at 0x6160000acba4 thread T0 > #0 0x55910c5694cf in ctx_info_from_zns zebra/zebra_dplane.c:3315 > #1 0x55910c569696 in dplane_ctx_ns_init zebra/zebra_dplane.c:3331 > #2 0x55910c56bf61 in dplane_ctx_nexthop_init zebra/zebra_dplane.c:3680 > #3 0x55910c5711ca in dplane_nexthop_update_internal zebra/zebra_dplane.c:4490 > #4 0x55910c571c5c in dplane_nexthop_delete zebra/zebra_dplane.c:4717 > #5 0x55910c61e90e in zebra_nhg_uninstall_kernel zebra/zebra_nhg.c:3413 > #6 0x55910c615d8a in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1919 > #7 0x55910c6404db in route_entry_update_nhe zebra/zebra_rib.c:454 > #8 0x55910c64c904 in rib_re_nhg_free zebra/zebra_rib.c:2822 > #9 0x55910c655be2 in rib_unlink zebra/zebra_rib.c:4212 > #10 0x55910c6430f9 in zebra_rtable_node_cleanup zebra/zebra_rib.c:968 > #11 0x7f26f275b8a9 in route_node_free lib/table.c:75 > #12 0x7f26f275bae4 in route_table_free lib/table.c:111 > #13 0x7f26f275b749 in route_table_finish lib/table.c:46 > #14 0x55910c65db17 in zebra_router_free_table zebra/zebra_router.c:191 > #15 0x55910c65dfb5 in zebra_router_terminate zebra/zebra_router.c:244 > #16 0x55910c4f40db in zebra_finalize zebra/main.c:249 > #17 0x7f26f2777108 in event_call lib/event.c:2011 > #18 0x7f26f264180e in frr_run lib/libfrr.c:1212 > #19 0x55910c4f49cb in main zebra/main.c:531 > #20 0x7f26f2029d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > #21 0x7f26f2029e3f in __libc_start_main_impl ../csu/libc-start.c:392 > #22 0x55910c4b0114 in _start (/usr/lib/frr/zebra+0x1ae114) It happens with FRR using the kernel. During shutdown, the namespace identifier is attempted to be obtained by zebra, in an attempt to prepare zebra dataplane nexthop messages. Fix this by accessing the ns structure. Signed-off-by: Philippe Guibert <[email protected]> (cherry picked from commit 7ae70eb)

When a failover happens on ECMP paths that use the same nexthop which is recursively resolved, ZEBRA replaces the old NHG with a new one, and updates the pointer of all routes using that nexthop. Actually, if only the recursive nexthop changed, there is no need to replace the old NHG. Modify the zebra_nhg_proto_add() function, by updating the recursive nexthop on the original NHG. Using this change replaces the old method that was consisting in allocating a new nhe. This change triggers an ASAN in the bgp_nhg_zapi_scalability test, function test_bgp_ipv4_simulate_r5_machine_going_down(). > r1: zebra triggered an exception by AddressSanitizer > AddressSanitizer error in topotest `test_bgp_nhg_zapi_scalability.py`, test `teardown_module`, router `r1` > > ERROR: AddressSanitizer: heap-use-after-free on address 0x60e00230afa0 at pc 0x55bfebc9681e bp 0x7ffd657ceb40 sp 0x7ffd657ceb30 > READ of size 4 at 0x60e00230afa0 thread T0 > #0 0x55bfebc9681d in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1855 > FRRouting#1 0x55bfebc967f7 in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1868 > FRRouting#2 0x55bfebcb32f6 in route_entry_update_nhe zebra/zebra_rib.c:460 > FRRouting#3 0x55bfebcb352f in rib_handle_nhg_replace zebra/zebra_rib.c:486 > FRRouting#4 0x55bfebc99c14 in zebra_nhg_proto_add zebra/zebra_nhg.c:3836 > FRRouting#5 0x55bfebcc4035 in process_subq_nhg zebra/zebra_rib.c:2763 > FRRouting#6 0x55bfebcc4035 in process_subq zebra/zebra_rib.c:3369 > FRRouting#7 0x55bfebcc4035 in meta_queue_process zebra/zebra_rib.c:3422 > FRRouting#8 0x7f458a518bff in work_queue_run lib/workqueue.c:282 > FRRouting#9 0x7f458a4fa24b in event_call lib/event.c:2019 > FRRouting#10 0x7f458a41f717 in frr_run lib/libfrr.c:1238 > FRRouting#11 0x55bfebb82cb4 in main zebra/main.c:528 > FRRouting#12 0x7f4589e29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > FRRouting#13 0x7f4589e29e3f in __libc_start_main_impl ../csu/libc-start.c:392 > FRRouting#14 0x55bfebb85c34 in _start (/usr/lib/frr/zebra+0x1abc34) > > 0x60e00230afa0 is located 96 bytes inside of 160-byte region [0x60e00230af40,0x60e00230afe0) > freed by thread T0 here: > #0 0x7f458a8b4537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127 > FRRouting#1 0x55bfebc967f7 in zebra_nhg_decrement_ref zebra/zebra_nhg.c:1868 > FRRouting#2 0x55bfebcb32f6 in route_entry_update_nhe zebra/zebra_rib.c:460 > FRRouting#3 0x55bfebcb352f in rib_handle_nhg_replace zebra/zebra_rib.c:486 > FRRouting#4 0x55bfebc99c14 in zebra_nhg_proto_add zebra/zebra_nhg.c:3836 > FRRouting#5 0x55bfebcc4035 in process_subq_nhg zebra/zebra_rib.c:2763 > FRRouting#6 0x55bfebcc4035 in process_subq zebra/zebra_rib.c:3369 > FRRouting#7 0x55bfebcc4035 in meta_queue_process zebra/zebra_rib.c:3422 > FRRouting#8 0x7f458a518bff in work_queue_run lib/workqueue.c:282 > FRRouting#9 0x7f458a4fa24b in event_call lib/event.c:2019 > FRRouting#10 0x7f458a41f717 in frr_run lib/libfrr.c:1238 > FRRouting#11 0x55bfebb82cb4 in main zebra/main.c:528 > FRRouting#12 0x7f4589e29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > previously allocated by thread T0 here: > #0 0x7f458a8b4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154 > FRRouting#1 0x7f458a43fb7e in qcalloc lib/memory.c:106 > FRRouting#2 0x55bfebc91f2e in zebra_nhg_alloc zebra/zebra_nhg.c:392 > FRRouting#3 0x55bfebc91f2e in zebra_nhe_copy zebra/zebra_nhg.c:499 > FRRouting#4 0x55bfebc922af in zebra_nhg_hash_alloc zebra/zebra_nhg.c:538 > FRRouting#5 0x7f458a3fd0bd in hash_get lib/hash.c:147 > FRRouting#6 0x55bfebc94d7a in zebra_nhe_find zebra/zebra_nhg.c:831 > FRRouting#7 0x55bfebc953ef in zebra_nhg_find zebra/zebra_nhg.c:1013 > FRRouting#8 0x55bfebc9585d in zebra_nhg_find_nexthop zebra/zebra_nhg.c:1030 > FRRouting#9 0x55bfebc94078 in depends_find_recursive zebra/zebra_nhg.c:1511 > FRRouting#10 0x55bfebc94078 in depends_find zebra/zebra_nhg.c:1560 > FRRouting#11 0x55bfebc94078 in depends_find_add zebra/zebra_nhg.c:1599 > FRRouting#12 0x55bfebc99e40 in zebra_nhg_update_nhe zebra/zebra_nhg.c:3732 > FRRouting#13 0x55bfebc99e40 in zebra_nhg_proto_add zebra/zebra_nhg.c:3819 > FRRouting#14 0x55bfebcc4035 in process_subq_nhg zebra/zebra_rib.c:2763 > FRRouting#15 0x55bfebcc4035 in process_subq zebra/zebra_rib.c:3369 > FRRouting#16 0x55bfebcc4035 in meta_queue_process zebra/zebra_rib.c:3422 > FRRouting#17 0x7f458a518bff in work_queue_run lib/workqueue.c:282 > FRRouting#18 0x7f458a4fa24b in event_call lib/event.c:2019 > FRRouting#19 0x7f458a41f717 in frr_run lib/libfrr.c:1238 > FRRouting#20 0x55bfebb82cb4 in main zebra/main.c:528 > FRRouting#21 0x7f4589e29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 > > SUMMARY: AddressSanitizer: heap-use-after-free zebra/zebra_nhg.c:1855 in zebra_nhg_decrement_ref > Shadow bytes around the buggy address: > 0x0c1c804595a0: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa > 0x0c1c804595b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c804595c0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd > 0x0c1c804595d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa > 0x0c1c804595e0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > =>0x0c1c804595f0: fd fd fd fd[fd]fd fd fd fd fd fd fd fa fa fa fa > 0x0c1c80459600: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c80459610: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa > 0x0c1c80459620: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c1c80459630: fd fd fd fa fa fa fa fa fa fa fa fa 00 00 00 00 > 0x0c1c80459640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > Shadow gap: cc > Signed-off-by: Philippe Guibert <[email protected]>

louberger and others added 6 commits December 18, 2016 19:51

prefix.c: replace free with XFREE for memory allocated with XMALLOC

1631fad

bgpd: fix unitialized value check id'ed by valgrind

74f361a

lib/command.c: fix leak id'ed by valgrind

e2c4c42

bgp: first pass at restoring missing vpn&encap show commands (issue#30)

6696566

rfapi: pickup recent changes

ddf0364

expose bgp_rfapi_get_group_by_lni_label for use by rfp add EVPN Ethernet Tag (VID) RT ensure as is init'ed fix spelling of information

donaldsharp reviewed Dec 19, 2016

View reviewed changes

louberger closed this Dec 20, 2016

louberger deleted the working/2.0/patch-set-161218a branch January 7, 2017 14:20

dslicenc mentioned this pull request Feb 6, 2017

zebra and bgp crash at startup in master #170

Closed

anithanarasimhamurthy mentioned this pull request May 17, 2017

nhrpd crash on unconfig of nhrp event socket #570

Closed

louberger mentioned this pull request May 15, 2018

Bgpd crash on shutdown when have VRFs and leaked routes #2184

Closed

donaldsharp mentioned this pull request Sep 21, 2018

EIGRP has some bad reads when being shutdown #3065

Closed

mdash-vmware mentioned this pull request Oct 8, 2018

BGPd crashed on 2 routers while running Topotest "test_bgp_l3vpn_to_bgp_vrf.py" #3144

Closed

karamalla0406 mentioned this pull request Nov 14, 2018

BGP daemon crash a L2VNI in unconfigured #3333

Closed

vijaykug mentioned this pull request Nov 16, 2018

BGP daemon cored after configuring default-originate in ipv4 label unicast family #3347

Closed

cfra referenced this pull request in opensourcerouting/frr Nov 29, 2018

Merge pull request #15 from opensourcerouting/tgen-diag

535b887

topotest diagnostics

mjstapp mentioned this pull request Mar 5, 2019

Privilege escalation can deadlock #3910

Closed

SathishVenkatesh7 mentioned this pull request Apr 29, 2019

OSPFd: Crashed on Hash table Access on Multiple Scenario's #4223

Closed

louberger mentioned this pull request May 1, 2019

Zebra diet #4236

Merged

bodleytunes mentioned this pull request Jun 11, 2019

FRR 7 Centos 7 - service won't start - symbol lookup error: /usr/lib64/libfrr.so.0: undefined symbol: #4117

Closed

santoshdoke mentioned this pull request Jan 22, 2020

Interface validation failures when issuing "interface VlanXYZ" command #5718

Closed

Spantik mentioned this pull request Feb 21, 2020

Zebra: Zebra gr dynamic client handling #5850

Merged

donaldsharp mentioned this pull request Jul 1, 2020

description line in route-maps leaks memory #6659

Closed

This was referenced Oct 16, 2020

zebra: Fix use after free in debug path #7330

Merged

[7.5]zebra: Fix use after free in debug path #7331

Merged

sridharsanthanam mentioned this pull request Oct 9, 2024

Zebra crash in route_node_delete() as the same route node is accessed in two different threads. #17047

Open

2 tasks

mergify bot mentioned this pull request Oct 16, 2024

zebra: fix heap-use-after free on ns shutdown (backport #17020) #17138

Merged

This was referenced Oct 16, 2024

zebra: fix heap-use-after free on ns shutdown (backport #17020) #17139

Merged

zebra: fix heap-use-after free on ns shutdown (backport #17020) #17140

Closed

mergify bot mentioned this pull request Oct 16, 2024

zebra: fix heap-use-after free on ns shutdown (backport #17020) #17141

Closed

mergify bot mentioned this pull request Oct 16, 2024

zebra: fix heap-use-after free on ns shutdown (backport #17020) #17143

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working/2.0/patch set 161218a #15

Working/2.0/patch set 161218a #15

louberger commented Dec 19, 2016

donaldsharp left a comment

donaldsharp Dec 19, 2016

louberger Dec 20, 2016

donaldsharp Dec 19, 2016

louberger Dec 20, 2016

eqvinox commented Dec 20, 2016

louberger commented Dec 20, 2016

Working/2.0/patch set 161218a #15

Working/2.0/patch set 161218a #15

Conversation

louberger commented Dec 19, 2016

donaldsharp left a comment

Choose a reason for hiding this comment

donaldsharp Dec 19, 2016

Choose a reason for hiding this comment

louberger Dec 20, 2016

Choose a reason for hiding this comment

donaldsharp Dec 19, 2016

Choose a reason for hiding this comment

louberger Dec 20, 2016

Choose a reason for hiding this comment

eqvinox commented Dec 20, 2016

louberger commented Dec 20, 2016