cherry-pick #7664 to v1.67.x branch

[arjan-bal]

Original PR: #7664
Addresses: #7657

RELEASE NOTES:

• xds/server: Fix xDS Server memory leak

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 81.77%. Comparing base (935f8cb) to head (d826085).
Report is 1 commits behind head on v1.67.x.

Files with missing lines	Patch %	Lines
xds/internal/server/listener_wrapper.go	93.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           v1.67.x    #7681      +/-   ##
===========================================
- Coverage    81.88%   81.77%   -0.12%     
===========================================
  Files          361      361              
  Lines        27813    27817       +4     
===========================================
- Hits         22776    22748      -28     
- Misses        3844     3864      +20     
- Partials      1193     1205      +12     

Files with missing lines	Coverage Δ
xds/internal/server/conn_wrapper.go	77.04% <100.00%> (-4.62%)	⬇️
xds/internal/server/listener_wrapper.go	79.01% <93.33%> (+0.39%)	⬆️

... and 23 files with indirect coverage changes