fix deadlock when closing the server and the connection at the same time

[marten-seemann]

Fixes #2716.

[codecov]

Codecov Report

Merging #2849 into master will decrease coverage by 0.03%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##           master    #2849      +/-   ##
==========================================
- Coverage   85.83%   85.81%   -0.03%     
==========================================
  Files         134      134              
  Lines        9036     9039       +3     
==========================================
  Hits         7756     7756              
- Misses        939      941       +2     
- Partials      341      342       +1     

Impacted Files	Coverage Δ
packet_handler_map.go	73.15% <0.00%> (-1.03%)	⬇️
server.go	83.29% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b765288...4c25cea. Read the comment docs.