Multiple endpoints

[michalpristas]

What is the problem this PR solves?

What this PR solves is a problem when agent got unenrolled on heavier load when agent managing fleet server cannot checkin to it's own server so it will fallback to unenroll.
Closes #741

How does this PR solve the problem?

Problem is solved by adding internal endpoint which is used for communication on local network (with agent handling fleet server)
It lets FS to spin up 2 set of handlers, one on public 8220 and one on port defined in config.

How to test this PR locally

This needs to be tested with work on elastic-agent Link: elastic/beats#28993

• Start stack
• Install agent with FS in a policy
• Check ports

sh-3.2# lsof -i -P | grep LISTEN | grep fleet
fleet-ser  7056            root   19u  IPv4 0xba7881a9227099a5      0t0    TCP localhost:{random_port} (LISTEN)
fleet-ser  7056            root   21u  IPv6 0xba7881a91284721d      0t0    TCP *:8220 (LISTEN)

• run wireshark, set filter to random port, there should be some comm
• set filter to 8220 port, there should be no comm
• enroll new agent, from another VM
• there should be some comm on both ports

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-11-23T08:02:41.102+0000

• Duration: 9 min 23 sec

• Commit: c1daf50

Test stats 🧪

Test	Results
Failed	0
Passed	213
Skipped	0
Total	213

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[michel-laterman]

Should we do direct a direct comparison of the error (err != http.ErrServerClosed), or use the errors.Is() method?

[michalpristas]

you mean replacing err != nil && err != http.ErrServerClosed with err != http.ErrServerClose?
if so you might end up with sending nil to a chan

[michel-laterman]

I'm a bit confused what the purpose of forceCh and this message are, does this occur when the server closes with a non context.Cancelled error?

[michalpristas]

yeah i'm not really sure why this is here. i understand it closes goroutine with main func but not sure why this was a problem and why this was not addressed with cancel contexts.
not touching this atm

[mergify]

This pull request is now in conflicts. Could you fix it @michalpristas? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b multiple-endpoints upstream/multiple-endpoints
git merge upstream/master
git push upstream multiple-endpoints