server: make node startup more principled

[couchand]

Peeled off of discussion on #25771.

Currently pkg/server is a bit messy. In addition to organization questions such as raised in #24989, there are concerns about pkg/server/server particularly. Two 500 line functions (NewServer and server.Start) are together responsible for coordinating the configuration and startup of basically every subsystem of a running node. For the most part they do it by delegating to other packages, but there are still many lines of implementation details in line. In addition, some subsystems rely on shared mutable state, and so uncertainty in initialization can cause hard-to-debug race conditions such as #25771.

Adding new node startup code is fraught: if we don't expect dependencies we'll be lax in where to put it, and even if we are thoughtful about the question it is hard to know where it should go. Ideally we would make these startup dependencies clear.

We should work to simplify the node startup code. Irrelevant details should be encapsulated, exposing a declarative initialization for each subsystem or component, and moving pure networking code into it's own domain. We'd like to make any (known) initialization dependencies explicit in code, but at least document them.

@tschottdorf @BramGruneir @vilterp

[kenliu]

this doesn't belong in the Web UI project, reassigning to Core

[irfansharif]

This area has seen a lot of flux lately (most recently #32574), and while we can always continue to improve our organization here, I'm going to close this out.