[nexus] Add HTTPS support, plumbing x509 certificates

[smklein]

Another attempt at #1287

In addition to launching an HTTPS server, this also launches an HTTP server so we can smoothly migrate clients (like the CLI).

Part of #249

[smklein]

This might be a longer-term issue to resolve, but I'd like for this to not necessarily be bundled with the package.

However, this may require a different mechanism for acquiring / rotation of the certificate.

[smklein]

I followed up today and met with Rick to discuss this.

My takeaways:

• RSS must provide a way for users to input the Nexus IP address + initial certificates
  • Later, an interface through Nexus will support modifying these values
• A key storage mechanism like https://github.com/hashicorp/vault - not necessarily this particular technology, but something highly available - should be used to store keys for both internal and externally facing services.
  • For internal services, we could concoct a scheme where:
    • When booted, services generate a public/private key pair
    • The private key stays private to the service while it exists in-memory
    • The public key gets published to "vault" - or whatever key management facility we use - and signed by the internal root CA. This should rely on trusting "who is launching the service" (namely, sled agent) to extend this trust.
  • For external services, we rely on user input for these certificates. We could store them in this "vault", but cannot generate them ourselves.

[smklein]

This is tracked by #1528

[smklein]

TODO(me): I wonder if I can fix #1398 and make this easier at the same time?

[smklein]

See: #1528. We're gonna need to grab / insert certs from somewhere.

I've updating the packaging hints to make this more obvious if an error is encountered.

[davepacheco]

Thanks for doing this...it'll be great to have TLS here!

I think all of my comments boil down to the way this gets configured. I didn't follow why we don't just pass through the corresponding dropshot config. I may well have missed something here about why that wouldn't work. But if we could do that, it'd be less code and at the same time preserve more configurability -- some of which seems missing here for the development machine use case.

[davepacheco]

Is there an analogous change for how-to-run-simulated? In general, the test suite matches what how-to-run-simulated does so if you've updated that then things should work.

[smklein]

There is not, because the implementation within Nexus checks for the existence of the certificate files before deciding whether or not to launch the HTTPS server.

For the simulated server, they won't exist, so only the HTTP server will be launched.

I definitely think this would be required before moving to "HTTPS only", but considered this PR an intermediate step.

[davepacheco]

Mentioned in the other comment, but that doesn't seem like the right behavior. It seems like in production, if they don't exist, that should be a fatal error. And in development, if they do exist, we should start the HTTP server. If this were part of configuration, then whoever's running Nexus can express their intent. (I'm not saying we have to do all that in this PR, but if we decide that is what we want, then I think this behavior is not really an intermediate step.)

[davepacheco]

This seems unlikely to work on development machines.

Why not make this part of dropshot_external, with /var/nexus/certs/cert.pem the value in the SMF config file and something like "out/certs" the value in nexus/examples/config.toml?

[smklein]

FWIW, this file is only used if cert_file.exists() && key_file.exists().

I was kinda opposed to using the config-file-supplied certificate, because longer-term, it seemed like this value would be:

• Passed from RSS into Nexus: RSS -> Nexus handoff plumbing #1528 , as part of the rack initialization request
• Stored in CRDB (and eventually a more secure key store, like Vault): https://github.com/oxidecomputer/omicron/pull/1500/files#r932591199

So although we are currently plumbing the certificates through the package system, I don't expect that to stay the case for very long.

[davepacheco]

FWIW, this file is only used if cert_file.exists() && key_file.exists().

What happens if the path is missing on a production system? It seems like that should produce a fatal error.

I was kinda opposed to using the config-file-supplied certificate

I can see where you're coming from. At the same time, it seems like an essential developer or demo tool to be able to start Nexus with a specific certificate?

Is that a good idea for the production config? I'm not sure. It might depend on the expected behavior in the longer term. Is it that Nexus reaches out to CockroachDB/Vault to get the certificate, then loads that into memory, and then starts the HTTPS server with it? If that's true, then what do we do if Vault is down? I could see: (1) not coming up at all until we can get the cert, (2) storing the last-known-good certificate to a file and trying to use that until we can get the cert, (3) generating a self-signed certificate and using that until we're able to access the real one. These all have different tradeoffs. I've been assuming Nexus could come up (albeit not very usefully) even if everything else in the world was down so that we could at least ask it why it wasn't able to come up. Anyway, if we go with (2) or (3), then it seems like accepting a file path in the config file would be a reasonable stepping stone.

(Along these lines, I guess hardcoded, deployment-specific absolute paths in source files seem like a code smell to me, even if we gracefully handle them being missing.)

[smklein]

I've tried to make this more amenable to the idea of "allow the ConfigDropshot structure to be plugged directly into Nexus", which attempted to address your comment here: #1500 (comment)

Since Sled Agent is the entity actually responsible for launching Nexus, it is now plumbing this information, by passing a Vec<ConfigDropshot> to represent any number of external interfaces. I'm hoping this addresses your comment here, in that it leaves the input to Nexus "somewhat general-purpose", so developer environments + test harnesses can do whatever they want to do.

I do still think there's a legitimate question of "how do we pass these certificates when they are not part of the Nexus package" - it seems like sled agent will need to load them into the filesystem somehow to force them into the Nexus zone, and ultimately the ConfigDropshot path, if we are still trying to stick to that configuration.

[davepacheco]

Awesome! I like this a lot better. Thanks. (I hope you at least don't hate it!)