Addresses for propolis instances at SLED_PREFIX + 0xFFFF

[andrewjstone]

Rather than allocating instance IPs starting at SLED_PREFIX + RSS_RESERVED_ADDRESSES + 1 where the 1 is the sled-agent allocated address of the GZ, we begin allocation from a larger block: SLED_PREFIX + CP_SERVICES_RESERVED_ADDRESSES. This gives us more room for nexus to allocate control plane services.

Implements #4765

[smklein]

At a high-level, I see how this bumps up the last_used_address value in the sled table, but I'm not sure how this resolves #4765.

To me, that issue implies that we basically have a "separate pool" for services and instance addresses.

First off, I'd recommend the following:

• schema/crdb/dbinit.sql - there's a comment on last_used_address suggesting this is for an "Oxide service" - now it isn't, it's for instances exclusively.
• nexus/db-model/src/sled.rs - there's a similar comment on last_used_address that needs updating
• nexus/db-queries/src/db/datastore/mod.rs - there's another comment on next_ipv6_address that indicates "pulling from last_used_address" is for Oxide services. This isn't true anymore, again, it's service-only.
• That next_ipv6_address method is called by a function called allocate_sled_ipv6, which again, is kinda a misnomer, since this should now only be used for instances.

But before we merge this:

• Suppose someone does want to provision a new IP for an "Oxide service" (and not an instance!) on a sled. How should they do this? Previously, they could have called next_ipv6_address / allocate_sled_ipv6 on the datastore. By using this field as a counter, there was no additional overhead. Now, this PR seems like it does not provide a mechanism for actually doing this allocation. Since we're inserting a schema change here, are we sure this is sufficient? Don't we need to now check for conflicts in the "reserved-for-control-plane-services" address space?

[davepacheco]

Suppose someone does want to provision a new IP for an "Oxide service" (and not an instance!) on a sled. How should they do this? Previously, they could have called next_ipv6_address / allocate_sled_ipv6 on the datastore. By using this field as a counter, there was no additional overhead. Now, this PR seems like it does not provide a mechanism for actually doing this allocation. Since we're inserting a schema change here, are we sure this is sufficient? Don't we need to now check for conflicts in the "reserved-for-control-plane-services" address space?

The plan I've been working from (which is really just an idea we've talked about, plus an implementation in my branch, so still somewhat speculative) is that the store of record for IP addresses in use by Omicron is the current target blueprint. Thus in generating a new blueprint the planner can pick any addresses it wants as long as they don't overlap with each other.

[smklein]

The plan I've been working from (which is really just an idea we've talked about, plus an implementation in my branch, so still somewhat speculative) is that the store of record for IP addresses in use by Omicron is the current target blueprint. Thus in generating a new blueprint the planner can pick any addresses it wants as long as they don't overlap with each other.

Gotcha, so this conflict resolution could all happen in-memory then, and we wouldn't need to do anything in the DB to help us do the allocation? That makes sense to me.

[smklein]

• schema/crdb/dbinit.sql - there's a comment on last_used_address suggesting this is for an "Oxide service" - now it isn't, it's for instances exclusively.
• nexus/db-model/src/sled.rs - there's a similar comment on last_used_address that needs updating
• nexus/db-queries/src/db/datastore/mod.rs - there's another comment on next_ipv6_address that indicates "pulling from last_used_address" is for Oxide services. This isn't true anymore, again, it's service-only.
• That next_ipv6_address method is called by a function called allocate_sled_ipv6, which again, is kinda a misnomer, since this should now only be used for instances.

@andrewjstone , if we can resolve these nits, this PR LGTM!

[andrewjstone]

• schema/crdb/dbinit.sql - there's a comment on last_used_address suggesting this is for an "Oxide service" - now it isn't, it's for instances exclusively.
• nexus/db-model/src/sled.rs - there's a similar comment on last_used_address that needs updating
• nexus/db-queries/src/db/datastore/mod.rs - there's another comment on next_ipv6_address that indicates "pulling from last_used_address" is for Oxide services. This isn't true anymore, again, it's service-only.
• That next_ipv6_address method is called by a function called allocate_sled_ipv6, which again, is kinda a misnomer, since this should now only be used for instances.

@andrewjstone , if we can resolve these nits, this PR LGTM!

These are great catches! I will fix them. Thank you very much for the thorough review!