[DRAFT] Allocate static Ipv6 addresses for propolis zones

[jmpesp]

Add ability for Nexus to hand static Ipv6 addresses for use by propolis
zones.

TODO still draft

[jmpesp]

There's a few XXX to deal with before merging, would like some suggestions on them please :)

[bnaecker]

I have two questions about this subnet:

1. What differentiates this from the subnet on the underlay assigned to the sled during rack setup?
2. It seems like this should be a /64, as with other sled-specific subnetworks, but I'm not 100% clear on the context.

[smklein]

heh, lots of ben's comments mirror my own. Left 'em in for posterity though.

[smklein]

If I'm reading https://rfd.shared.oxide.computer/rfd/0063#_ipv6_addressing correctly:

An availability-zone is represented by an IPv6 /48.
A rack and its resources is represented by an IPv6 /56.
A server and its resources are represented by an IPv6 /64.

It's unclear to me if this should currently change the allocation scheme - we aren't using the ULAs yet, really (though maybe we should) but it makes sense to consider a subsection of the /56 "rack-wide" physical address space for this purpose.

[smklein]

I added @bnaecker to this review because this definitely looks like it overlaps with

omicron/nexus/src/db/subnet_allocation.rs

Line 721 in b259f45

fn push_select_next_available_ip_subquery(

somewhat.

[smklein]

I think we may simplify the allocation space by allocating within the context of a sled.

This API implies that we're allocating from an IP space that's basically rack-wide - but if we're allocating within an individual sled's /64 space (which I think we planned to do anyway, according to RFD 63), we'll have less risk of collision, which should make the allocation problem a little easier?

Basically: WDYT about providing an identifier for the Sled as an input parameter here, and to the allocator? Then Nexus can basically make a "per-sled" subnet from which to allocate

[bnaecker]

That's a nice idea, and I think fits better with the rack setup service's behavior too. The operator will choose an AZ and rack number, and then each sled will have a further ID within those. That'll all combine to make an fd00::/64 for each sled, so a per-sled allocation here seems good.

It also suggests that the database table also might need the sled UUID in it, I believe.

[smklein]

Yeah, that tracks. Being able to predictably access all IP addresses allocated to a sled would also be important when we, for example, decommission a sled (or cope with it failing).

[smklein]

@bnaecker 's subnet allocation has an example of this:

omicron/nexus/src/db/subnet_allocation.rs

Lines 568 to 573 in b259f45

	// Primary key constraint violation. See notes above.
	Some(constraint) if constraint == PRIMARY_KEY_CONSTRAINT => {
	NetworkInterfaceError::DuplicatePrimaryKey(
	interface.identity.id,
	)
	}

[bnaecker]

@jmpesp If you find yourself checking specific database error conditions or messages, let me know. I've been prototyping some tooling to make that kind of thing easier. That hasn't been too valuable yet, with only 2 use sites, but if we add more...

[bnaecker]

I'm not sure how important this is, but it'll definitely be feasible to allocate addresses in a single query without a transcation. The push_next_available_ip_subquery function is probably a good place to start. One could a left outer join with the table itself, joining on the address being non-NULL, and taking only the first.

For example, this query should work OK:

 select 'fd00::'::INET + off as addr from generate_series(0, 10) as off left outer join static_v6_address on (address = 'fd00::'::INET + off) WHERE address is NULL LIMIT 1;

That's basically the same thing we have in the linked function, so it might be nice to factor it if possible. That takes the first address where there is no corresponding row already in the static_v6_address table. It's effectively the query you have here in the Rust transaction, just in one query. It's still linear in the number of allocated addresses, but requires no transaction or multiple attempts. Hope that helps!

[smklein]

Building on top of this: I wonder how long we could survive with a (simple) linear allocator by just starting at the "last allocated address", rather than "always start from the beginning". This would basically let us comb around the full address space, rather than repeatedly checking the first few addresses.

(Could definitely be a punted-until-later optimization, but seems like it would help both our address allocators)

[bnaecker]

Yeah, that's true. Part of the reason we're using IPv6 is that each sled has an entire /64 to work with. That's...a lot of addresses, so just picking the "next" rather than the lowest unallocated seems OK in the short term. Neither is great in the long term, but the "next after the last allocated address" approach is definitely simpler. The "first fit" one in the linked function makes more sense there, because we're talking about potentially tiny subnets in a customer's VPC Subnet, which can be as small as an IPv4 /26 I believe. That's only 16 addresses, 6 of which we reserve for ourselves!

[bnaecker]

This is a great start!

Besides my comments in the source, I'm not sure I understand the whole picture here. Are these just for propolis zones? Or for any Oxide service that the sled-agent starts and manages? The other main question is about the actual subnet these come from. IIUC, these should be on the underlay network, which means they must be allocated out of the /64 that the sled gets during rack setup. Not that we know what that subnet is, mind you, just that we know the size is /64/, not /96.

[smklein]

Besides my comments in the source, I'm not sure I understand the whole picture here. Are these just for propolis zones? Or for any Oxide service that the sled-agent starts and manages?

@jmpesp , please correct me if I'm wrong, but I believe this is attempting to address this chunk of code in instance creation:

omicron/sled-agent/src/instance.rs

Lines 475 to 477 in 015158d

	let running_zone = RunningZone::boot(installed_zone).await?;
	let network = running_zone.ensure_address(AddressRequest::Dhcp).await?;
	info!(inner.log, "Created address {} for zone: {}", network, zname);

Currently, when we allocate Propolis instances, we are giving them IPv4 addresses assigned via DHCP.

I think we want the underlay to actually be on IPv6, and chatting with James, I think there have been some difficulties faced by Propolis just trying to access other IPv6 addresses if it doesn't have one of it's own. I thiiiink this particularly popped up when trying to access Crucible services via IPv6 addresses - our workaround had been "allocate an arbitrary IPv6 address to Propolis, so it can speak to other IPv6 addresses", but that doesn't really scale.

[smklein]

I'm brainstorming a bit, but:

The proposal for the "bootstrap network" in RFD 63 suggests using a sequence of unique bits (i.e., a Sled's MAC address) prepended by a well-known prefix to form a unique IPv6 address.

What do you think about doing something similar here?

If we start with:

• A server and its resources are represented by an IPv6 /64...
• and we know that MAC addresses are 48 bits long...
• and we know that each Propolis Zone gets a dedicated VNIC for control-plane communication...

Couldn't we give static addresses to Propolis like the following:

128 bit IPv6 address = 
[ 64 bits of Sled-Specific Underlay Prefix ] [ 48 bits of Propolis Zone VNIC MAC address ] [ 12 bits of whatever we want ] 

This has some advantages:

• The address remains unique within the physical network, because of the prefix
• The address remains unique on the Sled, because of the VNIC's MAC address
• We don't need to allocate anything / track any of this state in Nexus

[rmustacc]

So IPv6 already has a well formed way of transforming a MAC address into an EUI64. So what I'd recommend is that in general we don't try to allocate specific ordering of addresses, that we always go and conjoin things based on the MAC address. This means that the lower 64-bits will always be unique and will always be appended to. Now, that still comes down to how that MAC address for an underlying propolis instance (as opposed to the guest-visible things are being done), but that just makes it an AZ-wide random value that we want to be unique.

[jmpesp]

Took another swing at this today:

• addresses are allocated as part of instance create and instance migrate sagas
• they're sent to the sled agent as part of the instance ensure request

lots of XXX aka things for me to resolve still.

[jmpesp]

Thoughts for tomorrow: I'm pretty sure control_ip is going to have to be a column in the instance, but I was hoping that the static_v6_address table would be a single location for allocation to avoid having to check many places.

[jmpesp]

Thoughts for tomorrow: I'm pretty sure control_ip is going to have to be a column in the instance, but I was hoping that the static_v6_address table would be a single location for allocation to avoid having to check many places.

This should have been titled "thoughts for hack night" :)

8aa5773 adds an associated UUID to the row, meaning callers that allocate a static IP have to now supply their UUID. It's difficult to go from this column to what actually allocated the address (read: one has to search every table for the ID), but this means that multiple types of resources can allocate IPs for themselves and conflicts are localized to one table.

[jmpesp]

Obviated by #891, closing