[sled agent] Add default routes, using GZ addresses as gateways

[smklein]

Follow-up to #1066

This ensures that L3 routing from off-device can reach the external Nexus interface.

[bnaecker]

Looks good!

Just to make sure I understand, the issue here was that by hanging the IP addresses of an etherstub, they're acting like a separate L2 segment, even though they're in the same L3 subnet of the sled. Thus we need forwarding, to get the OS to deliver packets from the GZ's L2 segment to the etherstub's, and routing for the OS to automatically update routing tables to describe the interfaces used to do that forwarding. Is that a fair summary?

[rmustacc]

So a non-global zone shouldn't need anything special here whether with maghemite or special services enabled here. What led us to this conclusion and were we setting a default gateway?

[smklein]

Just to make sure I understand, the issue here was that by hanging the IP addresses of an etherstub, they're acting like a separate L2 segment, even though they're in the same L3 subnet of the sled. Thus we need forwarding, to get the OS to deliver packets from the GZ's L2 segment to the etherstub's, and routing for the OS to automatically update routing tables to describe the interfaces used to do that forwarding. Is that a fair summary?

I believe so? The concrete issue I faced was:

• Try to ping sled agent address (in Helios GZ) from a different machine (my Linux laptop) -> Observe success
• Try to ping nexus address (in Helios non-GZ) from a different machine (my Linux laptop) -> Observe failure
• experimentally, updating the routeadm arguments fixed this issue.

So a non-global zone shouldn't need anything special here whether with maghemite or special services enabled here.

Just to make sure we're on the same page - Maghemite is not currently integrated with Omicron, so this code is not assuming it's present.

What led us to this conclusion and were we setting a default gateway?

We set routes in non-global zones here:

omicron/sled-agent/src/services.rs

Lines 232 to 249 in 1593b8b

	let gz_route_subnet = if !service.gz_addresses.is_empty() {
	// If this service supplies its own GZ address, add a route.
	//
	// This is currently being used for the DNS service.
	//
	// TODO: consider limitng the number of GZ addresses which
	// can be supplied - now that we're actively using it, we
	// aren't really handling the "many GZ addresses" case, and it
	// doesn't seem necessary now.
	Ipv6Network::new(service.gz_addresses[0], AZ_PREFIX).unwrap()
	} else {
	// Otherwise, add a route to the global Zone's sled address for
	// everything within the AZ.
	Ipv6Network::new(self.underlay_address, AZ_PREFIX).unwrap()
	};
	running_zone.add_route(gz_route_subnet).await.map_err(|err| {
	Error::ZoneCommand { intent: "Adding Route".to_string(), err }
	})?;

Which is implemented by this:

omicron/sled-agent/src/illumos/running_zone.rs

Lines 174 to 187 in 1593b8b

	pub async fn add_route(
	&self,
	destination: ipnetwork::Ipv6Network,
	) -> Result<(), RunCommandError> {
	self.run_cmd(&[
	"/usr/sbin/route",
	"add",
	"-inet6",
	&format!("{}/{}", destination.network(), destination.prefix()),
	"-inet6",
	&destination.ip().to_string(),
	])?;
	Ok(())
	}

To be clear, I'm totally open to alternate solutions in this space.

[rmustacc]

I believe so? The concrete issue I faced was:

* Try to ping sled agent address (in Helios GZ) from a different machine (my Linux laptop) -> Observe success

* Try to ping nexus address (in Helios non-GZ) from a different machine (my Linux laptop) -> Observe failure

* experimentally, updating the `routeadm` arguments fixed this issue.

I'd like to work with you to root cause this a bit more. In particular, the thing that's weird here is that in a normal context, we're not actually supposed to be forwarding in the non-global zone. Forwarding would suggest that we have multiple interfaces and want to take traffic that came in on one that wasn't destined for us and that it should be sent out another (or some other internal rewriting is going on with ipf which shouldn't be on the scene here). So in this context to me, the requirement to have forwarding doesn't make sense and seems like something else may be going on here.

So a non-global zone shouldn't need anything special here whether with maghemite or special services enabled here.

Just to make sure we're on the same page - Maghemite is not currently integrated with Omicron, so this code is not assuming it's present.

My point was that you said in the start of this:

Presumably we'll have a more sophisticated solution here once maghemite is integrated, but this works in the meantime.

And that the integration of maghemite should not impact the non-global zone as it will not run there or the set of configuration that it requires.

What led us to this conclusion and were we setting a default gateway?

We set routes in non-global zones here:

omicron/sled-agent/src/services.rs

Lines 232 to 249 in 1593b8b

	let gz_route_subnet = if !service.gz_addresses.is_empty() {
	// If this service supplies its own GZ address, add a route.
	//
	// This is currently being used for the DNS service.
	//
	// TODO: consider limitng the number of GZ addresses which
	// can be supplied - now that we're actively using it, we
	// aren't really handling the "many GZ addresses" case, and it
	// doesn't seem necessary now.
	Ipv6Network::new(service.gz_addresses[0], AZ_PREFIX).unwrap()
	} else {
	// Otherwise, add a route to the global Zone's sled address for
	// everything within the AZ.
	Ipv6Network::new(self.underlay_address, AZ_PREFIX).unwrap()
	};
	running_zone.add_route(gz_route_subnet).await.map_err(|err| {
	Error::ZoneCommand { intent: "Adding Route".to_string(), err }
	})?;

Which is implemented like this:

OK. I would have expected just a simple default route for everything to go back to the gateway. So I think we can simplify what's going on here.

[smklein]

Robert, reading your feedback, I agree with you - routing / forwarding really shouldn't be necessary for the non-global zones.

I went ahead and re-did this PR - it accomplishes the same goal, but through a different mechanism. Rather than enabling this routing in non-global zones, I changed the way routes are getting set up.

Rather than routing to addresses within the AZ of the system, I implemented your suggestion, and added a default route which points to a corresponding gateway address from the global zone.

This works - I can now access Nexus from off-machine. This is kinda interesting - I basically only changed the routing tables of the non-global zones, which... shouldn't have been necessary to actually reach those zones, but perhaps may have been necessary for ping to respond to addresses coming from outside the AZ subnet.

[rcgoodfellow]

Try to ping sled agent address (in Helios GZ) from a different machine (my Linux laptop) -> Observe success
Try to ping nexus address (in Helios non-GZ) from a different machine (my Linux laptop) -> Observe failure

Do I understand correctly that in the above, your laptop was pinging from a source address outside the AZ /48, the non-global zones were set up with a route onto the AZ /48, and changing that route to be a more general default route allowed communications to succeed from your laptop?

[smklein]

Try to ping sled agent address (in Helios GZ) from a different machine (my Linux laptop) -> Observe success
Try to ping nexus address (in Helios non-GZ) from a different machine (my Linux laptop) -> Observe failure

Do I understand correctly that in the above, your laptop was pinging from a source address outside the AZ /48, the non-global zones were set up with a route onto the AZ /48, and changing that route to be a more general default route allowed communications to succeed from your laptop?

This is accurate, as far as I understand it.

[rcgoodfellow]

LGTM