Nexus must create "Control Plane Disks" for disks automatically

[smklein]

This is the root case of #5502 .

(The decision to create a new issue is a little arbitrary, I just wanted to deep-dive onto the root cause here, while letting the symptoms stay visible on #5502).

As of #5172 , RSS / Nexus are responsible for managing "control plane disks" and "control plane zpools". In this world, the sled agent no longer autonomously formats disks, but waits for the "go-ahead" from Nexus.

For bootstrapping, RSS also has this responsibility:

• It creates a cofiguration based on inventory:

  omicron/sled-agent/src/rack_setup/plan/service.rs

  Lines 345 to 370 in 20299e0

  	for sled_info in sled_info.iter_mut() {
  	let disks = sled_info
  	.inventory
  	.disks
  	.iter()
  	.filter(|disk| {
  	matches!(disk.variant, SledAgentTypes::DiskVariant::U2)
  	})
  	.map(|disk| OmicronPhysicalDiskConfig {
  	identity: disk.identity.clone(),
  	id: Uuid::new_v4(),
  	pool_id: Uuid::new_v4(),
  	})
  	.collect();
  	sled_info.request.disks = OmicronPhysicalDisksConfig {
  	generation: Generation::new(),
  	disks,
  	};
  	sled_info.u2_zpools = sled_info
  	.request
  	.disks
  	.disks
  	.iter()
  	.map(|disk| ZpoolName::new_external(disk.pool_id))
  	.collect();
  	}

• Sends the configuration to sleds:

  omicron/sled-agent/src/rack_setup/service.rs

  Lines 348 to 350 in 20299e0

  	let result = client
  	.omicron_physical_disks_put(&storage_config.clone())
  	.await;

• And tells Nexus about it in the RSS handoff, where the info is stored in CRDB:

  omicron/nexus/db-queries/src/db/datastore/rack.rs

  Lines 739 to 760 in 20299e0

  	for physical_disk in physical_disks {
  	Self::physical_disk_upsert_on_connection(&conn, &opctx, physical_disk)
  	.await
  	.map_err(|e| {
  	error!(log, "Failed to upsert physical disk"; "err" => #%e);
  	err.set(RackInitError::PhysicalDiskInsert(e))
  	.unwrap();
  	DieselError::RollbackTransaction
  	})?;
  	}
  	info!(log, "Inserted physical disks");
  	for zpool in zpools {
  	Self::zpool_upsert_on_connection(&conn, &opctx, zpool).await.map_err(
  	|e| {
  	error!(log, "Failed to upsert zpool"; "err" => #%e);
  	err.set(RackInitError::ZpoolInsert(e)).unwrap();
  	DieselError::RollbackTransaction
  	},
  	)?;
  	}

So, this works in the initialization case.

However, in the case of "sled addition", we're missing the follow-up parts:

• Nexus needs to assemble the OmicronPhysicalDisksConfig that gets sent to sleds
  • For "new disks": This means scanning the inventory for physical disks, and creating a new control plane object if one does not exist. Note that "expunged disks" still exist, and should not be automatically re-added.
  • For "existing disks": This means using control plane disks that are not expunged.
• As a part of blueprint execution, Nexus needs to actually send OmicronPhysicalDisksConfig to individual sleds

Regarding new disk addition, there's a bit of uncertainty about "when we should perform this action".

• We could do it once during "sled add" - take a specific point-in-time view of inventory, adopt everything then, and subsequently refuse to auto-online disks
• ... or we could have "auto-online" be the default policy, until the disk is expunged, at which time it must be explicitly re-added

My opinion incoming: I think we've gone back-and-forth on this a bit because it's a policy decision. An operator could explicitly opt into an "add physical disk" API for each addition. Or they could just want all disks to auto-attach and start working. We could save this as a toggle, and enable both pathways? But that might be unnecessary effort, if we end up recommending / observing that customers prefer a particular option.

[andrewjstone]

Regarding new disk addition, there's a bit of uncertainty about "when we should perform this action".

We could do it once during "sled add" - take a specific point-in-time view of inventory, adopt everything then, and subsequently refuse to auto-online disks
... or we could have "auto-online" be the default policy, until the disk is expunged, at which time it must be explicitly re-added

I think my preference is to always 'auto-online' right now. We don't know timing wise which disks we'll get first and we still need to add APIs for explicit adoption. Once we add those APIs we can choose not to auto-online. If we don't auto-online, we may also want to show the customers what disks they are getting when they add a sled by showing them in list-uninitialized.