Instances' sled assignments won't change if an instance is stopped and restarted

[gjcolombo]

The Nexus external API's "instance start" command passes through to Nexus::instance_start_runtime:

omicron/nexus/src/external_api/http_entrypoints.rs

Line 3088 in 9d1bd55

let instance = nexus.instance_start(&opctx, &instance_lookup).await?;

omicron/nexus/src/app/instance.rs

Lines 358 to 377 in 9d1bd55

	/// Make sure the given Instance is running.
	pub async fn instance_start(
	&self,
	opctx: &OpContext,
	instance_lookup: &lookup::Instance<'_>,
	) -> UpdateResult<db::model::Instance> {
	let (.., authz_instance, db_instance) = instance_lookup.fetch().await?;
	let requested = InstanceRuntimeStateRequested {
	run_state: InstanceStateRequested::Running,
	migration_params: None,
	};
	self.instance_set_runtime(
	opctx,
	&authz_instance,
	&db_instance,
	requested,
	)
	.await?;
	self.db_datastore.instance_refetch(opctx, &authz_instance).await
	}

If I'm reading things right, this function selects the sled to which to send the runtime state update by looking at the instance record in CRDB without regard for the instance's current state:

omicron/nexus/src/app/instance.rs

Lines 608 to 619 in 9d1bd55

	let sa = self.instance_sled(&db_instance).await?;
	let instance_put_result = sa
	.instance_put(
	&db_instance.id(),
	&sled_agent_client::types::InstanceEnsureBody {
	initial: instance_hardware,
	target: requested.clone(),
	migrate: None,
	},
	)
	.await;

This seems like the right thing to do if the instance is already incarnated on a sled somewhere. But if the instance is stopped and doesn't exist on any sled, this will try to create the instance on the sled on which it most recently ran, which might not have capacity for it (even though some other sled might). This function should distinguish the "instance already incarnated" and "instance stopped" cases and select a new sled in the latter case.

[gjcolombo]

I suspect fixing this problem will more or less require instance start to become a saga, because once it's done, starting an instance will require a lot of attendant work (reserving space on a sled, setting up V2P mappings) that we need to be able to retry if interrupted and that may need to be undone if the entire attempt to start the instance fails.