Nexus needs reconciliation process for Sled failure / removal

[smklein]

Nexus currently stores information about Sled IDs within CRDB, including:

• Instances belonging to a particular Sled
• Zpools, datasets, and (pending) regions belonging to a particular Sled
• (soon) Versioning and Inventory information for a sled.

If a sled irrecoverably fails - not just reboots, but is either destroyed or unplugged - Nexus needs to have a process for recovering all data which has been detached from the Sled.

This will likely involve:

• Identifying all instances previously running on the Sled have faulted, if they do not have backups
• Migration of regions to new locations, if available
• Cleaning sled-specific information from the database

Additionally, it is important to note that "unplugging a sled" is unpredictable, and may occur at any time. There is separate, critical question of distinguishing between a temporary vs permanent failure (we would want to treat a crash + reboot much differently from a long-term sled removal). However, in the period of time while we're still making that decision, we need to have a policy for dealing with ongoing operations to that sled.

Arguably, the "deletion" operations are particularly nasty in this time period - we don't know if we can discard those requests ("the sled is gone, so we just need to clean the DB"), store them for later ("the sled if offline now, but if it comes back, we'll tell it to perform the deletion") or fail those requests ("deletion is not possible while this sled is offline!")

[andrewjstone]

Great writeup @smklein. I would just like to add that we have discussed sled removal in the past and I believe tentatively decided that "detecting" permanent removal should be done by a rack admin in a controlled operation. @rmustacc

[leftwo]

Yeah, @smklein great write up. I was thinking about this as well, specifically the case where a crucible region on a sled is deleted when that sled is not present and how do we deal with it when it comes back.

[davepacheco]

This process is fleshed out in RFD 457.

[davepacheco]

Closing this issue not because it's wholly done but because more specific pieces are tracked by other issues and the Reconfigurator project/board so I don't think this represents any particular open work or milestones any more. Feel free to reopen if that's wrong!