do crucible datasets want primarycache property set?

[wesolows]

The contents of crucible datasets are almost entirely data owned by guests. That data is going to be cached locally by those guest OSs whether we want them to or not, and those caches will be local to whatever sled is running the instance. Maintaining a sort of second-level cache in the form of the ARC on the sleds storing that data is of little to no value and will only consume memory that could otherwise be allocated to guests or used by system/control plane software, including crucible itself. Therefore we should set primarycache to either metadata or none for all crucible-managed datasets; which is preferable will need some investigation, but all (the default) is definitely wrong.

I'm tentatively setting this to MVP+1 because it's maybe marginally possible that we'll survive without this, but not having it makes us much much more likely to experience memory pressure. Yes, the ARC is supposed to yield memory gracefully, but we've all seen things not quite work the way they're supposed to. This could potentially be considered part of MVP depending on how the storage reliability product assurance work shakes out.

[mkeeter]

Marking this as R3, based on discussion on the 2024-01-22 storage huddle.

For reads, it makes sense that data is cached in the guest OS, but for writes, there's definitely performance implications to disabling the ARC. In the huddle, we discussed the need for careful benchmarking, which isn't going to happen before release 6.

[morlandi7]

8/12 Storage huddle - decided to not do this, has been superseded by other work