Document the process to perform a stable plugin upgrade

[mbrancato]

Is your feature request related to a problem? Please describe.

Yes. We have experienced stability issues with upgrading the mount path using a plugin. Simply redoing the catalog registration has caused some service disruption.

Describe the solution you'd like

I'd like for the process of upgrading a plugin mount to be described on the plugin page. Please answer questions like: What triggers Vault to know to kill the previous process and re-run the binary? Does replacing the binary while running cause a problem? What are the best practices here?

https://github.com/hashicorp/vault/blob/master/website/pages/docs/internals/plugins.mdx#plugin-catalog

Describe alternatives you've considered

Being prepared to restart Vault if it stops serving requests after a plugin change.

Explain any additional use-cases

The primary use-case here is plugins with partner vendors of HashiCorp which are not currently integrated into the Vault binary.

Additional context

[aphorise]

There's been the following page added since this request was raised:

• Upgrading Vault Plugins.

In my mind anything additional to that page in terms of SOP / process doc would simply say follow SRE and immutability practises by involving snapshots as well as provisioning things separate and independently testing them (smoke & end-to-end) with the same approach as any upgrade (aka Blue / Green or Canary deployments).

Additional considerations that make it challenge to enhance or provide any more are:

• No guide could guarantee or assure that any recommended approach will work or be valid until it's actually tested. This is particularly important as the same culprits of: Platform / OS, Vault Version and plugin version (and it's dependencies) are the the usual suspect blockers when plugin upgrades do not work. Having a guide that still says watch out for these spaces that you need to test anyway may be great education reference - but the guide itself can not provide for any assurance or outcomes so it could leave gaps in expectations in comparison to most other learn guides.
• The setup of each plugin differs; so how each should be upgraded is also different and especially with mounts that have an existing configurations in rafts storage already and which may need particular actions as opposed to just dropping upgraded binaries in the same places where they are working in a dirty mutable manner - expecting for things to just work.
• If you have enterprise - you can use DR / PR - promote - test independently (saves on 1 step replication / snapshots).
• What about folks that are incapable of immutability or somehow have other limitations where no additional redundancy have been made or where capacity planning is lacking? - this is the main issue that I feel no article could ever fully address and there might just be recommendations or advisory note against this in-place of attempting upgrades.
• Test test test - only way to confirm and more closely provide assurances.

@mbrancato hey - I'm curious if you agree that this request may be closed and if not I'd love to hear which plugins you are dealing with or with which of the plugins you've had upgrade pain points on.

[mbrancato]

As far as I'm concerned, this can be closed. The issue I had at the time (2.5 years ago) was a commercial plugin from a HashiCorp partner that needed to be upgraded and an enterprise licensed Vault cluster / clusters. At the time, there was no guide on how to reload the plugin catalog - mainly because support for this did not exist at the time (outside if a full restart). This has since been added in #8777 and #9347.