Use Longhorn storage provider for Vault

[matthewrossi]

Here is a summary of the changes:

• Use Longhorn for the provisioning of Persistent Volumes in Vault and MinIO tenants instead of the only option previously available (i.e., Local Path Provisioner)
• Remove the unused Local Path Provisioner

[ktatarnikovhiro]

Why is this necessary? what is rationale behind this?

[ktatarnikovhiro]

Why is this necessary? what is rationale behind this?

[matthewrossi]

Looking at the current cluster state, every service except ours are using Longhorn. So, the rationale is mainly to simplify the deployment by avoiding the use of two storage classes and their relative controller services.

Also according to a comment by a maintainer of both projects, the use of the Local Path Provisioner introduces restrictions in the scheduling of Kubernetes workloads and, overall, it lacks features like HA across the cluster, snapshot, backup, etc.

[ktatarnikovhiro]

You mentioned only theoretical arguments. Is there any technical rationale behind that?

If no, I'd rather keep minio running over local provisioner since longhorn is another level of unnecessary replication and complexity.
Minio is designed to run over local volumes and has erasure coding to handle replication failures. Running minio over longhorn is not recommended.

[ktatarnikovhiro]

For the vault and other things it is fine to use longhorn.

[matthewrossi]

Beside what previously mentioned, it may be that the scheduling restrictions of the Local Path Provisioner interfere with the AI-decision engine, but admittedly this is purely hypothetical.

In light of your considerations, I guess we can keep MinIO on the Local Path Provisioner, and move only Vault to the use of Longhorn.