Use an endpoint instead of a port for the Minio service

[Adam-D-Lewis]

Discussed in https://github.com/orgs/nebari-dev/discussions/2845

^{Originally posted by mcg1969 November 11, 2024}
Hey folks! I wanted to ask if there was a specific reason why the Minio service was chosen to be deployed on a dedicated port (9080) instead of somewhere on the path of the existing HTTPS port (443). In my experience, as long as the bucket name does not conflict with a path used by other applications, the ingress (e.g., Traefik) can handle routing S3 traffic just fine.

Reducing the use of custom ports will make it easier to deploy Nebari in situations where it is important to re-use an existing cluster's ingress.

[Adam-D-Lewis]

We discussed this on an internal call today. The consensus was it we think this should be possible to change. There may be an issue with conda store supporting minio on a subpath, but we agree it can/should be fixed if not. Thanks for calling this out and I'll transfer this to an issue.

[viniciusdc]

For additional context, this issue was initially discussed in #980 (back when it was related to QHub), which contains a bit more info on why that port specifically was chosen. Conda-store does allow customization of the bucket name and addresses, as shown in the configuration file:

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/config/conda_store_config.py

Lines 40 to 47 in 59a65e2

	c.S3Storage.internal_endpoint = f"{config['minio-service']}:9000"
	c.S3Storage.internal_secure = False
	c.S3Storage.external_endpoint = f"{config['external-url']}:9080"
	c.S3Storage.external_secure = True
	c.S3Storage.access_key = config["minio-username"]
	c.S3Storage.secret_key = config["minio-password"]
	c.S3Storage.region = "us-east-1" # minio region default
	c.S3Storage.bucket_name = "conda-store"

Since S3 (and MinIO) do not support subpaths (in the routing sense), changing the storage endpoint to port 443 would create a conflict between the conda-store's internal endpoints and storage access. We could rename the bucket to avoid the conflict; however, we need to plan how to migrate existing data safely.

I've tried some remediation in the past, but due to the internal signing of the URLs I ended encountering some problems with mismatches in the expected request bodies
https://github.com/conda-incubator/conda-store/blob/aac4f4b03033207566564efdaaa9fcc93c14d2d7/conda-store-server/conda_store_server/storage.py#L208

[mcg1969]

Since a standard S3 url always includes the bucket name at minimum on the path I suspect conda-store will be fine.

And if the bucket name does not conflict with other API paths you can put the bucket names directly in the ingress so the minio "base" has no subpath

[mcg1969]

We could rename the bucket to avoid the conflict; however, we need to plan how to migrate existing data safely.

This is definitely a good and important callout. One thing I've done in a similar project is to differentiate between fresh installs and upgrades. In short, upgrades would preserve the current configuration, leaving Minio on a dedicated port, while fresh installs would adopt the new approach. This buys time until a clean migration approach can be built.