Support Minio S3-Cache

[richardbischof]

Dear community,

today tegola can make use of s3 as a cache. For local testing and deployment in an fully open source environments, I want to use https://github.com/minio/minio as my object store of choice. Minio is an open source object store, which is broadly used and often considered as an alternative in comparable scenarios.

Sadly tegola can not use minio, because minio does not use the bucket name as an subdomain, like AWS S3 does.

Here is my docker-compose.yaml to demo the issue:

version: '3'
services:
  minio:
    image: minio/minio
    ports:
      - 9000:9000
    environment:
      MINIO_SECRET_KEY: ***
      MINIO_ACCESS_KEY: ***
    volumes:
      - ./minio/storage:/data
    entrypoint: sh
    command: -c 'mkdir -p /data/tegola && /usr/bin/minio server /data'
  tegola:
    image: gospatial/tegola
    ports:
      - 8080:8080
    volumes:
      - ./tegola/config:/config
    command: ["serve","--config","/config/config.toml"]

config.toml

[cache] 
type = "s3" 
bucket = "tegola"
endpoint = "http://minio:9000"
aws_access_key_id = "***"
aws_secret_access_key = "***"

With this configuration, tegola tries to connect to "http://tegola.minio:9000", but the correct path would be "http://minio:9000/tegola". This behavior can be changed by setting the aws-sdk-go S3Clients property S3ForcePathStyle to "true".

With this issue I propose to add this parameter in an configurable way. I already have a working solution, which covers the parsing of the configuration parameter from config.toml and apply it in the same way as other configuration parameters. I would love to make PR. You can find my work here: richardbischof@36b05ba

I would love to have your feedback.

Regards, Richard

[ARolek]

@richardbischof Thanks for the write-up. The S3 cache is designed to support other S3 API compatible back ends, so this request is in alignment with the current design. I have not used minio personally so I'm unfamiliar with their nuances. Adding this config option sounds like a win. I'm absolutely open for a PR if you want to open one up and we can review the proposed implementation there. One request, if it's reasonably easy, would be to add a test case against minio in our GH Action CI. That would be really helpful to make sure no one breaks the implementation going forward.

[richardbischof]

Thank you for your feedback. I took a look on GH Actions, but as far as I understood, there have to be an Minio instance running in the environment, where the action is executed. I have no experience with the setup of GH Actions. Maybe someone can help me?

[ARolek]

@richardbischof apologies for the slow response, I have been traveling but I'm back now.

I can help with setting up minio in the GH Actions. It looks like mino has a docker container we can use which makes this easy. Take a look at the services section of our Githbub action that's run on every push: https://github.com/go-spatial/tegola/blob/v0.13.x/.github/workflows/on_pr_push.yml#L12. We should be able to model the minio setup off of this config fairly easily.

I'm not familiar with running minio, but I'm happy to collaborate on getting this running in the CI. If it's easier we can try to do this in real-time on Slack (http://slack.go-spatial.org/) or you can send in a PR and we can interate there.

[ARolek]

closed by #745