Support for Ceph Object Storage Gateway

[axoxymous-coder]

Hi,

Is nginx-s3-gateway supported with Ceph Object Storage Gateway? I tried using it with Ceph, but it always errors out with AWS EC2 credentials not found. My Ceph cluster is created using Rook in local K8s cluster.

Here are the settings I am using for Ceph:

S3_BUCKET_NAME=mybucket
AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXX
AWS_SECRET_KEY=YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
S3_SERVER=172.17.0.1
S3_SERVER_PORT=31185
S3_SERVER_PROTO=http
S3_REGION=us-east-1
AWS_SIGS_VERSION=4
ALLOW_DIRECTORY_LIST=true
S3_STYLE=virtual
DEBUG=true
PROVIDE_INDEX_PAGE=false
APPEND_SLASH_FOR_POSSIBLE_DIRECTORY=true
PROXY_CACHE_VALID_OK=1h
PROXY_CACHE_VALID_NOTFOUND=1m
PROXY_CACHE_VALID_FORBIDDEN=30s

Does nginx-s3-gateway only works in AWS environment for S3? In the document however its mentioned that it works for all S3 compatible object storage systems.

Any help will be highly appreciated.

[dekobon]

Is there an easy way for me to set up Cepth to test? I saw that Ceph offers docker images. I'm wondering if there is an easy way to set up a test instance of Ceph configured for the object storage gateway use case.

Also, can you please add the error output from NGINX where it fails to connect to Ceph or "errors out with AWS credentials not found"? The exact text will be useful in debugging.

[axoxymous-coder]

Hi @dekobon

Thanks for your reply.

To setup a Ceph Object Storage in K8s, you can use a managed service like EKS or a local cluster using Kind or Minikube. You need a minimum of 1 control plane node and 3 worker nodes along with 3 5GB empty disk devices. Once you have your K8s cluster, you can use the Rook (https://rook.io/) K8s operator to setup the Ceph storage. Then follow the documentation at https://rook.io/docs/rook/latest-release/ to setup Ceph.

In my scenario, I have a Ceph object storage running locally at 127.0.0.1:31185. I have a bucket called www. With s5cmd tool, here is how I can access my Ceph Object Storage:

s5cmd --credentials-file s5cmd-nginx.cfg --endpoint-url=http://127.0.0.1:31185 ls s3://www/html/index.html 
2023/05/19 05:11:27               285 html/index.html

I have configured the environment settings file for nginx-s3-gateway as below:

S3_BUCKET_NAME=www
AWS_ACCESS_KEY_ID=4C6KA2HEOY19SZO9AY9V
AWS_SECRET_KEY=yuI9lMsAWui6s19FZo8nfXUCl48RALTgLvESAZpy
S3_SERVER=127.0.0.1
S3_SERVER_PORT=31185
S3_SERVER_PROTO=http
S3_REGION=us-east-1
AWS_SIGS_VERSION=4
ALLOW_DIRECTORY_LIST=true
S3_STYLE=virtual
DEBUG=true
PROVIDE_INDEX_PAGE=false
APPEND_SLASH_FOR_POSSIBLE_DIRECTORY=true

Then I started the nginx-s3-gateway (OSS version) docker container as below:

sudo docker run --env-file ./ceph-settings.env --publish 80:80 --name nginx-ceph-gw ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest

The logs are as below:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration 
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ 
/docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh 
S3 Backend Environment 
Access Key ID: 4C6KA2HEOY19SZO9AY9V 
Origin: http://www.127.0.0.1:31185 
Region: us-east-1 
Addressing Style: virtual 
AWS Signatures Version: v4 
DNS Resolvers:  172.31.0.2 
Directory Listing Enabled: true 
Provide Index Pages Enabled: false 
Append slash for directory enabled: true 
Stripping the following headers from responses: x-amz-; 
CORS Enabled: 0 
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version 
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3listing_location.conf.template to /etc/nginx/conf.d/gateway/s3listing_location.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificate.conf.template to /etc/nginx/conf.d/gateway/js_fetch_trusted_certificate.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_server.conf.template to /etc/nginx/conf.d/gateway/s3_server.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/nginx/conf.d/gateway/cors.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_location.conf.template to /etc/nginx/conf.d/gateway/s3_location.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf 
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/upstreams.conf.template to /etc/nginx/conf.d/upstreams.conf 
/docker-entrypoint.sh: Launching /docker-entrypoint.d/22-enable_js_fetch_trusted_certificate.sh 
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh 
/docker-entrypoint.sh: Configuration complete; ready for start up 
2023/05/21 03:01:15 [notice] 1#1: using the "epoll" event method 
2023/05/21 03:01:15 [notice] 1#1: nginx/1.24.0 
2023/05/21 03:01:15 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)  
2023/05/21 03:01:15 [notice] 1#1: OS: Linux 5.19.0-1025-aws 
2023/05/21 03:01:15 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 
2023/05/21 03:01:15 [notice] 1#1: start worker processes 
2023/05/21 03:01:15 [notice] 1#1: start worker process 75 
2023/05/21 03:01:15 [notice] 1#1: start cache manager process 76 
2023/05/21 03:01:15 [notice] 1#1: start cache loader process 77 
2023/05/21 03:02:14 [info] 75#75: *1 js: Cached credentials are expired or not present, requesting new ones 
2023/05/21 03:02:15 [notice] 77#77: http file cache: /var/cache/nginx/s3_proxy 0.000M, bsize: 4096 
2023/05/21 03:02:15 [notice] 1#1: signal 17 (SIGCHLD) received from 77 
2023/05/21 03:02:15 [notice] 1#1: cache loader process 77 exited with code 0 
2023/05/21 03:02:15 [notice] 1#1: signal 29 (SIGIO) received 

When I make a request to the NGiNX gateway with curl -iX GET http://localhost/html/index.html, I get a 500 error with the following log:

2023/05/21 03:02:47 [info] 75#75: *3 js: Cached credentials are expired or not present, requesting new ones 
2023/05/21 03:02:54 [info] 75#75: *5 js: Cached credentials are expired or not present, requesting new ones 
2023/05/21 03:03:15 [info] 75#75: *1 js: Could not load EC2 task role credentials: {} 
2023/05/21 03:03:15 [error] 75#75: *1 auth request unexpected status: 500, client: 172.17.0.1, server: , request: "GET /html/index.html HTTP/1.1", host: "localhost" 
172.17.0.1 - - [21/May/2023:03:03:15 +0000] "GET /html/index.html HTTP/1.1" 500 170 "-" "curl/7.81.0" "-"
2023/05/21 03:03:17 [info] 75#75: *7 js: Cached credentials are expired or not present, requesting new ones 
2023/05/21 03:03:25 [info] 75#75: *9 js: Cached credentials are expired or not present, requesting new ones 
2023/05/21 03:03:47 [info] 75#75: *3 js: Could not load EC2 task role credentials: {} 
2023/05/21 03:03:47 [error] 75#75: *3 auth request unexpected status: 500, client: 172.17.0.1, server: , request: "GET /html/index.html HTTP/1.1", host: "127.0.0.1" 
172.17.0.1 - - [21/May/2023:03:03:47 +0000] "GET /html/index.html HTTP/1.1" 500 170 "-" "curl/7.81.0" "-"
2023/05/21 03:03:54 [info] 75#75: *5 js: Could not load EC2 task role credentials: {} 
2023/05/21 03:03:54 [error] 75#75: *5 auth request unexpected status: 500, client: 172.17.0.1, server: , request: "GET / HTTP/1.1", host: "127.0.0.1" 
172.17.0.1 - - [21/May/2023:03:03:54 +0000] "GET / HTTP/1.1" 500 170 "-" "curl/7.81.0" "-"
2023/05/21 03:04:05 [info] 75#75: *11 js: Cached credentials are expired or not present, requesting new ones 
2023/05/21 03:04:17 [info] 75#75: *7 js: Could not load EC2 task role credentials: {} 

I am wondering since I am running my gateway in a non AWS environment, why its trying to get EC2 task role credentials. I looked at the code and it seems we try to get the AWS signatire and credentials (s3gateway.js) which emits these errors. Is it possible to somehow disable it if the gateway is running in a non AWS environment?

[dekobon]

Can you check that you are running the latest version of the gateway? Preferably you are running from the images from github (not dockerhub). This error looks a lot like an error that was fixed recently.

[dekobon]

I just hit the same problem that you did. The environment variable AWS_SECRET_KEY should be AWS_SECRET_ACCESS_KEY.

[axoxymous-coder]

@dekobon Thank you so much! I will try this. Appreciate your time on this.

[dekobon]

Did this solve your problem?

[mxm-tr]

In case you are stil having issues: lI think that S3_STYLE has to be configured with "path" for it to work with Ceph RADOS Gateway.

[dekobon]

I'm going to make this as closed. If you are still having problems, please comment or reopen the issue.