Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strip leading directory path from request #158

Closed
wants to merge 13 commits into from
Closed
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Dockerfile.oss
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ ENV PROXY_CACHE_VALID_NOTFOUND "1m"
ENV PROXY_CACHE_VALID_FORBIDDEN "30s"
ENV CORS_ENABLED 0
ENV DIRECTORY_LISTING_PATH_PREFIX ""
ENV STRIP_LEADING_DIRECTORY_PATH ""

# We modify the nginx base image by:
# 1. Adding configuration files needed for proxying private S3 buckets
Expand Down
1 change: 1 addition & 0 deletions Dockerfile.plus
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ ENV PROXY_CACHE_VALID_NOTFOUND "1m"
ENV PROXY_CACHE_VALID_FORBIDDEN "30s"
ENV CORS_ENABLED 0
ENV DIRECTORY_LISTING_PATH_PREFIX ""
ENV STRIP_LEADING_DIRECTORY_PATH ""

COPY plus/etc/ssl /etc/ssl
COPY plus/usr /usr
Expand Down
1 change: 1 addition & 0 deletions common/etc/nginx/nginx.conf
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ env PROXY_CACHE_VALID_NOTFOUND;
env PROXY_CACHE_VALID_FORBIDDEN;
env HEADER_PREFIXES_TO_STRIP;
env FOUR_O_FOUR_ON_EMPTY_BUCKET;
env STRIP_LEADING_DIRECTORY_PATH;

events {
worker_connections 1024;
Expand Down
7 changes: 6 additions & 1 deletion common/etc/nginx/templates/default.conf.template
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,15 @@ include /etc/nginx/conf.d/gateway/v${AWS_SIGS_VERSION}_js_vars.conf;
# Extracts only the path from the requested URI. This strips out all query
# parameters and anchors in order to prevent extraneous data from being sent
# to S3.
map $request_uri $uri_path {
map $request_uri $uri_full_path {
"~^(?P<path>.*?)(\?.*)*$" $path;
}

# Remove a portion of request URL (if configured)
map $uri_full_path $uri_path {
"~^$STRIP_LEADING_DIRECTORY_PATH(.*)" $1;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm in the process of trying to test this with the project's test suite, but I think this might need a default clause.

   map $uri_full_path $uri_path {
        "~^$STRIP_LEADING_DIRECTORY_PATH(.*)" $1;
        default $uri_full_path;
    }

The code in the PR will work when theSTRIP_LEADING_DIRECTORY_PATH is not specified, and for /mybucket/a/b/foo.txt if STRIP_LEADING_DIRECTORY_PATH=/mybucket is specified.

However, if STRIP_LEADING_DIRECTORY_PATH=/mybucket is specified, then uris that do not include that prefix I think will not match and then the value of $uri_path will be empty.

It's very possible I'm missing something but would like to get your thoughts.


map $S3_STYLE $s3_host_hdr {
virtual "${S3_BUCKET_NAME}.${S3_SERVER}";
path "${S3_SERVER}:${S3_SERVER_PORT}";
Expand Down
4 changes: 2 additions & 2 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ running as a Container or as a Systemd service.
| `JS_TRUSTED_CERT_PATH` | No | | | Enables the `js_fetch_trusted_certificate` directive when retrieving AWS credentials and sets the path (on the container) to the specified path |
| `HEADER_PREFIXES_TO_STRIP` | No | | | A list of HTTP header prefixes that exclude headers client responses. List should be specified in lower-case and a semicolon (;) should be used to as a deliminator between values. For example: `x-goog-;x-something-` |
| `CORS_ENABLED` | No | `true`, `false` | `false` | Flag that enables CORS headers on GET requests and enables pre-flight OPTIONS requests. If enabled, this will add CORS headers for "fully open" cross domain requests by default, meaning all domains are allowed, similar to the settings show in [this example](https://enable-cors.org/server_nginx.html). CORS settings can be fine-tuned by overwriting the [`cors.conf.template`](/common/etc/nginx/templates/gateway/cors.conf.template) file. |
| `CORS_ALLOWED_ORIGIN` | No | | | value to set to be returned from the CORS `Access-Control-Allow-Origin` header. This value is only used if CORS is enabled. (default: \*) |

| `CORS_ALLOWED_ORIGIN` | No | | | value to set to be returned from the CORS `Access-Control-Allow-Origin` header. This value is only used if CORS is enabled. (default: \*)
| `STRIP_LEADING_DIRECTORY_PATH` | No | | | Removes a portion of the path in the requested URL (if configured). Useful when deploying to an ALB under a folder (eg. www.mysite.com/mybucket).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💚 Thank you for the nice documentation entry


If you are using [AWS instance profile credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html),
you will need to omit the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` variables from
Expand Down
3 changes: 2 additions & 1 deletion settings.example
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ S3_BUCKET_NAME=my-bucket
AWS_ACCESS_KEY_ID=ZZZZZZZZZZZZZZZZZZZZ
AWS_SECRET_ACCESS_KEY=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
AWS_SESSION_TOKEN=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
S3_SERVER=s3-us-east-1.amazonaws.com
S3_SERVER=s3.us-east-1.amazonaws.com
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_REGION=us-east-1
Expand All @@ -18,3 +18,4 @@ PROXY_CACHE_INACTIVE=60m
PROXY_CACHE_VALID_OK=1h
PROXY_CACHE_VALID_NOTFOUND=1m
PROXY_CACHE_VALID_FORBIDDEN=30s
STRIP_LEADING_DIRECTORY_PATH=/somepath
2 changes: 2 additions & 0 deletions standalone_ubuntu_oss_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,8 @@ PROXY_CACHE_VALID_NOTFOUND=${PROXY_CACHE_VALID_NOTFOUND:-'1m'}
PROXY_CACHE_VALID_FORBIDDEN=${PROXY_CACHE_VALID_FORBIDDEN:-'30s'}
# Enables or disables CORS for the S3 Gateway (true=enabled, false=disabled)
CORS_ENABLED=${CORS_ENABLED:-'false'}
# Configure portion of URL to be removed (optional)
STRIP_LEADING_DIRECTORY_PATH=${STRIP_LEADING_DIRECTORY_PATH:-''}
EOF

# By enabling CORS, we also need to enable the OPTIONS method which
Expand Down