5 - Support uploads to S3 via CloudFront distribution

[rnicholus]

The current plan is to support simple (non-chunked) uploads to a CloudFront distribution. Chunked uploads are currently not possible when targeting a CloudFront distribution since CloudFront rips off the Authorization header containing the signature before forwarding the request on to S3. The Authorization header is a required field in the request when using any of the S3 multipart upload REST calls, which are needed to support Fine Uploader S3's chunking and auto-resume features. I have opened a request in the CloudFront forums asking for this behavior to be modified so multipart uploads requests can target a CloudFront distribution.

The planned support for this is still mostly undetermined, and it is also not determined if this support will be part of 4.0. Making this part of 4.0 is possible, but looking less likely as I run into issues with CloudFront's handling of upload-related requests. I'm currently struggling to get path patterns for upload requests to work. I've opened another thread in the forum detailing my issue at https://forums.aws.amazon.com/thread.jspa?threadID=137627&tstart=0.

[rnicholus]

No useful responses from Amazon in the forums. This is going to be postponed until a later release.

[rnicholus]

Amazon has confirmed that they have no plans to stop removing the Authorization header from requests. This means that we will not be able to make use of the multipart upload API via a CloudFront distribution.

[jasonshah]

So, is it fair to say that simple uploads to Cloudfront are not yet supported? We are evaluating uploaders, and one of our customers wants to upload huge (multi-GB) files. Uploading to S3 with Fine Uploader is proving to be severely limited by their proxy, and I wanted to try uploads straight to Cloudfront.

[rnicholus]

The only possible way to upload files that target a CloudFront distribution is to send the entire file in one multipart encoded POST request. This means that chunking, resume, and pause features are not possible. If a file fails mid-upload, each retry attempt will need to start over from the first byte. In fact, no credentialed REST calls are possible at all, since AWS rips off the Authorization header.

If you do not enable the chunking/resume features in Fine Uploader S3, theoretically, uploads to a CF distribution should work, but there is another problem: Fine Uploader S3 expects to be able to determine the bucket name from your endpoint. If you are uploading to a CF distribution, this assumption is no longer valid. We would have to allow a bucket name to be specified via an option and API method for CF distro endpoints.

[rnicholus]

@jasonshah What sort of limits does your customer's proxy enforce?

[jasonshah]

@rnicholus the customer's proxy has some kind of bandwidth throttling in place. In one location, they get 3.5MB/s up (which is very acceptable), whereas in another they get <100K/s up (which isn't). Running some traceroutes and nslookups against the slow computer reveals a proxy server in the way, which is likely limiting bandwidth in some way. We've asked to see if there's a way to get in touch with their network engineers, but it's a huge company and they treat that as a last resort. So one theory is that perhaps uploading to Cloudfront might help speed this up.

[rnicholus]

@jasonshah If uploading to CF fixes the issue, I can see why that might be appealing. The reason I haven't pursued support for CF in Fine Uploader S3 is due to a few reasons:

• I see chunking, resume, and efficient retries as very important features. These would have to be turned off if uploading to CF.
• Since a bucket name would have to be explicitly specified alongside a CF endpoint URL, this adds more complexity and confusion to the API/options.
• There are other outstanding issues with CF that make uploads potentially difficult, such as this one: https://forums.aws.amazon.com/thread.jspa?threadID=137627&tstart=0.
• AWS doesn't seem interested in supporting the upload-to-CF workflow. There's no telling what other issues may appear over time if we explicitly support this.

[jasonshah]

@rnicholus that last issue seems to be the biggest one. The feature was announced four months ago, however I've yet to find a single example of how this might work from them (though perhaps I haven't looked hard enough).

Thanks for thinking of it. We'll keep looking to see if we can solve this problem another way.

[jasonshah]

@rnicholus Some interesting data, FYI: our customer demonstrated that using the Fine Uploader test to upload from NYC to US East (Virginia), he can achieve ~3-7MB/s. Using another product's uploader, which does a simple PUT to a CDN (EdgeCast in this case), he can achieve 55MB/s. The CDN can provide huge speed increases.

[rnicholus]

I'm afraid I'm not familiar with this CDN. Is the customer cutting AWS out of the picture entirely?

[pulkitjalan]

Amazon have now said that cloudfront does not remove the Authorization header on PUT, POST, PATCH, DELETE, and OPTIONS.

https://forums.aws.amazon.com/message.jspa?messageID=528729#528729

[rnicholus]

Excellent. Thanks for the update. That should make it possible for us to modify Fine Uploader S3 to allow uploads through a CF distribution, theoretically.

[pulkitjalan]

I was testing this with fine uploader and I ran into another issue. It was to do with the fact that cloudfront adds the 'X-Amz-Cf-Id' header to the request. I got past this issue by using Origin Access Identities as outlined in this forum post: https://forums.aws.amazon.com/thread.jspa?messageID=345913&#345913.

Looking forward it seeing this feature in fine uploader :)

[jasonshah]

@pulkit-clowdy after setting up the OAI, you were able to use FineUploader to upload to S3 via CloudFront?
If yes, did you experience any performance improvements?

[pulkitjalan]

I was able to upload via cloudfront and use chunking. Yes there was a significant improvement in performance considering my bucket is in the us-east-1 region and im uploading from the UK.

Direct to S3 = ~1MB/s
S3 via Cloudfront = ~4-6MB/s

[pulkitjalan]

At the moment its quite hacky to get this working, almost all security checks have to be disabled.
Is this feature going to be implemented into fineuploader and if so, which version it is planned?

[rnicholus]

Probably 5.1. 5.0 is currently in development.

On Sunday, March 23, 2014, Pulkit Jalan [email protected] wrote:

At the moment its quite hacky to get this working, almost all security
checks have to be disabled.
Is this feature going to be implemented into fineuploader and if so, which
version it is planned?

Reply to this email directly or view it on GitHubhttps://github.com//issues/1016#issuecomment-38382491
.

[pulkitjalan]

Ok, thanks for the update

[cybertrand]

First of all, thank you @rnicholus for building such a useful piece of software. I'm also looking to use Fine Uploader to upload content to S3 via CloudFront. I understand from this thread that this should be released in 5.1 and was wondering if you had an idea when that might be?

The company I work for is looking to implement a Web uploader with this specific feature. We might be able to contribute to the project to help develop it if that's something you're interested in.

[rnicholus]

Thanks for the kind words @cybertrand. Fine Uploader wouldn't be where it is today without my employer, Widen, and the development help of @feltnerm along with the input of @uriahcarpenter as well.

You are correct that this feature is scheduled for 5.1, along with several others. I don't think this will be terribly difficult, assuming there aren't any further hidden obstacles (such as the issue where CF stripped Authorization headers a while back, making this impossible - since fixed).

We are currently working on a hotfix and some administrative (non-feature) tasks at the moment. Once those are complete, #1198 is first in line, followed by uploads to CF.

I suspect that the code changes to Fine Uploader S3 will be minimal to support uploads to CF. One thing that will need to change is the code that uses the AWS endpoint to determine the bucket name. You see, we must embed the bucket name in the request, and, with uploads directly to S3, we can programmatically determine the bucket name simply by looking at the S3 endpoint URL. With uploads to a CF distro, that will no longer be a safe assumption, so we will need to solicit the actual bucket name from integrators via an additional option (and provide an API method for dynamic adjustment).

[cybertrand]

Thank you very much for the additional information @rnicholus; makes sense about the S3 bucket name. It's great to hear that this should be straightforward to implement and that it should happen soon. On that last note, are you able to share any rough timeframes: are we talking about 1, 3, 6 months? I'm asking so that we can make the best decision on waiting vs. implementing now.

I was also wondering: will the implementation of this feature include the ability to upload multiple chunks/parts of the same file in parallel? (to S3 via CloudFront). That's what we're after in order to accelerate file uploads and this would provide a really nice commodity solution, instead of having to buy and implement a UDP based transfer acceleration solution. Note: it would be really useful to have a similar solution for accelerated downloads, whereby your JS client would perform multiple HTTP Range requests to CloudFront/S3 in parallel do download a single large file (I believe both S3 and CloudFront support this).

Thanks again!

[rnicholus]

Have you read about the concurrent chunking feature we released in 5.0?
That allows multiple chunks for a single large file to be uploaded in
parallel to any endpoint. That feature is aimed at single-file large
uploads, since multiple chunks have always been uploaded in parallel (one
per file) if multiple files are selected.
http://docs.fineuploader.com/branch/master/features/concurrent-chunking.html

On Wed, Jun 25, 2014 at 3:22 PM, cybertrand [email protected]
wrote:

Thank you very much for the additional information @rnicholus
https://github.com/rnicholus; makes sense about the S3 bucket name.
It's great to hear that this should be straightforward to implement and
that it should happen soon. On that last note, are you able to share any
rough timeframes: are we talking about 1, 3, 6 months? I'm asking so that
we can make the best decision on waiting vs. implementing now.

I was also wondering: will the implementation of this feature include the
ability to upload multiple chunks/parts of the same file in parallel? (to
S3 via CloudFront). That's what we're after in order to accelerate file
uploads and this would provide a really nice commodity solution, instead of
having to buy and implement a UDP based transfer acceleration solution.
Note: it would be really useful to have a similar solution for accelerated
downloads, whereby your JS client would perform multiple HTTP Range
requests to CloudFront/S3 in parallel do download a single large file (I
believe both S3 and CloudFront support this).

Thanks again!

—
Reply to this email directly or view it on GitHub
#1016 (comment)
.

[cybertrand]

I did read about it @rnicholus and was wondering if it would be supported for uploads to S3 via CF and for a single file specifically.

I ran some tests doing multipart uploads of single large files to S3 with 3-5 chunks in parallel via a native client app (Cyberduck, also open source) from Los Angeles to a S3 bucket in US Standard and can vouch for the fact that it yields huge gains in throughput (consistently reaching 90+Mbps on a dedicated 100Mbps). Similarly, although a single file/chunk upload to CF will yield much better throughput than one to S3 (since it's uploading to a local CF node), transfers could be further accelerated by allowing multiple chunks in parallel. I don't know how hard this would be to implement and if it requires a lot of changes compared to what you built for S3, just some food for thoughts..

I can also say from my experience that your call to only implement this feature with chunking in order to allow pause/resume as well as the automatic retry of failed upload was a good call. We've had recurring issues uploading single large files to S3 via another CDN: some uploads to S3 fail, our CDN provider confirms a failed PUT to S3 but we're not get any additional information from Amazon...

Sorry, don't mean a nag, but are you able to share any timeframe for the release of 5.1? Thanks a lot for responding so quickly to everything!

[rnicholus]

The concurrent chunking feature was implemented as a core feature in Fine Uploader 5.0. This means that it is supported for all endpoint types: Azure, S3, custom/traditional. I see no reason why a CF distro would be a special case.

I don't yet have a time estimate for 5.1. Stay tuned, and I'll try to post it when I know more.

[cybertrand]

Great. I'll watch the thread for updates on 5.1. Thanks again!

[rbliss]

I've been testing the x-amz-meta-#### headers, and they're working great. Other headers appear to come through fine too.

After further discussion with the guys at Amazon, it appears the specific issue I was having, where the headers don't appear to come through, is really due to setting up an 'Origin Access Identity' (OAI) on Cloudfront. You have to be very careful when you setup an OAI to understand what's going on. An OAI will cause Cloudfront to behave almost as if a different user (the OAI) is creating the S3 object, rather than the user whose access key you specify in FineUploader.

An OAI is optional, but If using an OAI, you should use the acl of bucket-owner-full-control instead of private to get an S3 object that resembles what would normally happen if you hit S3 directly.

Other than that, I'm happy with the results I'm seeing from Cloudfront and it appears the earlier issues are no longer a problem.

[rnicholus]

My understanding was that an OAI is required when making signed requests, otherwise the additional headers attached to the request by CF will be rejected by S3 as they are not accounted for in the signature.

In the case of "bucket-owner-full-control", I'm guessing the "owner" is CF when using an OAI. Is this your understanding? If this is true, then based on the Canned ACL descriptions, public-read will result in the same problem for the bucket owner it seems.

[rbliss]

Just a point of clarification: by signed requests, are you referring to the signed Authorization header required to authenticate a request in the S3 REST API, or signed urls used to restrict access to objects in S3?

[rnicholus]

The former. This is how Fine Uploader signs all upload requests.

On Mon, May 18, 2015 at 4:05 PM, Richard Bliss [email protected]
wrote:

Just a point of clarification: by signed requests, are you referring to
the signed Authorization header required to authenticate a request in the
S3 REST API, or signed urls used to restrict access to objects in S3?

—
Reply to this email directly or view it on GitHub
#1016 (comment)
.

[rbliss]

Just wanted to double check. You do not need an OAI to send signed requests. In fact, if you turn off the OAI (or never set one up) on your Cloudfront distro, everything should just work like you were interacting with S3 directly.

[rnicholus]

That wasn't my experience before. In fact, the use of OAIs in this context
was discussed on the AWS forums. One example can be found at
https://forums.aws.amazon.com/thread.jspa?messageID=345913&#345913. I'll
have to try again without an OAI at some point.

On Mon, May 18, 2015 at 4:24 PM, Richard Bliss [email protected]
wrote:

Just wanted to double check. You do not need an OAI to send signed
requests. In fact, if you turn off the OAI (or never set one up) on your
Cloudfront distro, everything should just work like you were
interacting with S3 directly.

—
Reply to this email directly or view it on GitHub
#1016 (comment)
.

[jasonshah]

@rbliss Have you had a chance to measure what kind of performance or stability gains you are seeing with uploads to Cloudfront vs. uploads to S3?

[rbliss]

@jasonshah I haven't had a chance. Anecdotally, it does seem faster
@rnicholus Definitely give it a shot without OAI. Life will be glorious.

[rbliss]

I do have to warn anyone attempting to use Cloudfront, Cloudfront is a bit fiddly in terms of configuration. If you're having issues, it's likely with how you've setup Cloudfront. I can write up configuration details if anyone is testing this.

[winzig]

I've been following this thread for a while with great interest. It sounds as if this is now working, but it's not totally clear what the procedure is to set CF up to use with FineUploader. Will this be documented?

[rnicholus]

Once this is tested and verified on our end, we'll update the S3 Uploads feature page with CF-specific details.

[rbliss]

Here are my notes on setting up a proper CF distribution:

Origin Settings

• Restrict Bucket Access (shows up after you choose your S3 bucket) - Easiest if you choose No see Notes on OAI below.

Default Cache Behavior Settings

• Allowed HTTP Methods - Must choose GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE else you can't create an S3 object.
• Forward Query String - Must choose Yes, else chunked uploads will fail.
• Restrict Viewer Access - Easiest if you choose No else you'll have to do a lot of tweaking to send a signed Cloudfront URL in addition to all the S3 url parameters.

Notes on OAI: If you do setup an OAI, make sure to set FineUploader's objectProperties.acl to bucket-owner-full-control or appropriate equivalent. A Cloudfront distribution with an OAI setup on an origin pointing to a S3 bucket will cause Cloudfront to act as the OAI user when interacting with the S3 bucket regardless of the access key specified in FineUploader. Hence any objects with an acl of private will belong to Cloudfront's OAI user and not the access key's user.

[ludofleury]

My bad, typo. @rbliss I confirm that with your informations, the upload is going smoothly through Cloudfront to S3. Thank you very much.

[rnicholus]

Sounds like this is finally working with Fine Uploader S3. I'm continually
busy with support and licensing requests, as well as the S3 V4 support
feature, but it is still on my radar to verify and update the docs.

On Mon, Oct 5, 2015 at 9:36 AM Ludovic Fleury [email protected]
wrote:

My bad, typo. @rbliss https://github.com/rbliss I confirm that with
your information, the upload is going smoothly through Cloudfront to S3.
Thank you very much.

—
Reply to this email directly or view it on GitHub
#1016 (comment)
.

[andreij]

I don't know if it's the right place to ask, but maybe your answers can help other people stuck like me.

I have set up a basic uploader page, S3 + CORS and works as expected.

When I switch to the CDN upload, the response is:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
   <Code>MethodNotAllowed</Code>
   <Message>The specified method is not allowed against this resource.</Message>  
   <Method>POST</Method>
   <ResourceType>OBJECT</ResourceType>
   <RequestId>2B0507CBD403F9C3</RequestId>    
 <HostId>nEV/gHkAE3wTCKP+ZJzV/VQXNEvFP+/w2UZtyIwTrkkV+/i9aFK/Ap8qTfGfDZ+PAl2kAYD7vE4=</HostId>
</Error>

The response header are:

Access-Control-Allow-Method POST, PUT, DELETE
Access-Control-Allow-Origin *
Access-Control-Max-Age 3000
Allow GET, DELETE, HEAD, PUT
Connection keep-alive
Content-Type application/xml
Server AmazonS3
Transfer-Encoding chunked
Vary    Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Via 1.1 5d53b9570a535c2d94ce93c20abbd471.cloudfront.net (CloudFront)
X-Amz-Cf-Id dw-N_pLmOmEqlbri-B2l7l2a6TGWm2_tAhr5y9_InMU8ZuHYunZSEw==
X-Cache Error from cloudfront

Below the default behavior configuration:

No OAI set.

What I have changed into fineuploader conf is:

request: {
  endpoint: 'dqu6rri____.cloudfront.net',
},             
objectProperties: {
  bucket: 'fineup____-test'
},

Bucket policy:

{
    "Version": "2012-10-17",
    "Id": "Policy1447166234666",
    "Statement": [
        {
            "Sid": "Stmt1447166228927",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::fineup____-test/*"
        }
    ]
}

Cors:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>ETag</ExposeHeader>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

The bucket is located in Ireland.

I'm quite clueless, so any help on how to track down the problem would help.

[rnicholus]

Youlll need to fix your comment above. Take another look.

[rnicholus]

Either way, this is a question for the AWS forums, as it appears to be some type of configuration issue on your end.

[andreij]

Dear @rnicholus if you think this is the wrong place I can remove it in order to not clutter the issue's conversation.

[rbliss]

@andreij Place it in AWS Cloudfront forums and I'll respond.

[andreij]

@rbliss Thanks! Here the follow up https://forums.aws.amazon.com/thread.jspa?threadID=219620

[nalbion]

@rnicholus @rbliss I'm getting an error when I try to upload large files to S3 through Cloudfront:

<Error>
  <Code>AccessDenied</Code>
  <Message>There were headers present in the request which were not signed</Message>
  <HeadersNotSigned>x-amz-cf-id</HeadersNotSigned>
  <RequestId>123</RequestId>
  <HostId>abc=</HostId>
</Error>

FineUploader config:

var endpoint = 'mydomain.com';
var uploader = new qq.s3.FineUploader({
            // debug: true,
            element: el,
            request: {
                endpoint: 'https://' + endpoint,
                accessKey: s3AccessKey,
                params: metadata
            },
            chunking: {
                enabled: true
            },
            objectProperties: {
                bucket: bucket,
                host: endpoint,
                region: 'ap-southeast-2',
                serverSideEncryption: true
            },
            signature: {
                endpoint: domain + '/file-uploads/sign-request',
                version: 4,
                customHeaders: customHeaders
            },

Cloudformation template:

Resources:
  Distribution:
    Type: "AWS::CloudFront::Distribution"
    Properties:
      DistributionConfig:
        Aliases:
          - !Sub files.${HostedZone}
        DefaultCacheBehavior:
          AllowedMethods:
            - "DELETE" - "GET" - "HEAD" - "OPTIONS" - "PATCH" - "POST" - "PUT"
          CachedMethods:
            - "GET" - "HEAD" - "OPTIONS"
          Compress: true
          ForwardedValues:
            QueryString: true
          TargetOriginId: "Upload Bucket"
          ViewerProtocolPolicy : "redirect-to-https"
        Enabled: true
        HttpVersion: "http2"
        Origins:
          - DomainName: !Sub ${BucketName}.s3-ap-southeast-2.amazonaws.com
            Id: "Upload Bucket"
            CustomOriginConfig:
              HTTPPort: 80
              HTTPSPort: 443
              OriginProtocolPolicy: https-only
        ViewerCertificate:
          AcmCertificateArn: !Ref Certificate
          SslSupportMethod: sni-only

  Bucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: !Ref BucketName
      CorsConfiguration:
        CorsRules:
          - AllowedOrigins:
              - "*"
            AllowedHeaders:
              - "*"
            AllowedMethods:
              - "GET" - "POST" - "PUT" - "DELETE"
            ExposedHeaders:
              - "Date"
              - "ETag"

  S3BucketPolicy:
    Type: "AWS::S3::BucketPolicy"
    DependsOn: Bucket
    Properties:
      Bucket:
        !Ref BucketName
      PolicyDocument:
        Statement:
          - Sid: DenyIncorrectEncryptionHeader
            Action:
              - "s3:PutObject"
            Effect: "Deny"
            Resource: !Sub arn:aws:s3:::${BucketName}/*
            Principal: "*"
            Condition:
              StringNotEquals:
                s3:x-amz-server-side-encryption:
                  - "AES256"
          - Sid: DenyUnEncryptedObjectUploads
            Action:
              - "s3:PutObject"
            Effect: "Deny"
            Resource: !Sub arn:aws:s3:::${BucketName}/*
            Principal: "*"
            Condition:
              StringEquals:
                s3:x-amz-server-side-encryption:
                  - !Ref AWS::NoValue

[nalbion]

Okay... I changed

- DomainName: !Sub ${BucketName}.s3-ap-southeast-2.amazonaws.com

to

- DomainName: !Sub ${BucketName}.s3.amazonaws.com

(removed the region) and ...I've moved onto the next error: The request signature we calculated does not match the signature you provided. Check your key and signing method.

The error says that the CanonicalRequest included host:my-bucket.s3.amazonaws.com

...oh - objectProperties.host should be my-bucket.s3.amazonaws.com, not endpoint - it's all working now 🍾 🎉