cloudfront invalidation error

[levelos]

First, just want to say thanks for such a great library. We just launched our new Jekyll site on S3 and it's been a big help. Occasionally when running s3_website push I get the following cloudfront error:

AWS API call failed. Reason:

<?xml version="1.0"?>
<ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2012-05-05/">
<Error>
<Type>Sender</Type>
<Code>InvalidArgument</Code>
<Message>Your request contains one or more invalid invalidation paths.</Message>
</Error>
<RequestId></RequestId>
</ErrorResponse> 

(RuntimeError)

Any thoughts?

[laurilehmijoki]

Hi @levelos

Thanks for the encouraging feedback!

The error comes directly from CloudFront. S3_website just echoes it.

Add puts statements into https://github.com/laurilehmijoki/s3_website/blob/master/lib/cloudfront/invalidator.rb. What do you see? Any suspicious paths?

[levelos]

Nope. I think it might happen when a file is deleted from the origin bucket.

[laurilehmijoki]

I'm able to delete a file and then invalidate the distribution. Can you give detailed steps on how to reproduce this bug?

[laurilehmijoki]

I'm closing this issue as there is not enough information to act on.

If you discover the steps to reproduce this problem, please reopen.

[natelandau]

Unfortunately, I'm not a ruby programmer so I don't know how to effectively troubleshoot the issue. I will just chime in and say that I see this ~90% of the times I deploy

[laurilehmijoki]

Reopening. Please provide hints on how to reproduce this bug. Share the git url of your site if it is public. That will help in debugging.

[stammy]

" I will just chime in and say that I see this ~90% of the times I deploy"

I get a cloudfront issue like this with most of my deploys as well, but I think for me because I have 1000+ changed files sometimes (when I change template stuff and it has to update all posts in my jekyll)

[willyt]

I've had it and I suspect it was caused by an apostrophe in a post filename as I haven't had the problem again since I change the ' to a ' I'll try and hunt down the invalidator file and add a puts. I guess I should I just puts s3_object_keys?

[willyt]

here is the output with a puts added to s3_object_keys

Deploying _site/* to thornewyness.co.uk
Calculating diff ... done
Uploading 21 changed file(s)
Upload contact.html [gzipped]: Success!
Upload cv/jenny_wyness.html [gzipped]: Success!
Upload about.html [gzipped]: Success!
Upload project/2010/11/01/Taigh-Solais-Shelters.html [gzipped]: Success!
Upload index.html [gzipped]: Success!
Upload cv/will_thorne.html [gzipped]: Success!
Upload project/2011/08/01/The-Old-Manse.html [gzipped]: Success!
Upload project/2011/08/01/Gorsten-Jetty.html [gzipped]: Success!
Upload project/2011/01/01/House-in-Kilchoan.html [gzipped]: Success!
Upload project/2012/05/01/Morvern.html [gzipped]: Success!
Upload project/2011/08/05/Dunard.html [gzipped]: Success!
Upload project/2012/06/01/Linndhu.html [gzipped]: Success!
Upload project/2013/03/01/Kellan-Mill-Lodge.html [gzipped]: Success!
Upload project/2013/06/01/Seabank-cottage.html [gzipped]: Success!
Upload project/2013/04/01/Derryguaig.html [gzipped]: Success!
Upload project/2013/08/01/Rainydays-soft-play.html [gzipped]: Success!
Upload project/2013/12/01/Brown's-Shed.html [gzipped]: Success!
Upload project/2014/01/01/USCA-community-housing.html [gzipped]: Success!
Upload services.html [gzipped]: Success!
Upload project/2014/02/01/Old-School-House.html [gzipped]: Success!
Upload work.html [gzipped]: Success!
Done! Go visit: http://thornewyness.co.uk.s3-website-us-east-1.amazonaws.com/index.html
contact.html
about.html
cv/jenny_wyness.html
cv/will_thorne.html
index.html
project/2010/11/01/Taigh-Solais-Shelters.html
project/2011/01/01/House-in-Kilchoan.html
project/2011/08/01/Gorsten-Jetty.html
project/2011/08/01/The-Old-Manse.html
project/2012/05/01/Morvern.html
project/2011/08/05/Dunard.html
project/2012/06/01/Linndhu.html
project/2013/04/01/Derryguaig.html
project/2013/03/01/Kellan-Mill-Lodge.html
project/2013/06/01/Seabank-cottage.html
project/2013/12/01/Brown's-Shed.html
project/2013/08/01/Rainydays-soft-play.html
project/2014/01/01/USCA-community-housing.html
project/2014/02/01/Old-School-House.html
services.html
work.html

AWS API call failed. Reason:
<?xml version="1.0"?>
<ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2012-05-05/"><Error><Type>Sender</Type><Code>InvalidArgument</Code><Message>Your request contains one or more invalid invalidation paths.</Message></Error><RequestId>8c29afca-9e6e-11e3-8928-210ba62fdd28</RequestId></ErrorResponse> (RuntimeError)

It still does it even with the '

[willyt]

And if I get rid of the apostrophe either as a ' or as a ' I have no problem.

[laurilehmijoki]

I just released the version 1.6.12, which should fix this problem.

Before sending the CloudFront invalidation request, S3_website did not properly encode the file paths. Now it encodes them like this: 55397d2.

Please get back to me if you spot any errors on the solution.

[levelos]

That's great, thanks!

[tkrotoff]

I still have this problem using v2.8.3 on OS X 10.10:

[info] Deploying _site/* to osteo15.com
[succ] Updated sitemap.xml (max-age=864000 | application/xml | gzip)
[succ] Updated 2013/11/22/Interview-sur-l'osteopathie-et-les-femmes-enceintes/index.html (max-age=300 | text/html; charset=utf-8 | gzip)
[succ] Updated tags/index.html (max-age=300 | text/html; charset=utf-8 | gzip)
[succ] Updated feed.xml (max-age=864000 | application/rss+xml | gzip)
[fail] Failed to invalidate the CloudFront distribution (Your request contains one or more invalid invalidation paths. (Service: AmazonCloudFront; Status Code: 400; Error Code: InvalidArgument; Request ID: 72171ed8-bc2f-11e4-b56b-83215c24fdb7))
[info] Summary: Updated 4 files. Transferred 44.5 kB, 144.0 kB/s.
[fail] Failed to push the website to ...

If I modify a file that does not contain ' in its name + push, then no problem:

[info] Deploying _site/* to osteo15.com
[succ] Updated sitemap.xml (max-age=864000 | application/xml | gzip)
[succ] Updated 2014/02/17/Medicaments,-grossesse-et-allaitement/index.html (max-age=300 | text/html; charset=utf-8 | gzip)
[succ] Updated feed.xml (max-age=864000 | application/rss+xml | gzip)
[succ] Invalidated 4 items on CloudFront
[info] Summary: Updated 3 files. Transferred 35.3 kB, 108.0 kB/s.
[info] Successfully pushed the website to ...

Another test by modifying another file containing a ' + push:

[info] Deploying _site/* to osteo15.com
[succ] Updated sitemap.xml (max-age=864000 | application/xml | gzip)
[succ] Updated 2014/02/19/Questions-reponses-sur-l'hypnose-avec-Delphine-Bourdet/index.html (max-age=300 | text/html; charset=utf-8 | gzip)
[succ] Updated feed.xml (max-age=864000 | application/rss+xml | gzip)
[fail] Failed to invalidate the CloudFront distribution (Your request contains one or more invalid invalidation paths. (Service: AmazonCloudFront; Status Code: 400; Error Code: InvalidArgument; Request ID: e0919706-bc2f-11e4-b56b-83215c24fdb7))
[info] Summary: Updated 3 files. Transferred 38.2 kB, 123.0 kB/s.
[fail] Failed to push the website to ...

FYI

• List of valid URL characters: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;=
• List of invalid filename characters under Windows: \/:*?"<>|
• List of invalid filename characters under UNIX (depends on the filesystem): NUL (\0) and /

[laurilehmijoki]

Hi @tkrotoff, I just released the version 2.8.4, which URL-encodes the ' character.

Even though ' is a valid URL character, CloudFront does not accept it unless it's URL-encoded. I don't know why this is the case.

Here's some documentation on the requirements of a valid invalidation path.

[tkrotoff]

@laurilehmijoki tested, works great 👍

[laurilehmijoki]

Thanks for testing, @tkrotoff!

[mobidev111]

@laurilehmijoki I get the InvalidArgument error with 2.8.4 when there are filenames with special characters: ä ö ü

[laurilehmijoki]

@mobidev111 try upgrading to 2.8.5 – does the problem disappear?

I made a specific rule for the chars (ä|ö|ü), because I didn't have time to write the general rule.

Can someone help in defining the general rule?

[akshaykarle]

@laurilehmijoki I've attempted to generalize this by escaping all non-ascii characters in #190. I feel this should handle most of the cases. Can you take a look?

[laurilehmijoki]

Thanks @akshaykarle! Your pull request is released in version 2.12.2.