-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Long time delay associated with head requests to Amazon S3 #2461
Comments
Hi. Yes, I have similar issue with |
Solved issue with replacing |
Had the same issue. I used |
@alexshk @felipefava do you still have the HEAD requests with For me it's too slow with |
Anyone still having the issue? |
Hi, we had a similar problem and looked into this problem deeper with stack profiling. This is the minimum reproducible example class AvatarUploader < CarrierWave::Uploader::Base
version :jpg_version, if: :webp? do
process convert: :jpg
end
protected
def webp?(new_file)
new_file.content_type == 'image/webp'
end
end
class User < ApplicationRecord
mount_uploader :avatar, AvatarUploader
end
# This evaluates the condition of :jpg_version even :jpg_version is not used
# which requests to S3 to get the content_type if storage is remote
User.first.avatar.url I think that the condition should not be evaluated until the actual version is called on retrieving, but should always be called on storing, ended up with this fix #2669 |
Thank you for the report. It's very helpful to understand the actual use-case that causes this issue. But before proceeding I need to confirm one thing. Why do you need to check if the file is webp or not? Isn't it simpler to convert every image file into jpg, regardless of the source format? |
Thank you for looking into this issue.
We were using conditional processing, because we don't need to convert the image in most cases, and we thought that it would be much more cost efficient (don't need to process nor uploading duplicated jpg files). The example in above might be a specific usage, but I think that lot of people are doing similar conditional processings, for example, this wiki example needs remote request too https://github.com/carrierwaveuploader/carrierwave/wiki/How-to:-Do-conditional-processing. Also, we, and I guess most users that are reporting similar issues too, were not aware that the all versions' conditions are checked every time we called the mounted column (#2132). Much more, I believe it is difficult to assume that the HEAD request is called even when we are not using the actual version. |
@mshibuya |
Hello. We use carrierwave to handle attachments stored by Amazon S3. Files in bucket are private. Our API sometimes has to return data about many attachments (100 for instance). In this endpoints we're observing a big delay in response (the more attachments, the longer response time). We've used New Relic to investigate what affects this time, and turns out that HEAD requests are performing one by one (not simultaneously):
Moreover, this causes high CPU utilization (sometimes over 50%) - the more S3 requests, the more CPU utilization.
Has anyone encountered a similar problem?
The text was updated successfully, but these errors were encountered: