Wrong mtime usage for item.source_type=file converjon images - HIGH load #135
Comments
|
Even though converjon should be used behind a caching reverse proxy that should mitigate this problem a bit it should be working better. Marked as bug. |
|
The problem remains. Old image timestamp1 In case of:
Debug-Output 1) original Image stats stat fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg 2) .../cGc/meta file aside the generated image-cacheobject {"headers":{"Cache-Control":"max-age=3600","date":"Mon, 15 Aug 2016 06:47:21 GMT"},"analysis":{"format":"jpeg","width":540,"height":270}} 3) stat of generated image cache-object stat .../cGc/source/dHRvc2l6/ZV9fNTQw/XzI3MF9j/ZmVmNGJi/ZWVjNmI5/YmE1MzE3/MDk2Z 4) Debug of converjon [DEBUG] 30 Found local file .../fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg [DEBUG] 31 Found local file .../fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg Proposal in case of max-age:
|
@neffets but that proposal is exactly how it worked before I changed it to using file-stats. The cache item's timestamp is that to the current date, the moment it is created and it is used until it's headers:
Cache-Control: "max-age:3600"This will make the items valid for 1 hour and it will not be copied again during that time. |
|
After thinking about it some more: using stat-mtime for the timestamp of the cache items is actually a regression and could severely reduce performance. I'll revert that change and publish a new version with the "old" method that uses the time of cache-item creation. |
|
Hmhm?
All with an old image. Looking at line 219 in lib/source.js (which was changed): yes it try to use the stat of the file source_headers The debug-log shows, that
I want to see the message "Metadata is still fresh" from function "meta_data_still_fresh" ath line 143 of file lib/cache.js |
|
To fix the original issue here: Can you provide the configuration for the alias that you're using? Specifically the |
|
headers: |
|
Make sure that the garbage collector config is not set to 'immediate' for target files (the default value). Set it to 'cache'. Otherwise the GC will clear the generated images after a very short time regardless of their timestamp and headers:
Cache-Control: "max-age=3600"
garbage_collector:
target: 'cache'When using this configuration, clearing the entire cache and requesting the same image with the same parameters, I see one request like this: And two other ones, each a few minutes minutes later: |
|
Your test image is fresh no problem. garbage_collector now set to "cache", max-age to 3600, still problem `[DEBUG] 2 Found local file htdocs/fotolia/essen-trinken/fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg [DEBUG] 2 Locked: source file [DEBUG] 2 Locked: metadata [DEBUG] 2 Locked: target file [DEBUG] 2 Metadata is stale [DEBUG] 2 Item directory cleared [DEBUG] 2 Requesting source htdocs/fotolia/essen-trinken/fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg Mon, 15 Aug 2016 06:47:21 GMT [DEBUG] 2 Requesting analysis [DEBUG] 2 Analysis successful [DEBUG] 2 Writing metadata file [DEBUG] 2 Metadata ready [DEBUG] 2 Source file ready [DEBUG] 2 Target file doesn't exist. Must be regenerated. [DEBUG] 2 Applying cropping and constriants [DEBUG] 2 Requesting image conversion [DEBUG] 2 Image conversion successful [DEBUG] 2 Released lock: metadata [DEBUG] 2 Released lock: source file [DEBUG] 2 Releasing lock: target file "GET /?ts=1471243641&height=270&width=540&file=img%3A%2Ffotolia%2Fessen-trinken%2Ffittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg HTTP 1.0" 200 61614 [DEBUG] 3 Found local file htdocs/fotolia/essen-trinken/fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg [DEBUG] 3 Locked: source file [DEBUG] 3 Locked: metadata [DEBUG] 3 Locked: target file [DEBUG] 3 Metadata is stale [DEBUG] 3 Item directory cleared [DEBUG] 3 Requesting source htdocs/fotolia/essen-trinken/fittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg Mon, 15 Aug 2016 06:47:21 GMT [DEBUG] 3 Requesting analysis [DEBUG] 3 Analysis successful [DEBUG] 3 Writing metadata file [DEBUG] 3 Metadata ready [DEBUG] 3 Source file ready [DEBUG] 3 Target file doesn't exist. Must be regenerated. [DEBUG] 3 Applying cropping and constriants [DEBUG] 3 Requesting image conversion [DEBUG] 3 Image conversion successful [DEBUG] 3 Released lock: metadata [DEBUG] 3 Released lock: source file [DEBUG] 3 Releasing lock: target file "GET /?ts=1471243641&height=270&width=540&file=img%3A%2Ffotolia%2Fessen-trinken%2Ffittosize__540_270_cfef4bbeec6b9ba5317096e074a38794_fotolia_42317032_gerhard_seybert.jpg HTTP 1.0" 200 61614 ` Can you test with making your test-image old? > touch -d"2001-01-01" test/resources/images/test_image_sparrow_smaller.jpg |
|
The test image is also quite old. It has an mtime from 2015 so it has definitely exceeded a max-age of 3600. Just to make sure, I also tried setting its time to 2001-01-01 but that had the same result. Did you update to the latest version 2.7.4 before testing with the garbage collector setting I suggested? With version 2.7.3 the problem will always happen. |
If you use not urls for your images, but local file-system,
the images will be copied on every request again and will be always generated again (minimum every minute)
Wrong usage in converjon/lib/source.js Line 205
It should use the mtime from the file-system-available image. (fs.stat()).
Further it can be improved, if it can use symbolic links for /source (if symbolic links available)
The text was updated successfully, but these errors were encountered: