Cache invalidation for scripts in symlinked folders #126
Comments
|
Do you pass the real path names (all symlinks resolved) to opcache_invalidate()? Otherwise if you change the symlink, file names might be resolved to different real paths. |
|
I mean automatic invalidation, not via opcache_invalidate. |
|
In case you just change symlink, the scripts laying in old directory are On Mon, Aug 26, 2013 at 11:22 AM, Kalashnikov Igor <[email protected]
|
|
And it will be great if Zend OPCache will handle it. |
|
The opcode cache uses the realpath of the files, so if a different symlink points to the same file you will get the same set of opcodes. You also have it configured to only check every 2 seconds, so for 2 seconds after pointing your "production" symlink at another target you are going to get the old target files. You can read about how to properly manage a symlinked docroot with opcache here: http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ |
|
Too complicated, to be the best solution :) And realpath makes symlinks resolved. |
|
Well, then just turn off opcache.revalidate_freq so it will revalidate on every request. Your deploys won't be atomic, but it should never load the wrong file. |
|
+1 on this, we are using Capifony.org / Capistrano to deploy our project, and it uses symlink... |
|
+1 on what? There is no bug here. What is most likely happening is that the failed requests are the ones that get screwed over when the symlink switch happens while they are executing, or at least within the revalidate_freq window. You can shrink this window by setting opcache.revalidate_freq to 0. It doesn't entirely eliminate the problem, but it comes very close, at least if your site isn't very busy. To completely eliminate the problem, read on: opcache has no concept of the start of a request. It works on individual opcode arrays. An opcode array is what is generated and cached for each included file and the key for each included file is the fully qualified path for the script that was compiled. How you got there, via a symlink, various relative path specifiers (think ../path/file.php or ../../other/path/file.php) is irrelevant, the fully qualified path (or the realpath) to that file is the same and the access mechanism is not maintained. So, when you deploy via something like Capistrano which does a symlink swap on the document root, you want all new requests to get the new files, but you don't want to screw over requests that are currently executing as the deploy is happening. What you really need to create a robust deploy environment is to have your web server be in charge of this. The web server is the piece of the stack that understands when a new request is starting. The opcode cache is too deep in the stack to know or care about that. With nginx this is quite simple. Just add this to your config: This tells nginx to realpath resolve the docroot symlink meaning that as far as your PHP application knows, the target of the symlink if the real document_root. Now, once a request starts, nginx will resolve the symlink as it stands at that point and for the duration of the request it will use the same docroot directory, even if the symlink switch happening mid-request. This entirely eliminates the symptoms described here and it is the correct approach. This isn't something that can be solved at the opcache level. Apache doesn't have this same mechanism to resolve a docroot symlink at the start of the request, but I have written an Apache module that does it. See https://github.com/etsy/mod_realdoc |
|
The FYI the symlink issue doesn't only happen in the 2 seconds Thanks a lot, this should go into FAQ in my opinion. I would submit a pull request if my English is good enough. |
|
Yes, only service restart or invalidate function call helps. $realpath_root isn't a great solution cause it is undocumented. |
|
Actually, invalidating the cache doesn't solve anything. Requests that are already running which started on version A of the code, if you suddenly deploy and invalidate the cache before that request has finished, may very well do another include and at that point it will be including files from version B and you are back to an unknown state for that request. A web server restart, assuming it is a graceful restart that lets existing requests finish, can be made to work, but the timing is a bit tricky. You have to combine the graceful restart with a config change so the new requests will see the new docroot and the requests that are finishing up continue on the previous. Or you need to get into load balancer tricks where you stop sending new requests to a subset of your servers. Then you wait a while to let existing requests finish, then deploy to that subset and repool them. You may not have enough servers to do this without seriously affecting site performance though, and it also slows down deploys significantly. So I still think having the web server realpath the docroot symlink and setting the effective docroot to the target of that symlink is the slickest and most complete solution to this problem. |
|
+1 on Rasmus's points here. The issue that you face is that the PHPs Zend VM maps in new sources at runtime as it executes the INCLUDE_OR_EVAL opcodes which reference the source. If you want to avoid scripts barfing in indeterminate ways during version cutover, you must maintain path integrity across all of these includes (and the requested script). If you defer symbolic resolution of paths to the runtime system then you will occasionally hit the asynchronous edge effects, unless you take the steps that Rasmus describes. Of course, this is also easy to implement at an application level -- but only if you can set coding standards or modify code, e.g. using tricks like using complete install hierarchies and setting: define('ROOT_DIR', dirname(__FILE__)); (or using a
@rasmus, would you agree with this analysis or have I missed something? |
|
Yes, you can do it entirely in userspace if have a front controller that is always run first and you are very strict about always using includes relative to that initial path. It is almost exactly the same thing as my approach. I simply do it at the web server level instead of in PHP which means you have a bit more flexibilty at the PHP level and don't have to be quite as vigilant about how you write your code. You still can't refer to files via the symlink, of course. It has to be relative to DOCUMENT_ROOT at all times. |
|
We use |
|
I got lof of strange behavior with symlink, for instances here a piece of opcache_get_status() result : I don't know if it's related to symlink or not, basically I use Nginx with A symlink that goes to the last version (here /var/www/production/20140708212446). After create the last version folder and changed the symlink destination, I call opcache_reset() (by HTTP curl). But my index.php stays always at the previous version, with a timestamp of 0. My web site is very high traffic (about 100 request/secondes). |
|
I experience the same as @ebuildy . The |
|
Reloading nginx using $real_path doesn't solve the issue for me: I still have to reload php5-fpm as well otherwise php5-fpm is still pointing to the previous $real_path. |
|
You have to reload php5-fpm. Reloading Nginx will not help. On Mon, Sep 22, 2014 at 9:40 PM, cirpo [email protected] wrote:
|
|
Reloading fpm I still got some requestes failing... What if I create a new nginx conf during the deployment with the real path and then do an nginx reload? |
|
The $realpath_root solution in nginx doesn't work if you run fpm on different nodes than your nginx nodes, because nginx nodes doesn't have access to the fpm nodes directory tree. In this case, the solutions seem to be:
|
|
Ok, I think I understand the "problem" now. It's a bit silly in retrospect, but I'm documenting it just in case someone else faces it in the future. Pointing nginx to /var/www/app and then redefining SCRIPT_FILENAME/DOCUMENT_ROOT using $realpath_root in nginx will work as intended. But if your symlink is this: And your root on nginx is @marcmillien I see, that makes sense. I suppose you could put nginx instances or some other sort of middle man in front of the nodes running php-fpm to solve that, and then forward the requests via @dmaicher thanks for testing that, just wanted to let you know it helped us found the issue. |
|
@jportoles this is one of the solutions yes :). |
|
But honestly, how is this not a bug? Look at what @jportoles described. Nginx doesn't help there. If you have something like: So the way it works seems to be the following:
WAIT A SECOND! How did it connect So did it actually cache the pretty symlink path as well? It seems so to me. Step 6 should be the following: I have not dived into the source code but the culprit is probably OPcache using internal PHP realpath cache for symlinks. So in step 6 it doesn't see that symlink is pointing to the new file now since it already cached the realpath for that location in step 2. Does this sound right? |
Not OPcache's fault in this case, PHP has a built in realpath cache with a default TTL of 2 minutes, see here: http://php.net/manual/en/ini.core.php#ini.realpath-cache-size You can disable it (set the TTL to 0) but even if the entry point is correct, you will still have issues with includes being desynchronized while in the middle of a symlink change. This is why it's better to let the server handle it beforehand and use $realpath_root on nginx where possible. |
|
Just tested this and it seems that it is not connected to PHP's realpath cache. PHP's realpath gets updated after 2 minutes but OPcache still sees the old path. |
|
I'm not too intimate with PHP internals so someone correct me if I'm wrong, but as far as I understand, when a request hits the interpreter, the opcode cache resolves the path first (which may or may not be cached in the realpath cache), and then proceeds to cache the resolved path. With the realpath cache off, the opcode cache should be hitting system calls to resolve the path before fetching a cached entry every time. So with the realpath cache off, the opcode cache shouldn't be the culprit for whatever is failing. What could be happening is that your application crashes in the middle of a symlink change because you are referencing the unresolved symlink (e.g. |
|
|
@ifeltsweet the real way to address this is at the web server level as I explained above. There isn't anything we can do at the PHP level. Opcache is working as expected when it comes to resolving symlinks. |
|
@rlerdorf shouldn't disabling the realpath cache also work though? I was under the impression that you could solve this either at a web server level (e.g. |
|
@rlerdorf sure and I agree with you, but there is still something wrong with the way OPcache resolves symlinks, it seems to just cache resolved realpaths forever. What if Nginx also cached once resolved "$realpath_root" forever? You wouldn't be able to use "$realpath_root" then. |
|
But it doesn't cache resolved paths forever at all. If it did, then the deploy strategy I described wouldn't work and it has worked on a very large site with 40+ deploys per day with a ton of traffic for a couple of years now. I think you need to go back and look at your assumptions and perhaps create some test scenarios to figure out what you are doing wrong. |
|
The strategy that you are using at etsy relies on mod_realdoc to resolve your realpath. It also caches those paths for 2 seconds (only 2 seconds). The strategy that I am describing relies on OPcache to resolve the realpath. The difference is that OPcache doesn't seem to see that symlink is now pointing to the new location. Not after 2 seconds, 2 minutes or even an hour. I will create a test and hopefully we can all take a look at it together. |
|
There is blacklist where you can specify a list of file you don't want to cache. This list works based on my test even on symlink path. Then if you set in this list something like /my/current/workspace even "current" is a symlink the file the symlink point to will never be included in the cache. So for those who wants to control from userspace thanks to a front controller they may eventually achieve this if they are ok to afford the cost of one never cached PHP file... See: opcache.blacklist_filename |
|
TLDR I had issues with OPcache using Capistrano for years. I tried a lot of suggested fixes and workarounds In the end this article helped: The final fix was adding this to opcache.use_cwd = 1
opcache.revalidate_path = 1This finally solved all my problems. Maybe this helps someone else too. |
|
Also, about the way the cache is storing the data and access them. One of your assumption is that it always try to perform a realpath on cache entry tentative and that it should then spot the change with a same symlink to invalidate the cache and cache the new one. Turn out it is not true, it depends on the scenario and is rather complex. The cache key is build in this way (in order specifically to avoid to have to perform a realpath anytime): Then in the case of you have let's say a script in a symlinked folder included by another script in a non symlinked folder the key will looks like: So technically, changing the symlink in that case is not going to change the key name and eventually could end in the original old script remains in the cache if non of the other invalidation mechanism is triggered and the realpath things has little or nothing to do with that use case... Now another interesting use case, if the front controller is already in a symlinked folder, the key generated to identify the file in the cache for this very first file is just the path of the file still including the symlink. Ex: /my/root/folder/symlinktoV1/frontController.php => key=/my/root/folder/symlinktoV1/frontController.php not /my/root/folder/V1/frontController.php So without a cache reset it looks unlikely that the cache will as quickly as possible get updated. Also don't get fool by the opcode status function. It will display things like: So you might trust the output to be like the array key in scripts match the cache key to identify a file where it's always a realpath returned but it's actually not always really the key used internally. Well to be fair, there are actually several different ways to access the cache value and depending on how it has been cached in the first place but it will be retrieve later on in this order for a FastCGI request:
So if 1 or 2 is match before 3 then 3 is never called. Which means it all depends on how it has been cache in the first place. FastCGI and require_once and include_once use similar algorithm to cache or access a cached file. So I tried to reproduce the FastCGI situation but using the cli by creating a test script in that way (in cli opcache is not persistent, it's just keep compiled script in opcache until end of the script execution): opcache_compile_file(using_non_symlink_path); And the include_once then hit the cache. Now a little more closer to what happen in the first hit when the file is not yet cached with an include_once on a symlinked path. First the include_once is trigger and actually try to resolve the path (default PHP behavior) and then the opcache take its chance by hooking the process there to identify the already eventually compiled script saved in the cache and a key as well as resolving the path. As the script is not yet cached this step is simply memorizing that it will be needed to cache this script and returning the resolved path to PHP engine. Then surprisingly persistent_zend_resolve_path got hit again still by the include_once but this time in the context of the php_stream_open_for_zend_ex, interestingly here we can note that the given filename is already the resolved path, not anymore the symlink. But this does not matter that much. Finally we hit the persistent_compile_file in the context of the include_once. However the symlink information is gone already and we are just left with the real path that will be really use as the key to store the value. So, sorry for the very long comment, in the case of an include_once, require_once or any FastCGI request as the trigger, the resolution of the symlink will be left actually to the PHP original engine function (phar_resolve_path=>phar_find_in_include_path=>phar_save_resolve_path=>php_resolve_path 🍡 ), it will happen systematically due to include_once/require_once behavior and the opcode will always get a resolved path, never a symlinked path if the script has not been cached previously using a different way (like opcache_compile_file). So if php_resolve_path return anytime the last resolved path then the opcache will create a new entry as soon as the link points to a new location as it will be a new key (which as few to do with the invalidation of the old keys that could persist depending on the cache ttl, which means that several version of the same file may coexist in the cache at the same time). However if for some reason the resolved path return by the PHP engine is still the old one then you will get the old script.
Hope it may help to understand a little more the inside of the opcache :) I did reproduce apache prefork (most commonly used configuration for apache+php when php loaded as a module) and attached a gdb on it and I confirmed that for the very first file (I mean actually any files open up by Apache from the document root as the first entry point of your request out of the include_once, require_once mechanism etc...) is indeed stored in the cache by full path which means NOT the real path. So it will let you with the problem of the real path cache of php to deal with the entry point. Then it seems if you really don't want to have to restart Apache but continue to deploy based on symlink that you'll have to disable the realpath_cache and accept performance consequences. By the way, if you want to get the "same" results than with @rlerdorf Apache extension but you rather prefer patch on PHP side (if you can compile it) to avoid the VirtualHost issue then here you go: File: ./sapi/apache2handler/sapi_apache2.c The native realpath function is not going through the php realpath cache system. |
|
Is this issue by PHP7 still present? |
|
@vingrad this is not an issue. This behavior still persist in PHP7 as well |
|
This IS a bug! Opcache remembers symlink-to-realpath mapping internally forever, like already stated above, so this should be fixed! And still persists in php7. |
|
@sulate See problem explain here: http://jpauli.github.io/2014/06/30/realpath-cache.html I don't think they are ever gonna fix this. |
|
yes and it's referenced on Symfony NGINX documentation : https://symfony.com/doc/current/setup/web_server_configuration.html (symfony/symfony-docs#5758 @kendrick-k) |
ghost
mentioned this issue
+1 how can anyone deny this?!? Opcache resolves symlinks and does not revalidate the result ever again. Clearly a BUG. If the thought ( dead link: http://jpauli.github.io/2014/06/30/realpath-cache.html ) is, that during one request the path to a library should not change, then the opcache should at least revalidate its symlinks after a fpm request finished. That would actually be a really nice feature if a symlink could not change during a request, but changes are recognized once a new request starts |
|
here is the working link http://blog.jpauli.tech/2014-06-30-realpath-cache-html/ it says (wrongly) "that opcode cache solutions (at least OPCache and APC) rely on the internal realpath cache from PHP. as mentioned above, with at least PHP7.0/7.2 the opcache seems not to be connected to the realpath cache. The opcache has its own realpath cache and it never expires. |
|
I didn't find an explicit answer in documentation or StackOverflow, and prior digging deeper and adding more logs, may I clarify about directive Considering
Example$ pwd -P;
/var/www/html/example.com
$ ls -la | tail -n +4;
lrwxrwxrwx 1 www-data www-data 25 Jul 9 19:38 active -> /var/www/html/example.com/builds/1234
drwxrwxr-x 1 www-data www-data 2 Jul 9 19:38 buildsserver {
# ...
root '/var/www/html/example.com/active';
location /_extra/(.*) {
expires max;
# Would this catch-up with a symlink `active` update (i.e. absence of $realpath_root)?
try_files "/extra/static/$1" =404;
}
location / {
try_files "/static/$uri" @app;
}
# ...
}Thank you in advance! |
Hi!
We are trying to use your opcode cache with our project.
Our code versions are managed in production with symlinks. Sometimes it seems that opcode cacher timestamp invalidation fails after project symlink is changed and we need to clear the cache manually.
Can you help us with any advice? Thank you.
Versions:
Config:
The text was updated successfully, but these errors were encountered: