-
Notifications
You must be signed in to change notification settings - Fork 376
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Panic on vcl.load: Wrong turn at cache/cache_backend_probe.c:710: running #4199
Comments
This is an excellent issue report, thank you! You even made an effort to patch vmod_dynamic with a cut&paste (edit) The actual reproducer is a delay added for |
So what I think is happening here: vmod_dynamic creates domain directors (basically resolver threads) also during If the VCL temperature is still varnish-cache/bin/varnishd/cache/cache_vcl.c Lines 633 to 637 in e142af7
But by adding the delay, we prolong the This touches on #4142 , but from the other end: Here we would need a director list from "before the warm event" and only these would need to get the event sent... |
Dynamic backends may get created while we transition a VCL to the WARM temperature: Because we first set the new temperature and then post the event, backends might get created when the temperature is already WARM, but before the VCL_EVENT_WARM gets posted, which leads to double warming. This can be demonstrated with an appropriate delay during vcl_send_event(). The solution to this problem is simple: Only post WARM events to backends from before the start of the transition. In code, it looks a little more involved, but it is not complicated: Under the vcl mutex, we take all directors onto a separate "cold" list and change the temperature, and send the warm event only to the "cold" directors. When all are warm, we concatenate. To undo, we basically do the inverse: Save the warm directors, put the cold ones back, send a cold event to all warm ones and concatenate again. Fixes varnishcache#4199
Dynamic backends may get created while we transition a VCL to the WARM temperature: Because we first set the new temperature and then post the event, backends might get created when the temperature is already WARM, but before the VCL_EVENT_WARM gets posted, which leads to double warming. This can be demonstrated with an appropriate delay during vcl_send_event(). The solution to this problem is simple: Only post WARM events to backends from before the start of the transition. In code, it looks a little more involved, but it is not complicated: Under the vcl mutex, we take all directors onto a separate "cold" list and change the temperature, and send the warm event only to the "cold" directors. When all are warm, we concatenate. To undo, we basically do the inverse: Save the warm directors, put the cold ones back, send a cold event to all warm ones and concatenate again. Fixes varnishcache#4199
Thanks for the patch! #4205 fixes the issue on my end, the test case passes. 👍 |
Dynamic backends may get created while we transition a VCL to the WARM temperature: Because we first set the new temperature and then post the event, backends might get created when the temperature is already WARM, but before the VCL_EVENT_WARM gets posted, which leads to double warming. This can be demonstrated with an appropriate delay during vcl_send_event(). The solution to this problem is simple: Only post WARM events to backends from before the start of the transition. In code, it looks a little more involved, but it is not complicated: Under the vcl mutex, we take all directors onto a separate "cold" list and change the temperature, and send the warm event only to the "cold" directors. When all are warm, we concatenate. To undo, we basically do the inverse: Save the warm directors, put the cold ones back, send a cold event to all warm ones and concatenate again. Fixes varnishcache#4199
Expected Behavior
No panic on the following VTC.
Important reproduction notes
This is against Varnish master + libvmod-dynamic master. I can only reproduce this when intentionally "slowing down" vmod_event with a sleep.
In my case in production, I originally had the panic when using libvmod-redis, and without adding its
import
in my VCL, the panic would not be reproduced. I suspected some kind of race condition that only happens when waiting in vmod_event somehow (because that would be the only side effect of loading libvmod-redis).So, in order to reproduce the panic in the VCL below, add at the start of
libvmod-dynamic/src/vmod_dynamic.c
vmod_event
:(Or run the following from
libvmod-dynamic
:sed -z -i -E 's|(vmod_event\(VRT_CTX, struct vmod_priv \*priv, enum vcl_event_e e\)\n\{\n)|\1\#include \<unistd.h\>\nusleep\(100000\)\;\n|g' src/vmod_dynamic.c
)This will simulate a slow down in vmod_event by sleeping for 100ms.
Current Behavior
The panic is quite random, so might require several tries:
The panic is:
But sometimes the VCL state is
running
instead ofscheduled
.Possible Solution
No response
Steps to Reproduce (for bugs)
No response
Context
This was particularly hard to track down to a MWE VTC 😛
Varnish Cache version
varnishd master; libvmod-dynamic master + usleep patch from ticket
Operating system
No response
Source of binary packages used (if any)
No response
The text was updated successfully, but these errors were encountered: