-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DNS Resolution failed - socket busy #12568
Comments
Could it be related to this issue (currently open): #12290 |
We found the change that We updated our
|
Possibly related to #3223? |
Any input on this? |
@chobits , could you give us some comments about this? |
will look into it later |
This might be sock:setpeername() is trying to resolve the name, during this process, if someone tries to execute
So it is as expected when you enable sync, which means a large number network queries for the same domain at the same time in one worker have been merged into one coroutine. It is like fewer DNS queries trigger fewer DNS errors. If you could provide debug log in the context of error message, it would be beneficial to debug for us. |
Hi @chobits, thanks for chipping in Here are the logs. I included some startup logs as well so you can see the configuration. It happened right off the bat as well The problem is quite severe for us (at least with the DNS sync enabled) and happens even if there are very few request being made. The example above are a few system tests that run through. I guess that there are about 50 calls within 2 minutes. I don't imagine it is a heavy load problem, rather a configuration problem on our end or something else Some more information that could possibly be relevant:
|
Sry for late reply, I have reviewed your kong-logs.csv, but I dont find some useful logs to help us to investigate. Because most useful debug logs have been commentted in DNS client library. And I guess there might be some incorrect DNS resoluton operate executed by BTW, Is |
hi I have thought of a method to mitigate your issue, you could configure |
Thanks for the reply. Yes It seems like the staleTtl is already at
We don't override any DNS related options in the config file. Here our current
Thanks @chobits for your support 🙏 |
Could you try this docker image: In this PR, we refactor our DNS client, if you could try this PR's docker image, it could help us to verify whether this issue still be there in new DNS client. If both versions of kong have this problem, I guess DNS client socket might be messed up by some other subsystem in kong. |
Thanks for the update @chobits. We get the following error with
Additionally in the beginning we get the following (still the same)
|
Did it constantly occur if you request this Did your plugin call this kind of methods: From kong's logic, it seems that the
So it might be the dns socket messed up, but currently your image has no nginx debug log enabled like |
No only sporadically. It happens even less with the
No we don't call any of these methods. We have a static |
I just tried again with the latest |
False alarm. Still happens, just less often |
Because of a hunch I replaced the |
Hi @henrythethird could you please help what exact change did you make, how did you replace the dependency ? |
The |
@mayank-allen Removed local request = http.new_from_uri(uri)
request.follow_redirects = false
request.headers:upsert(header_key, key)
request.headers:upsert(":method", "POST")
local headers, stream = request:go(10) Instead added local http_client = http.new()
local response, err = http_client:request_uri(url, {
method = "POST",
path = path,
headers = {
[header_key] = key,
}
}) This change is also reflected in the
@Tieske it's maybe worth adding that to the documentation then (if it's not already in there). We struggled with this quite a bit. |
Is there an existing issue for this?
Kong version (
$ kong version
)Kong 3.5 and 3.6
Current Behavior
Requests fail randomly with the following log:
The client gets the following message back:
Expected Behavior
Requests go through as per Kong 3.4.2
Steps To Reproduce
I don't know, since it happens only sporadicallyEdit: It happens quite frequently when the configuration option dns_no_sync is turned off (so dns sync is on -> default behavior as of Kong 3.5.0) and quite sporadically with it turned on.
We have the following setup:
Anything else?
It works perfecly with Kong 3.4.2. We tried upgrading back in November to 3.5, which gave us the same flaky behavior. Found some very old issues, which seem to be resolved.
The text was updated successfully, but these errors were encountered: