"Failed to connect before the deadline" errors being seen #818
Comments
bcoe
added
type: question
|
as discussed here this is a known, non-fatal, warning coming out of |
|
@bcoe including the "Failed to connect before the deadline" error? It happened again today with the debug flags on: |
npomfret
changed the title
|
@callmehiphop can you speak to the above ☝️ , what situations can lead to this (I believe you were helping a user with a similar issue last week). |
|
@bcoe this is not the same issue, this looks like the same issue we were having with gRPC previously. That specific error occurs when a timeout occurs while waiting for the gRPC channel to get put into a ready state. |
|
@callmehiphop @alexander-fenster I was wondering if there's a parameter that can be tuned to increase the connect timeout. @npomfret would you be able to provide the detailed debug output when this connection error occurs? |
|
@bcoe there is, but the default value is 5 minutes, so I'm not sure I think that increasing the timeout here is the right solution. |
I have (above). I tuned on
All other logging at that time came from my app. |
|
@npomfret it sounds like your logs have given @murgatroid99 a hunch, as to what's happening. If you're open to it, we can try floating a patch to you early next week? CC: @alexander-fenster who I've been working on a process with to facilitate releasing candidate versions to folks. |
|
@bcoe happy to help |
|
I've seen 4 of these in the past 24h:
I didn't see any logging of interest before of after this time (the TRANSIENT_FAILURE I reported above is not present). But, each of them occurred shortly after (like a second or two) this log entry:
|
|
Every time we see this error, the subscription stops working and no new messages are consumed until we restart the POD. |
|
@npomfret we've released If you have a chance, try it out? and we'll release more widely after the american holidays? |
|
I'll do a release today using @google-cloud/pubsub@next and run it for a few days |
|
I've had two of these in the past 24h with the new code:
My app is still running and receiving messages. I accidentally turned off |
|
[update] I've had zero errors since the 24th November. |
|
👍 @npomfret that's great to hear, I'll close as soon as we've ensure that all of our libraries are on |
|
Sadly I spoke too soon. I upgraded from So no errors from 24th November to the 6th December (on There seem to be 2 types of error: and The 1st error 'Failed to connect before the deadline' (the more common one), was preceded by the following in stackdriver: The 2nd error 'Deadline exceeded' was preceded by the following in stackdriver: ... all within about 1 second before the error. |
|
@murgatroid99 ☝️ does anything jump out at you in these logs, is there anything additional that @npomfret can provide to help you debug. |
npomfret
changed the title
google-cloud-label-sync
bot
added
the
api: pubsub
|
I'm still getting these with the latest release, although it has to be said they do appear to be reducing in frequency (2 in the past 4 days, before it was more like 4 per day on my system). Using... |
|
I may be speaking too soon, but it looks like the latest release may have fixed this issue. Recent releases seem to have steadily reduced the number of errors, but until recently I was getting one or two errors per day. I've had zero for five days now. The upgrade in question took my deps from: to .. |
|
Closing this as no errors have occurred in my system for 9 days now. |
|
Just wanted to jump in and say that I was having this issue previously as well. The issue arose from the GRPC Subscriber Client not being the same as ProjectID the Subscription Path. Once these two were set as the same, then the issue cleared up. In case anyone Googling this issue finds it, hopefully this can help them. |
This PR was generated using Autosynth. 🌈 Synth log will be available here: https://source.cloud.google.com/results/invocations/9f0ce149-63bd-4aad-917d-44f82c8912a6/targets - [ ] To automatically regenerate this PR, check this box. Source-Link: googleapis/synthtool@ba9918c
Since upgrading to the latest pubsub (
"@google-cloud/pubsub": "1.1.5") my node is still getting occasional errors from the underlying grpc library. My app is using [email protected]:$ npm ls | grep grpc-js
│ │ ├─┬ @grpc/[email protected]
It's not as bad as it was. When I first upgraded to pubsub 1.x it simply didn't work at all, I'd get an error within a few seconds of the app starting and from then no messages would get delivered.
Now the problem is much less bad (my app is working), but I still see the following errors in my logs:
From
subscription.on(error, (err) => {I see this:And in my logs I've also noticed these (excuse the formatting):
Error: Failed to add metadata entry @���: Wed, 13 Nov 2019 07:13:16 GMT. Metadata key "@���" contains illegal characters at validate (/home/nodeapp/apps/crypto-trader-prototype/shared-code/node_modules/@grpc/grpc-js/build/src/metadata.js:35:15) at Metadata.add (/home/nodeapp/apps/crypto-trader-prototype/shared-code/node_modules/@grpc/grpc-js/build/src/metadata.js:87:9) at values.split.forEach.v (/home/nodeapp/apps/crypto-trader-prototype/shared-code/node_modules/@grpc/grpc-js/build/src/metadata.js:226:63) at Array.forEach (<anonymous>) at Object.keys.forEach.key (/home/nodeapp/apps/crypto-trader-prototype/shared-code/node_modules/@grpc/grpc-js/build/src/metadata.js:226:43) at Array.forEach (<anonymous>) at Function.fromHttp2Headers (/home/nodeapp/apps/crypto-trader-prototype/shared-code/node_modules/@grpc/grpc-js/build/src/metadata.js:200:30) at ClientHttp2Stream.stream.on (/home/nodeapp/apps/crypto-trader-prototype/shared-code/node_modules/@grpc/grpc-js/build/src/call-stream.js:211:56) at ClientHttp2Stream.emit (events.js:193:13) at emit (internal/http2/core.js:244:8) at processTicksAndRejections (internal/process/task_queues.js:81:17).. but the two don't seem to be related.
I will start running with the debug flags enabled and report back
The text was updated successfully, but these errors were encountered: