Error: 14 UNAVAILABLE: No connection established

[AkhilA4]

Hello everyone, I've been trying to create a secret in the GCP from running the GCP APIs from AWS lambda. I tried setting the environment variable at the run time like process.env.GOOGLE_APPLICATION_CREDENTIALS = '/tmp/serviceAccountCreds.json'; . The secret manager API was running fine till DEC 29th on my environment. But from then on the API is throwing below error

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/home/akhil/onboard/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus /home/akhil/onboard/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus /home/akhil/onboard/node_modules/@grpc/grpc-js/src/client-interceptors.ts:434:34)
    at Object.onReceiveStatus (/home/akhil/onboard/node_modules/@grpc/grpc-js/src/client-interceptors.ts:397:48)
    at /home/akhil/onboard/node_modules/@grpc/grpc-js/src/call-stream.ts:237:24
    at processTicksAndRejections (internal/process/task_queues.js:79:11) {
  code: 14,
  details: 'No connection established',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }

The secrets manager's version is 3.2.3

"dependencies": {
  "@google-cloud/secret-manager": "^3.2.3",

@grpc/grpc-js version installed is 1.1.8

{
"_from": "@grpc/grpc-js@~1.1.1",
"_id": "@grpc/[email protected]",
"_inBundle": false,
"_integrity": "sha1-KEXw/D0b+7FQ7Xp4p2vfQbEm02c=",
"_location": "/@grpc/grpc-js",
"_phantomChildren": {
 "@types/long": "4.0.1",
 "lodash.camelcase": "4.3.0",
 "long": "4.0.0",
 "protobufjs": "6.10.2",
 "yargs": "15.4.1"
},

Looks like this is an old issue which has resurfaced again: grpc/grpc-node#1064

[sofisl]

@AkhilA4, I am unable to reproduce the issue so I'll defer to see if others have heard similar things. @alexander-fenster or @bcoe, do we have any other reports about this issue or do you know if this might be a known issue?

[AkhilA4]

@sofisl , when I run the GCP API for creating secrets on windows, the API is working just fine. But, when I run the same API in the AWS environment (AWS lambda) I'm encountering this No connection Establishment issue. As I mentioned previously, in the AWS environment as well the API worked well for few days but started encountering this issue recently. Do we need to take care of any house keeping with regards to connections made with the previous requests to the GCP secrets manager or anything like that ? (DNS ? load balancing issues ? )

Can you guys please suggest any other work-around or versions so that it can start working again.

[bcoe]

@AkhilA4 could you please run your application on AWS with the following environment variables:

GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG

This will provide information about why the connection is having trouble establishing.

[AkhilA4]

Hello @bcoe , thanks for the reply. Please find the logs below

2021-01-08T06:23:57.269Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 IDLE -> IDLE
2021-01-08T06:23:57.282Z | connectivity_state | dns:secretmanager.googleapis.com:443 IDLE -> IDLE
2021-01-08T06:23:57.286Z | dns_resolver | Resolver constructed for target dns:secretmanager.googleapis.com:443
2021-01-08T06:23:57.290Z | channel | dns:secretmanager.googleapis.com:443 createCall [32] method="/google.cloud.secretmanager.v1.SecretManagerService/CreateSecret", deadline=Fri Jan 08 2021 11:54:57 GMT+0530 (India Standard Time)
2021-01-08T06:23:57.302Z | call_stream | [32] Sending metadata
2021-01-08T06:23:57.313Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.378Z | call_stream | [32] write() called with message of length 54
2021-01-08T06:23:57.392Z | call_stream | [32] end() called
2021-01-08T06:23:57.406Z | call_stream | [32] deferring writing data chunk of length 59
2021-01-08T06:23:57.409Z | dns_resolver | Resolution update requested for target dns:secretmanager.googleapis.com:443
2021-01-08T06:23:57.445Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 IDLE -> CONNECTING
2021-01-08T06:23:57.459Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.463Z | connectivity_state | dns:secretmanager.googleapis.com:443 IDLE -> CONNECTING
2021-01-08T06:23:57.474Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.477Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.491Z | connectivity_state | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.508Z | dns_resolver | Resolved addresses for target dns:secretmanager.googleapis.com:443: [2404:6800:4007:801::200a:443,142.250.67.74:443]
2021-01-08T06:23:57.519Z | pick_first | Connect to address list 2404:6800:4007:801::200a:443,142.250.67.74:443
2021-01-08T06:23:57.524Z | subchannel_refcount | 2404:6800:4007:801::200a:443 refcount 0 -> 1
2021-01-08T06:23:57.530Z | subchannel_refcount | 142.250.67.74:443 refcount 0 -> 1
2021-01-08T06:23:57.538Z | subchannel_refcount | 2404:6800:4007:801::200a:443 refcount 1 -> 2
2021-01-08T06:23:57.539Z | subchannel_refcount | 142.250.67.74:443 refcount 1 -> 2
2021-01-08T06:23:57.540Z | pick_first | Start connecting to subchannel with address 2404:6800:4007:801::200a:443
2021-01-08T06:23:57.547Z | pick_first | IDLE -> CONNECTING
2021-01-08T06:23:57.549Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.564Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.565Z | connectivity_state | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.568Z | subchannel | 2404:6800:4007:801::200a:443 IDLE -> CONNECTING
2021-01-08T06:23:57.574Z | pick_first | CONNECTING -> CONNECTING
2021-01-08T06:23:57.577Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.592Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.598Z | connectivity_state | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.600Z | subchannel | 142.250.67.74:443 IDLE -> CONNECTING
2021-01-08T06:23:57.613Z | pick_first | CONNECTING -> CONNECTING
2021-01-08T06:23:57.621Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.635Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.639Z | connectivity_state | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.652Z | subchannel | 2404:6800:4007:801::200a:443 connection closed with error connect ENETUNREACH 2404:6800:4007:801::200a:443 - Local (:::0)
2021-01-08T06:23:57.661Z | subchannel | 2404:6800:4007:801::200a:443 CONNECTING -> TRANSIENT_FAILURE
2021-01-08T06:23:57.672Z | pick_first | CONNECTING -> CONNECTING
2021-01-08T06:23:57.675Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.677Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-08T06:23:57.694Z | connectivity_state | dns:secretmanager.googleapis.com:443 CONNECTING -> CONNECTING
2021-01-08T06:23:57.729Z | subchannel | 142.250.67.74:443 connection closed with error self signed certificate in certificate chain
2021-01-08T06:23:57.740Z | subchannel | 142.250.67.74:443 CONNECTING -> TRANSIENT_FAILURE
2021-01-08T06:23:57.745Z | pick_first | CONNECTING -> TRANSIENT_FAILURE
2021-01-08T06:23:57.748Z | resolving_load_balancer | dns:secretmanager.googleapis.com:443 CONNECTING -> TRANSIENT_FAILURE
2021-01-08T06:23:57.750Z | channel | Pick result: TRANSIENT_FAILURE subchannel: undefined status: 14 No connection established
2021-01-08T06:23:57.761Z | call_stream | [32] cancelWithStatus code: 14 details: "No connection established"
2021-01-08T06:23:57.762Z | call_stream | [32] ended with status: code=14 details="No connection established"
2021-01-08T06:23:57.765Z | connectivity_state | dns:secretmanager.googleapis.com:443 CONNECTING -> TRANSIENT_FAILURE

The error lines that seem to be important for me are below

2021-01-08T06:23:57.652Z | subchannel | 2404:6800:4007:801::200a:443 connection closed with error connect ENETUNREACH
2021-01-08T06:23:57.729Z | subchannel | 142.250.67.74:443 connection closed with error self signed certificate in certificate chain

I faced similar SSL termination issue while I tried installing GCP CLI on my Ubuntu machine and I'm running my AWS lambda on Ubuntu machine (local debug). My Ubuntu machine seems to have SSL certificates issue. But, after seeing the above error messages, I tried running the lambda from the AWS API gateway and the lambda ran fine and the secret got created in the GCP successfully. Tried on multiple runs and the secrets are getting created every time I hit the GCP API.

Wanted to your thoughts as well. Is it because of this SSL termination issue, the connection was never established or is it something it started working again without any fix ? If so, just wondering how did it work at the starting of the local lambda runs.

[bcoe]

@AkhilA4 thank you adding to our weekly meeting 👍 will keep this updated.

[sofisl]

For Googlers, internal tracking number b/183523715

[shawn98ag]

@bcoe
Any updates on this issue?
I think I just ran into this same problem today.
I get the same error message, "connection closed with error self signed certificate in certificate chain".

[bcoe]

@shawn98ag I believe your problem is most likely this one described in Node.js:

nodejs/node#37025 (comment)

@alexander-fenster do you know how we would set the CN parameter in a GAPIC client? Perhaps we could add this to our advanced documentation in GAX?