Emulator Functions Regression - Initial Function Requests Time Out

[dereekb]

[REQUIRED] Environment info

firebase-tools:13.2.1

Platform:macOS

[REQUIRED] Test case

I recently updated a number of firebase dependencies on my library and noticed that hitting the emulator's Firebase functions took a long time and would often time out and be left with an internal error.

I realized that if I turned my wifi/internet connection off that they would run quickly again. I kept noticing the following line:

⚠  External network resource requested!
   - URL: "http://169.254.169.254/computeMetadata/v1/universe/universe_domain"

This is followed by the function timing out:

i  Your function timed out after ~60s. To configure this timeout, see
      https://firebase.google.com/docs/functions/manage-functions#set_timeout_and_memory_allocation.
i  Request to function failed: Error: socket hang up

I dug around and found out that this request was being made through gcp-metadata and timing out since it can't reach the specified IP which I'm guessing is an internal IP to Google Cloud.

I run the emulators in a Docker container and think that the environment is triggering some part of gcp-metadata or the firebase emulators to believe it is running on a Google Cloud instance and is timing out. I didn't notice this behavior while on firebase-tools version 11.17.0.

I checked the versions of gcp-metadata:

dereekb@dbMBP dbcomponents % npm ls gcp-metadata google-auth-library 
@dereekb/[email protected] /Users/dereekb/development/git/dbcomponents
├─┬ @google-cloud/[email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └── [email protected]
├─┬ @google-cloud/[email protected]
│ └─┬ [email protected]
│   └── [email protected]
├─┬ [email protected]
│ └─┬ @google-cloud/[email protected]
│   └─┬ [email protected]
│     └── [email protected]
├─┬ [email protected]                            <---- imports older dependencies
│ ├─┬ @google-cloud/[email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ └─┬ [email protected]
│ │   └─┬ [email protected]
│ │     └── [email protected]
│ └─┬ [email protected]
│   └── [email protected]          <---- version 4.3.1
├── [email protected]
└─┬ [email protected]
  └── [email protected] deduped

For now as a work around I am using the environment variable GCE_METADATA_HOST to set the host to 0.0.0.0 for the docker container so that the request fails immediately.

⚠  External network resource requested!
   - URL: "http://0.0.0.0/computeMetadata/v1/universe/universe_domain"
 - Be careful, this may be a production service.

In the newer versions of gcp-metadata (^5.3.0) I'd be able to use METADATA_SERVER_DETECTION=none.

[REQUIRED] Steps to reproduce

You can run the project at https://github.com/dereekb/dbx-components on version v10.0.17 (version v10.0.18 will have the workaround added to the Dockerfile). Run both the server ./serve-server.sh and web client ./serve-web.sh and go to http://localhost:9010/demo/app/profile/view. Change the profile info to kick off a firebase functions request.

[REQUIRED] Expected behavior

Emulator functions should not attempt to connect to or time out while trying to reach "http://169.254.169.254/computeMetadata/v1/universe/universe_domain".

[REQUIRED] Actual behavior

The first emulator function invocation attempts to reach 169.254.169.254 and times out after 60 seconds, causing the initial invocation to return an internal error.

[naoina]

I encountered the same issue and probably it is caused by this changes of google-gax v4.3.0.
I backed to use google-gax v4.2.1, then it works well even if firebase-tools v13.2.1.

[joehan]

Thanks for reporting this! This call is to the compute metadata server, which will only ever work on cloud compute instances. Ideally, google-gax would smoothly handle uses outside of cloud, so I've opened googleapis/gax-nodejs#1566

[inlined]

A fix will hopefully come out in next week's CLI release. In the meantime you can work around the issue by putting METADATA_SERVER_DETECTION=none in $YOUR_FUNCTIONS_DIRECTORY/.env.local

[inlined]

Closing as the fix is in next week's release.

[apeiniger]

I'm seeing the same behavior in the newest version (13.6.1):

Tried to fix it via the env variable mentioned above but it doesn't work.

Any ideas what could cause this?

[Akronae]

Same issue with [email protected] on Windows WSL.
However it's not happening on MacOS.

[Akronae]

I forgot to npx firebase login on my machine. This solved the error.
Errors messages should really be improved.

[EnduringBeta]

A fix will hopefully come out in next week's CLI release. In the meantime you can work around the issue by putting METADATA_SERVER_DETECTION=none in $YOUR_FUNCTIONS_DIRECTORY/.env.local

The timeout this issue refers to is still happening to me in GitHub Actions, and this .env.local suggestion did not fix it.

Edit: This issue does not occur if I'm logged in via firebase login. The error does occur if I'm logged out (using FIREBASE_TOKEN), like in a CI environment.

Even though METADATA_SERVER_DETECTION="none" should be valid, and a search in my repo shows 6.1.0, it doesn't work for me. I've decided to go with OP's suggestion of GCE_METADATA_HOST="0.0.0.0".

Maybe my repo's 6.1.0 version doesn't matter since firebase-tools is global.

This is withfirebase-tools 13.8.3.

[glorat]

Just to note I have also run into this issue if I'm using a local development environment with no GCP (firebase nor application default) credentials and am running the emulator under a demo project. The GCE_METADATA_HOST="0.0.0.0" trick also works for me.

I'm on latest firebase-tools of 13.12.0

Since I have this reproduced on my local laptop, ping me if further diagnostics are needed to figure out what's up

[james-dellow]

I'm encountering this on firebase-tools 13.15.0 running on Windows 11. Neither METADATA_SERVER_DETECTION="none" or GCE_METADATA_HOST="0.0.0.0" worked. I'm only using functions, firestore, hosting in the emulator.

[EnduringBeta]

@inlined Should this issue be reopened? I and several others are reporting that we're still dealing with the same behavior.

[afmeirelles]

@inlined Should this issue be reopened? I and several others are reporting that we're still dealing with the same behavior.

I do think so. In my case, I'm running the emulator inside a docker container that has no credentials as well, so even with the ENV VARs set, I get the error 'GaxiosError: request to http://0.0.0.0/computeMetadata/v1/universe/universe-domain failed, reason: connect ECONNREFUSED 0.0.0.0:80'

[james-dellow]

The METADATA_SERVER_DETECTION override isn't working for me. I've resolved this temporarily:

• Not calling functions.config() when running on the emulator.
• Editing google-auth-library\build\src\auth\googleauth.js to return false on _checkIsGCE(), as it was trigging the call to the metadata endpoint.

I'll open a separate issue for this.

[jirawatee]

I also found this issue with Firebase CLI v13.18.0 and I fixed it by adding "GCE_METADATA_HOST=localhost:8080" to .env.local

Hope this help :)

[EnduringBeta]

@joehan I suggest this issue be reopened. It may have a workaround, but people are still encountering it.

[EduardJoy]

Greetings!

This issue is still happening as of this moment.

The proposed fix, to set GCE_METADATA_HOST="0.0.0.0" causes an error. See attached.

[rahul-sekhar]

The METADATA_SERVER_DETECTION override didn't work for me either. I don't see this env variable being checked in the codepath in gcp-metadata 6.1.0.

I'm instead forcing the GCP residency check to false, and this seems to be working for me:

import { setGCPResidency } from "gcp-metadata";

if (process.env.FUNCTIONS_EMULATOR !== undefined) {
  console.log("Running in emulator mode - forcing GCP residency to false");
  setGCPResidency(false);
}

[rahul-sekhar]

I found the root cause on my end and no longer need to use the fix above. We were using FIREBASE_TOKEN to run the emulators on our CI, which is now deprecated. If I switch to using a service account and GOOGLE_APPLICATION_CREDENTIALS instead, I don't see the issue any more.