Functions fail when using Firebase Admin SDK

[kristapsPelna]

Functions fail when using Firebase Admin SDK.
When I am requesting data from firestore using firebase admin SDK, I get this error:

(node:14684) Warning: protos.json not found in any of the include paths \protos,\protos
(Use node --trace-warnings ... to show where the warning was created)
Error: ENOENT: no such file or directory, open 'protos.json'
at Object.openSync (fs.js:466:3)
at Object.readFileSync (fs.js:369:35)

Can this be fixed?
Currently the admin SDK is not usable on the functions and getServerSideProps

P.S. Locally works just fine when running next dev but fails when running next build && next start

[lindsaylevine]

hey @kristapsPelna ! saw you post on a thread on our community forum. sorry you're experiencing these issues! this is the first i'm learning of firebase <-> netlify functions issues, and another person just opened a firebase issue here on next-on-netlify yesterday as well.

as i asked them: did you happen to see this thread/solution? https://community.netlify.com/t/netlify-function-timeout-after-10-seconds-when-published/18220/5

let me know if that happens to help.

[kristapsPelna]

@lindsaylevine Thanks for the reply, but unfortunately that is a different issue.
The problem in the mentioned thread refers to getting correct results, but not getting a proper response as the function times out.

In my situation the function fails with an error mentioned above and doesnt even reach the end of the function where I would be able to call admin.app().delete() (which fixes their problem).

[kristapsPelna]

This is a test function that I've been using.
file:
pages/api/test.ts

import { NextApiRequest, NextApiResponse } from 'next';
import admin from 'firebase-admin';

if (!admin.apps.length) {
  admin.initializeApp({
    credential: ...,
    databaseURL: ...,
  });
}
const test = async (req: NextApiRequest, res: NextApiResponse) => {
  const { docs } = await admin
    .firestore()
    .collection('testing')
    .where('name', '==', 'test')
    .limit(1)
    .get();

  const [firstDoc] = docs;
  return res.status(200).json({ result: firstDoc?.data() });
};

export default test;

[FinnWoelm]

Hey @kristapsPelna,

thanks for posting your function code! That's super helpful.

I created a minimal repo to reproduce the issue: https://github.com/FinnWoelm/next-on-netlify-firebase

Here is my API endpoint with Cloud Firestore: https://github.com/FinnWoelm/next-on-netlify-firebase/blob/main/pages/api/test.js
The function in action is here: https://next-on-netlify-firebase.netlify.app/api/test

The issue Warning: protos.json not found in any of the include paths \protos,\protos occurs because Cloud Firestore relies on binary code. By default, Next.js tries to inline all dependencies into one big JS file. That works really well, but not when a function relies on binaries. In that case, we need to tell Next.js to not inline its dependencies by setting target: "experimental-serverless-trace" in next.config.js. Like this:

// next.config.js

module.exports = {
  target: "experimental-serverless-trace",
};

Longer explanation here (the Q focuses on Prisma, but it's the same issue of bundling binaries): #55 (comment)

Can you try that and let us know if that resolves your issue? 😊

[kristapsPelna]

Hey @FinnWoelm,

Thanks for the suggestion!
It works like a charm locally when I do next build and then next start.
But unfortunately it fails when running on Netlify.
The build step for building the Netlify functions took almost 9min
(@netlify/plugin-functions-core onBuild completed in 8m 42.2s)
And failed in the end with Request must be smaller than 69905067 bytes for the CreateFunction operation

[FinnWoelm]

Hi @kristapsPelna,

Hmm... There is a 64MB limit for each Netlify Function (imposed by AWS Lambda). I don't think this is related to API endpoint using firebase-admin that you shared above (since that works here: https://github.com/FinnWoelm/next-on-netlify-firebase/blob/main/pages/api/test.js). The two functions took 50 seconds to package (@netlify/plugin-functions-core onBuild completed in 52.1s) and total build time was 1min 30 seconds.

Is it possible that you have a Next.js page that is bundling a lot of dependencies? I'd be happy to take a look at your code, if it's open. Or, if you prefer to reproduce the issue in a minimal repo, I'd be happy to look at it, too.

You can bundle your functions locally to see which function is exceeding the size (and even inspect the dependencies that are being bundled):

1. npm install -g @netlify/zip-it-and-ship-it
2. npm run build (locally build the Next app and run next-on-netlify — you will have an out_functions folder).
3. zip-it-and-ship-it out_functions zipped_functions

Be patient while your functions are analyzed and bundled with the required node modules. When done, you can open the zipped_functions directory and see the file sizes of the .zip archives.

If you see one with 64MB+ size, that's the one to investigate further. By the way, I also noticed that my zip archives were smalled when switching from Next v10 to Next v9.5. Next v10 uses a dependency called sharp that is used for transforming images, that adds ~ 10 MB zipped.

Hopefully we can narrow in on the issue! 😊

[kristapsPelna]

@FinnWoelm Thanks for the reply.
I checked that most of my functions are 50-76 MB when doing the zip-it.
It also works just fine on a really simple project.
Looks like as a project grows bigger the more dependencies it has and then all of the dependencies the page uses the heavier the functions get.
For me the ones that are 64+MB are the ones which are created from getServerSideProps, while the ones from /api are 45-50 MB.
This is really odd.

P.S. I tried the same project on Vercel and it builds and runs just fine.

[emmbyiringiro]

@lindsaylevine , thanks for quick feedback, with zip-it-and-ship-it I build and bundle my app and found 2 zipped files 49MB and 39MB respectively. By downgrade Next from V10 to 9.5 as @FinnWoelm recommended the bundled files size shrink to 4MB and 5MB respectively 🤯 and now built and deployed successfully.

It seems that Next latest version has large bundle builds and hope you will work on it and optimize experimental-serverless-trace targeted app for it.

[lindsaylevine]

hey all! i'm going to close this issue in favor of #120, which groups several next-on-netlify issues together related to experimental-serverless-trace. this issue mentions that an improvement has been made to our internal bundling library zip-it-and-ship-it, to be released in the new year.

please feel free to re-open this issue if you think #120 does not suffice or if there's a specific piece separate from experimental-serverless-trace that still needs addressing.

[jaobrown]

Hesitant to reopen this, because I'm not sure I have enough context to say that this issue hasn't been solved—however, I tracked with #120 which was updated as closed, and mention of it being released into the build system—but I'm still getting the error in this issue, and can't seem to figure out if there has been a solve.

Any solutions other than downgrading Next to 9.5 at the moment? Or working for anyone else?

[lindsaylevine]

@jaobrown which error exactly are you getting? there are a few different mentioned in here 😅

[jaobrown]

@lindsaylevine Whoops! Sorry about that.

The OP's error—more specifically, mine is verbatim:

Error: ENOENT: no such file or directory, open 'protos.json'

[lindsaylevine]

@jaobrown oh! have you changed your target in your next.config.js to experimental-serverless-trace?

// next.config.js

module.exports = {
  target: "experimental-serverless-trace",
};