Jest fails with Couldn't find temp query result error

[ascorbic]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

As part of a site's unit tests, it may need to compile the site. Previously this was simple to do by running the cli via child_process or execa. This works until [email protected], but since [email protected] this fails. The site builds correctly if build manually from the CLI.

Reproduction Link

https://github.com/ascorbic/gatsby-jest-bug

Steps to Reproduce

1. Clone the repo
2. Run yarn install
3. Run yarn test

Expected Result

Site builds and test passes

Actual Result

Site build fails with the following error:

success Rewriting compilation hashes - 0.001s

     ERROR

    UNHANDLED REJECTION Couldn't find temp query result for "/404/".



      Error: Couldn't find temp query result for "/404/".

      - page-data.ts:104 readPageQueryResult
        [jest-demo]/[gatsby]/src/utils/page-data.ts:104:11

      - page-data.ts:121 writePageData
        [jest-demo]/[gatsby]/src/utils/page-data.ts:121:18

      - page-data.ts:232
        [jest-demo]/[gatsby]/src/utils/page-data.ts:232:24

Environment

System:
    OS: macOS 11.6.4
    CPU: (8) arm64 Apple M1
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.11.0 - /var/folders/k2/ffw3_41j7h17rzsmx25wcwl80000gn/T/yarn--1646216980464-0.9843919045068601/node
    Yarn: 1.22.17 - /var/folders/k2/ffw3_41j7h17rzsmx25wcwl80000gn/T/yarn--1646216980464-0.9843919045068601/yarn
    npm: 8.3.0 - ~/.nvm/versions/node/v16.11.0/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 98.0.4758.102
    Safari: 14.1.2
  npmPackages:
    gatsby: ^4.9.0 => 4.9.0

Config Flags

No response

[JeffersonCarvalh0]

I'm also experiencing this problem with [email protected]

[Nowaker]

Same for me.

es-createPages: 39.095s
success createPages - 89.772s
success createPagesStatefully - 0.051s
info Total nodes: 6030, SitePage nodes: 1642 (use --verbose for breakdown)
verbose 99.876148055 Number of node types: 10. Nodes per type: Directory: 750, File: 1801, JekyllConfigBundled: 1, JekyllData: 647, JekyllDataBundled: 1, JekyllHtml: 1154, Site: 1, SiteBuildMetadata: 1, SitePage: 1642, SitePlugin: 32
verbose 99.876242955 Checking for deleted pages
verbose 99.877802396 Deleted 0 pages
verbose 99.878584229 Found 1642 changed pages
success Checking for changed pages - 0.001s
success onPreExtractQueries - 0.000s
success extract queries from components - 1.687s
success write out redirect data - 0.004s
success onPostBootstrap - 0.001s
info bootstrap finished - 101.578s
success write out requires - 0.015s
success Building production JavaScript and CSS bundles - 5.234s
success Building HTML renderer - 9.245s
success Execute page configs - 0.019s
success Caching Webpack compilations - 0.001s
success run queries in workers - 2.095s - 1642/1642 783.85/s
success Merge worker state - 0.007s
success Rewriting compilation hashes - 0.001s
error UNHANDLED REJECTION Couldn't find temp query result for "/es/thank-you/index.html".
  Error: Couldn't find temp query result for "/es/thank-you/index.html".
  
  - page-data.ts:104 readPageQueryResult
    [_gatsby]/[gatsby]/src/utils/page-data.ts:104:11
  
  - runMicrotasks
  
  - task_queues:61 runNextTicks
    node:internal/process/task_queues:61:5
  
  - timers:437 processImmediate
    node:internal/timers:437:9
  
  - page-data.ts:121 writePageData
    [_gatsby]/[gatsby]/src/utils/page-data.ts:121:18
  
  - page-data.ts:232 
    [_gatsby]/[gatsby]/src/utils/page-data.ts:232:24
  
not finished Writing page-data.json files to public directory - 4.315s

Env: Linux (CI), Gatsby 4.7.2.

Note: it started happening recently. I haven't seen this error before. No changes were done to the site in between. CI just started encountering this problem out of the blue moon. Trying yarn install --frozen-lockfile now to see if that fixes the problem. In which case a diff between by locked dep resolution and most recent dep resolution will be the issue. Otherwise... who knows.

[Nowaker]

yarn install --frozen-lockfile helped.

[marciolima-pray]

I'm experiencing the same issue with gatsby: ^4.11.0.

Running yarn install --frozen-lockfile doesn't help.. 😔

[ZeldOcarina]

Same issue here! :(
I am in 4.14.0

[Oakwhisper]

Creating a new blank gatsby project builds locally on MacOS Monterrey (ARM & x64) but fails on jenkins server build (Debian 11).
Error is:

success Merge worker state - 0.003s
success Rewriting compilation hashes - 0.062s
error UNHANDLED REJECTION Couldn't find temp query result for "/404/".

�[0m
�[0m  �[0m�[97m�[41mError�[0m�[37m�[41m:�[0m�[37m�[41m �[0m�[97m�[41mCouldn't find temp query result for "/404/".�[0m
�[0m  �[0m
�[0m  �[0m�[90m-�[0m �[0m�[93mpage-data.ts�[0m�[90m:�[0m�[93m104�[0m�[37m �[0m�[37mreadPageQueryResult�[0m
�[0m  �[0m  �[0m�[90m[gatsby-test-project]/[gatsby]/src/utils/page-data.ts:104:11�[0m
�[0m  �[0m
�[0m  �[0m�[90m-�[0m �[0m�[93mpage-data.ts�[0m�[90m:�[0m�[93m121�[0m�[37m �[0m�[37mwritePageData�[0m
�[0m  �[0m  �[0m�[90m[gatsby-test-project]/[gatsby]/src/utils/page-data.ts:121:18�[0m
�[0m  �[0m
�[0m  �[0m�[90m-�[0m �[0m�[93mpage-data.ts�[0m�[90m:�[0m�[93m232�[0m�[37m �[0m
�[0m  �[0m  �[0m�[90m[gatsby-test-project]/[gatsby]/src/utils/page-data.ts:232:24�[0m
�[0m  �[0m
�[0m
not finished Writing page-data.json files to public directory - 0.408s

Dependencies for brand new gatsby project:

  "dependencies": {
    "@emotion/react": "^11.9.0",
    "@emotion/styled": "^11.8.1",
    "gatsby": "^4.14.1",
    "gatsby-plugin-emotion": "^7.14.0",
    "gatsby-plugin-manifest": "^4.14.0",
    "react": "^17.0.1",
    "react-dom": "^17.0.1"
  }

[enriqueArb]

I had this issue too with v4.12.1.
For me, the solution was to run gastby clean. After that, build ran correctly.

Hope it helps someone.

[belfz]

Same on gatsby v4.15.2. Neither gatsby clean nor yarn install --frozen-lockfile helped. The error also feels very random, each time mentioning other page within the Couldn't find temp query result for (...) error message.

Precisely I'm getting

 ERROR 

UNHANDLED REJECTION Couldn't find temp query result for "/learn-more/en-us".



  Error: Couldn't find temp query result for "/learn-more/en-us".
  
  - page-data.ts:104 readPageQueryResult
    [www-2022]/[gatsby]/src/utils/page-data.ts:104:11
  
  - task_queues:60 runNextTicks
    node:internal/process/task_queues:60:5
  
  - timers:442 processImmediate
    node:internal/timers:442:9
  
  - page-data.ts:121 writePageData
    [www-2022]/[gatsby]/src/utils/page-data.ts:121:18
  
  - page-data.ts:232 
    [www-2022]/[gatsby]/src/utils/page-data.ts:232:24

upon running gatsby build.

learn-more is created using createPage API, precisely:

actions.createPage({
        path: `/learn-more/en-us`,
        component: path.resolve(`src/non-cms-pages/learn-more.tsx`),
        context: {
          lang,
        },
      });

[belfz]

@LekoArts here's a URL to the faulty project (minimal reproduction): https://github.com/belfz/gatsby-bug-reproduce

[LekoArts]

@belfz Your reproduction shows the problem because you're using the createPage action in the wrong lifecycle hook. It's only allowed in createPages while you use it in onCreateWebpackConfig.

@ascorbic The PR #35968 should fix this, wasn't easy to spot this 😅

[ascorbic]

Ooh, good find! Sneaky one

[Nowaker]

I wouldn't consider it resolved. My use case doesn't involve executing createPage in onCreateWebpackConfig.

[LekoArts]

If you can provide a minimal reproduction we'd be able to look at it

[belfz]

big thank you @LekoArts, indeed that was it - I didn't spot that I was using a wrong lifecycle hook 😅

[beamercola]

@Nowaker Agreed - I'm getting this intermittently on netlify deploys and I'm not doing anything funny with createPage. Did you resolve your problem?

[Nowaker]

@beamercola Not really. Going back to some older lockfile did resolve most of it, but not all. It happens every once in a while.

[quanglam2807]

We're facing the same issue when running build on Netlify. It only happens sometimes.

[zolon4]

I'm getting the same issue on netlify when running build. Sometimes clearing the netlify cache fixes but not always.

[ZeldOcarina]

Yeah and just so you know, this exact problem drove me to using Jenkins and dedicated servers and even local builds instead of Gatsby Cloud as it was always failing there.. and working on my local mac..

[cesarvarela]

Here is the error in all its glory; it happens at "random" and usually is fixed by rerunning the build with an empty cache:

https://app.netlify.com/sites/aiid/deploys/633bd0132ef6160009949592

(I know, I know, netlify, but this is the example I have at hand)

The repo: https://github.com/responsible-ai-collaborative/aiid

My spider sense tells me that it has to do with concurrency/size, and a minimal repro case is impossible.

edit: we are also considering moving to Jenkins :/

[graysonhicks]

@coreyward @ZeldOcarina @cesarvarela Thanks, I know it's frustrating, it's just near impossible for us to address without a reproduction locally. Do any of you have links to a build where this is happening on Gatsby Cloud? I have more tools available on that end, but I don't see any links to builds on Gatsby Cloud, or any links to reproductions that happen locally.

[ZeldOcarina]

I do!
https://www.gatsbyjs.com/dashboard/e968afff-0cff-4103-8d02-5dbbe05c6a6f/sites/26587904-b4fa-402f-b570-2cb1bbabca80/builds/61d86c44-1010-4154-a00e-f82d681b9823/details#rawLogs
If you check inside that project there are so many of these builds failing while locally these are working!

[graysonhicks]

Thanks @ZeldOcarina! I'll open up a ticket on the Gatsby Cloud side and take a look. Do you mind sending an email to [email protected] and tag me in that email and we can track it that way and report back on this thread.

[graysonhicks]

Actually @ZeldOcarina I don't see the Couldn't find temp query result error in any of those builds...

[graysonhicks]

@ZeldOcarina For the builds you sent, they are OOM'ing. Often, it OOMs and exits before the error can be written to the logs. Without looking at the code for a while, I can't say why exactly (at a glance I think adding width option to your image queries would help), but this doc is useful for improving memory consumption.

But I still don't see an instance of the temp query result error and I don't think memory usage is related. When I bumped that site to 8GB of memory (from 4GB) it builds successfully.

[ZeldOcarina]

Hi @graysonhicks,
You are right it is failing out of memory heap as it has so many GatsbyImages.
Actually I couldn't handle that error so I just used normal img tags unfortunately on all next projects.
When building locally it sometimes builds and it sometimes fails with cannot find property "id" of null, with appearently no resons.
Weird..

[TRegourd]

I am also experiencing it, randomly. It's a shame because my site is supposed to rebuild automatically on a certain interval, but one time in two I have to clear cache and relaunch the build myself ...

[hotfix519]

Also running into this since updating to v4 (from v2). Very intermittent. It definitely seems to happen more frequently with our larger sites. If this issue is related to PQR/LMDB stuff, is it possible to add additional debug logging to it? Or perhaps a way to open up the cache db to try and figure out where things have gone off the rails.

I have a netlify instance where it fails maybe 1/6th of the time. I would be willing to possibly run some tests with our data set.

We are not using GatsbyImages in our site.

[cesarvarela]

I've had some success setting GATSBY_CPU_COUNT=2.

I guess that it limits how much stuff Gatsby tries to do at once, limiting memory consumption.

Also, if you are on CI and have a limited number of cores, it leaves free cores for other processes, like Cypress, local servers, etc.

[hotfix519]

I've had some success setting GATSBY_CPU_COUNT=2.

I don't want to jinx it, b/c it's still not a terribly large sample size, but this has shown improvements for us. I was getting the temp query failure intermittently, but it's been clear for a bit. 🤞 It doesn't seem to significantly affect build times either, which was an initial concern.

I'll report back if it pops back up or stays clear over a larger sample size.

[Nowaker]

Limiting the concurrency is a very bad workaround. The whole selling point of Node.js and Gatsby is the fact it's highly concurrent. No or low concurrency is like using Jekyll - a huge downgrade.

[cesarvarela]

It is a bad workaround depending on how much it affects your build time, which in my case, was not that much different.

[hotfix519]

Also, I don't think any of us are suggesting that this issue can be closed or that this "solves" it. But perhaps it can allow some of us that are stuck w/ these errors to move forward in the interim.

[jfgilmore]

Also, I don't think any of us are suggesting that this issue can be closed or that this "solves" it. But perhaps it can allow some of us that are stuck w/ these errors to move forward in the interim.

This issue has been closed though. @cesarvarela would you be able to put your repo and reproduction details in a new issue? It doesn't look like anyone has done this as yet for builds, this issue was related to Jest. Although I don't think Jest had anything to do with it I believe a new issue needs to be opened and linked here as people are still experiencing this problem.

Unfortunately the repo I am having the issue with is private 😞

[hotfix519]

Unfortunately the repo I am having the issue with is private disappointed

Same here. I do feel like it has something to do with larger amts of data/pages. I wonder if we tried to make a really basic project that's pushing a TON of faked data thru the gatsby pipeline and creating thousands of pages, maybe we can get sth public. I may give it a shot if I have time, sad thing is, I rarely do 😞

[coreyward]

As an additional data point, I too am observing relief from this issue with GATSBY_CPU_COUNT=4 and no discernible difference in build times on Netlify. This makes me wonder if this is partly related to resource contention of virtualized CPUs.

[Nowaker]

This makes me wonder if this is partly related to resource contention of virtualized CPUs.

Even if a problem manifests itself when resource use is high, the core issue in such situations is race conditions caused by buggy concurrent code. It's easy to make bugs even in green threads environment when working with multiple variables, let alone when real workers are involved. It's always easier to trigger race conditions in CPU-contested environments.

[ZeldOcarina]

Hey @Nowaker what is a "green thread" environment?

[Nowaker]

@ZeldOcarina Green threads: https://en.wikipedia.org/wiki/Green_thread - and Node.js itself uses green threads for "concurrency", while some other language may or may not use it. "Concurrency" and not concurrency because Node.js has GIL: https://en.wikipedia.org/wiki/Global_interpreter_lock, though it's still more concurrent than other languages with GIL because I/O waits put its green thread in sleep mode, and a different non-blocked green thread is running.

[ZeldOcarina]

@Nowaker wow what a deep insight, thank you so much!

[iamabdullahmalik]

Did anyone find a solution for this?
I have the same issue with 5.0, I have almost 60k posts and 20GB of images.

[joshuar500]

for netlify, creating an environment variable GATSBY_CPU_COUNT=2 worked for me

[jfgilmore]

for netlify, creating an environment variable GATSBY_CPU_COUNT=2 worked for me

I have this in place but am still getting the error, just less frequently than before (we used to have around 50% of builds fail, now it's probably more like 5%).

[flangham]

Am also having this issue.

[dougkellermeyer]

Still observing this issue - random pages error as @belfz mentioned. Updated my build Netlify commands to gatsby clean && gatsby build and issue seems to have resolved itself.

I built a custom front end used the Gatsby Shopify Starter - https://github.com/gatsbyjs/gatsby-starter-shopify

Also, feel like it's worth mentioning that I experience the same error for local builds and running gatsby clean solves it as well 🤷

[techbecca]

I am also experiencing this issue, gatsby clean solved it.

[julianCast]

Just a reminder, this is still happening. We can't provide a reproducible sample since it appears randomly (racing condition? memory issue?)

1:21:37 PM: error Failed to write page-data for ""/it-it/blog/page/ Couldn't find temp query result for "/it-it/blog/page/".
1:21:37 PM: Error: Couldn't find temp query result for "/it-it/blog/page/".