Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation Fault Crash when using Prisma #7864

Closed
monkfromearth opened this issue Dec 27, 2023 · 15 comments · Fixed by #8995
Closed

Segmentation Fault Crash when using Prisma #7864

monkfromearth opened this issue Dec 27, 2023 · 15 comments · Fixed by #8995
Labels
bug Something isn't working crash An issue that could cause a crash needs investigate Needs to be investigated to find the root cause

Comments

@monkfromearth
Copy link

monkfromearth commented Dec 27, 2023

What version of Bun is running?

1.0.20+09d51486e

What platform is your computer?

Linux 5.15.133.1-microsoft-standard-WSL2 x86_64 x86_64

What steps can reproduce the bug?

  • Perform a CPU-intensive task in the same method as where the database insertion can happen
  • Use sharp to transform the image, for example, and try to insert some data

What is the expected behavior?

The insertion should happen normally.

What do you see instead?

error: script was terminated by signal SIGSEGV (Address boundary error)
Segmentation fault

Additional information

We tried to transform an image through sharp and store the S3 path to where they were uploaded.
We refactored the functions so that the transformation happens in another method. The transformations and uploads would happen normally, but during Prisma insertion, the script crashes directly.

This works on 1.0.15, but not since.

Output from bunx prisma -v:

prisma                  : 5.7.1
@prisma/client          : 5.7.1
Computed binaryTarget   : debian-openssl-1.1.x
Operating System        : linux
Architecture            : x64
Node.js                 : v18.17.0
Query Engine (Node-API) : libquery-engine 0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5 (at node_modules/@prisma/engines/libquery_engine-debian-openssl-1.1.x.so.node)
Schema Engine           : schema-engine-cli 0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5 (at node_modules/@prisma/engines/schema-engine-debian-openssl-1.1.x)
Schema Wasm             : @prisma/prisma-schema-wasm 5.7.1-1.0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5
Default Engines Hash    : 0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5
Studio                  : 0.495.0
@monkfromearth monkfromearth added the bug Something isn't working label Dec 27, 2023
@monkfromearth monkfromearth changed the title Segmentation Fault when using Prisma Segmentation Fault Crash when using Prisma Dec 27, 2023
@Jarred-Sumner
Copy link
Collaborator

Jarred-Sumner commented Dec 27, 2023

sorry

to any contributors: without seeing the stack trace, my guess is it's an async context thing. Prisma uses async hooks + napi. Sharp & Prisma uses threadsafe functions. Threadsafe functions sometimes attempt to read from hooks. It's possible something is going wrong there.

Another thought: maybe it's because we don't implement escapable handle scopes and the inner function is async. This would be reproducible for any async NAPI function (which is truly async, not just microtasks). Something like await Bun.sleep(1) inside a NAPI threadsafe function (such as Sharp or Prisma) would reproduce this.

@Electroid Electroid added crash An issue that could cause a crash needs investigate Needs to be investigated to find the root cause labels Dec 27, 2023
@JosephClay
Copy link

JosephClay commented Jan 12, 2024

is there documentation for getting a stack trace for these segfaults? i'm running into this issue as well and would like to help if i can

@Hrdtr
Copy link

Hrdtr commented Jan 12, 2024

I also experienced this issue without any clues.
Similar to the OP, after processing the image using Sharp and uploading it to S3, then the data insertion process is carried out using Prisma and a segvault occurs.

What I can do now to keep my service running is to use bun v1.0.18

@rizrmd
Copy link

rizrmd commented Jan 16, 2024

I can confirm this bug still present on latest bun 1.0.23

@rizrmd
Copy link

rizrmd commented Jan 21, 2024

This bug still exists on 1.0.25 1.0.26. If you want to use prisma in bun, v1.0.18 is your best bet.

GWMCwing added a commit to GWMCwing/dynamic-langchain that referenced this issue Feb 2, 2024
@jontybrook
Copy link

I can confirm I was seeing this in bun v1.0.26. Downgrading to v1.0.18 with the following command (on Linux) seems to have made the problem go away for now...

curl -fsSL https://bun.sh/install | bash -s "bun-v1.0.18"

@AdrianoFerrari
Copy link

I upgraded from bun v1.0.0 to v1.0.26 and I started getting this error intermittently when starting my server.
Downgrading to 1.0.18 made it go away.

Not using prisma. My server code has the following dependencies:

  • bun:sqlite
  • node:fs/promises
  • uuid
  • redis
  • cookie
  • cookie-signatures
  • pizzip
  • docxtemplater

@rizrmd
Copy link

rizrmd commented Feb 11, 2024

Something must gone very wrong in 1.0.19 upwards. I really want to dig it deeper, but... yeah...

@rizrmd
Copy link

rizrmd commented Feb 17, 2024

1.0.27 released, yay!

Still error, but It is throwing different message than previous versions!!

thread 'tokio-runtime-worker' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.30.0/src/sync/rwlock.rs:427:17:
internal error: entered unreachable code
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace 


...minified-js-code-truncated...

                                                                                                                                                                                                                ^
PrismaClientRustPanicError: 
Invalid `prisma.site.findFirst()` invocation:


PANIC: internal error: entered unreachable code

This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.

....minified-js-code-truncated....


If you want the Prisma team to look into it, please open the link above 🙏
To increase the chance of success, please post your schema and a snippet of
how you used Prisma Client in the issue. 

EDIT: After I ran it several times, it throws usual segmentation fault. Back to 1.0.18 i guess...

@camero2734
Copy link
Contributor

camero2734 commented Feb 17, 2024

1.0.27 released, yay!

Still error, but It is throwing different message than previous versions!!

This is the same error I was getting in 1.0.26 as well when running oven/bun:1.0.26-slim in Docker (confirmed also in oven/bun:1.0.27-slim) with Postgres. I'll try to make a minimum example repo that exhibits the behavior in a bit

@camero2734
Copy link
Contributor

camero2734 commented Feb 17, 2024

Repro repo is here: https://github.com/camero2734/prisma-bun-issue (run w/ docker-compose up)

With FROM oven/bun:1.0.27-slim:

prisma-bun-app  | PrismaClientRustPanicError:
prisma-bun-app  | Invalid `prisma.$executeRaw()` invocation:
prisma-bun-app  |
prisma-bun-app  |
prisma-bun-app  | PANIC: internal error: entered unreachable code
prisma-bun-app  |
prisma-bun-app  | This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.
prisma-bun-app  |
prisma-bun-app  | https://github.com/prisma/prisma/issues/new?body=SNIPPED_OUT
prisma-bun-app  |
prisma-bun-app  | If you want the Prisma team to look into it, please open the link above 🙏
prisma-bun-app  | To increase the chance of success, please post your schema and a snippet of
prisma-bun-app  | how you used Prisma Client in the issue.
prisma-bun-app  |
prisma-bun-app  |       at new be (/code/node_modules/@prisma/client/runtime/library.js:26:2141)
prisma-bun-app  |       at handleRequestError (/code/node_modules/@prisma/client/runtime/library.js:126:7136)
prisma-bun-app  |       at handleAndLogRequestError (/code/node_modules/@prisma/client/runtime/library.js:126:6104)
prisma-bun-app  |       at /code/node_modules/@prisma/client/runtime/library.js:126:5812
prisma-bun-app  |
prisma-bun-app  | Encountered error after 14 iterations

thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.30.0/src/sync/rwlock.rs:427:17:
prisma-bun-app  | internal error: entered unreachable code
prisma-bun-app  | stack backtrace:
prisma-bun-app  |    0:     0x14a9b786f34f - <unknown>
prisma-bun-app  |    1:     0x14a9b70a5480 - <unknown>
prisma-bun-app  |    2:     0x14a9b78463be - <unknown>
prisma-bun-app  |    3:     0x14a9b7870d5e - <unknown>
prisma-bun-app  |    4:     0x14a9b7870665 - <unknown>
prisma-bun-app  |    5:     0x14a9b78715c5 - <unknown>
prisma-bun-app  |    6:     0x14a9b7871098 - <unknown>
prisma-bun-app  |    7:     0x14a9b7871026 - <unknown>
prisma-bun-app  |    8:     0x14a9b7871011 - <unknown>
prisma-bun-app  |    9:     0x14a9b6f5a184 - <unknown>
prisma-bun-app  |   10:     0x14a9b6f5a352 - <unknown>
prisma-bun-app  |   11:     0x14a9b6fee9c9 - <unknown>
prisma-bun-app  |   12:     0x14a9b700d71e - <unknown>
prisma-bun-app  |   13:     0x14a9b6fe1f24 - <unknown>
prisma-bun-app  |   14:     0x14a9b6ff4517 - <unknown>
prisma-bun-app  |   15:     0x14a9b7886619 - <unknown>
prisma-bun-app  |   16:     0x14a9b788fde1 - <unknown>
prisma-bun-app  |   17:     0x14a9b7884afd - <unknown>
prisma-bun-app  |   18:     0x14a9b78845a6 - <unknown>
prisma-bun-app  |   19:     0x14a9b7889b57 - <unknown>
prisma-bun-app  |   20:     0x14a9b7872186 - <unknown>
prisma-bun-app  |   21:     0x14aa18327ea7 - start_thread
prisma-bun-app  |   22:     0x14aa1843da6f - clone
prisma-bun-app  |   23:                0x0 - <unknown>

With FROM oven/bun:1.0.18-slim:

prisma-bun-app  |
prisma-bun-app  | No error

The repro code is:

import { createCanvas } from "@napi-rs/canvas";
import { PrismaClient } from '@prisma/client';

const client = new PrismaClient();

async function causeError() {
  // 1. Some DB query
  const tx = client.$executeRaw`SELECT pg_sleep(0.01)`;
  // 2. Some napi-based operation
  createCanvas(1000, 1000);
  // 3. Wait for the DB query to finish
  await tx;
}

let success = true;
for (let i = 0; i < 50; i++) {
  try {
    await causeError();
  } catch (e) {
    console.error(e);
    console.log(`Encountered error after ${i + 1} iterations`);
    success = false;
    break;
  }
}

if (success) console.log("No error");

If I remove 1, 2, or 3, the issue does not reproduce in the latest version of Bun.

Edit: Also found that the problem is gone if I build main with this commit reverted a93f467

@camero2734
Copy link
Contributor

camero2734 commented Feb 17, 2024

Okay, I've narrowed down that set of changes to just a single line in napi_delete_reference:

// src/bun.js/bindings/napi.cpp
extern "C" napi_status napi_delete_reference(napi_env env, napi_ref ref)
{
    NAPI_PREMABLE
    NapiRef* napiRef = toJS(ref);
    
    napiRef->~NapiRef(); // PREVIOUS -- reverting to this fixes the issues with Prisma
    delete napiRef       // CURRENT -- breaks Prisma

    return napi_ok;
}

@rizrmd
Copy link

rizrmd commented Feb 19, 2024

Pinging @Jarred-Sumner, can we just revert this line to solve this issue ?

Jarred-Sumner added a commit that referenced this issue Feb 19, 2024
Jarred-Sumner added a commit that referenced this issue Feb 19, 2024
* Add test for #7864

* Add napi-rs/canvas

---------

Co-authored-by: Jarred Sumner <[email protected]>
@itorz7
Copy link

itorz7 commented Sep 13, 2024

any update ?

image

@rizrmd
Copy link

rizrmd commented Sep 14, 2024

any update ?

image

The segmentation fault issue has been solved, I am not experiencing segfault anymore in recent builds. I am guessing this is memory leak problem, maybe create another issue ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working crash An issue that could cause a crash needs investigate Needs to be investigated to find the root cause
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants