Node 20.3 Crashes all the time when executed inside docker #48444

ronag · 2023-06-13T08:02:20Z

Node 20.3.0 crashes on start in non-bullseye docker container
In bullseye container node-gyp-build (yarn add bufferutil) fails with Text file busy
In bullseye container + UV_USE_IO_URING=0 everything works

The text was updated successfully, but these errors were encountered:

ronag · 2023-06-14T06:27:53Z

Another possible ref electron/rebuild#1085

ronag · 2023-06-14T06:28:01Z

@nodejs/libuv

bnoordhuis · 2023-06-14T11:31:03Z

"Text file busy" means trying to write a shared object or binary that's already in use.

My hunch is that node-gyp has some race condition in reading/writing files that wasn't manifesting (much) when everything still went through the much slower thread pool, whereas io_uring is fast enough to make it much more visible.

ronag · 2023-06-14T11:45:32Z

Is there a way to disable ioring using a env variable as a temporary workaround when running node-gyp?

bnoordhuis · 2023-06-14T19:51:25Z

Yes, set UV_USE_IO_URING=0 in the environment. Use at your own risk: not a stable thing, will disappear again the future.

ronag · 2023-06-15T08:19:37Z

test-srv3.hq.lan:~# docker run -it node:20.3.0 bash
root@6f6c66eb5077:/# UV_USE_IO_URING=0 yarn add bufferutil
node[8]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xc8e4a0 node::Abort() [node]
 2: 0xc8e51e  [node]
 3: 0xd0a059 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
 4: 0xd0a17c node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xc4bbc4 node::V8Platform::Initialize(int) [node]
 6: 0xc49408  [node]
 7: 0xc497db node::Start(int, char**) [node]
 8: 0x7ff94ec5818a  [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x7ff94ec58245 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xba9ade _start [node]
Aborted

ronag · 2023-06-15T08:34:49Z

This is actually worse than I thought. Node doesn't run at all with 20.3

ronag · 2023-06-15T08:35:22Z

test-srv3.hq.lan:~# docker run -it node:20.3.0 node
node[1]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xc8e4a0 node::Abort() [node]
 2: 0xc8e51e  [node]
 3: 0xd0a059 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
 4: 0xd0a17c node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xc4bbc4 node::V8Platform::Initialize(int) [node]
 6: 0xc49408  [node]
 7: 0xc497db node::Start(int, char**) [node]
 8: 0x7f5c393be18a  [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x7f5c393be245 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xba9ade _start [node]
test-srv3.hq.lan:~# docker run -it node:20.2.0 node
Welcome to Node.js v20.2.0.
Type ".help" for more information.
>

mcollina · 2023-06-15T12:11:48Z

I can reproduce this, and this is quite critical.

mcollina · 2023-06-15T12:11:56Z

cc @nodejs/tsc for visibility

richardlau · 2023-06-15T12:50:33Z

FWIW on two systems I have access to (a Red Hat owned RHEL 8 machine and test-digitalocean-ubuntu1804-docker-x64-1 from the Build infra) docker run -it node:20.3.0 node is fine:

root@test-digitalocean-ubuntu1804-docker-x64-1:~# docker run -it node:20.3.0 node
Unable to find image 'node:20.3.0' locally
20.3.0: Pulling from library/node
bba7bb10d5ba: Pull complete
ec2b820b8e87: Pull complete
284f2345db05: Pull complete
fea23129f080: Pull complete
9063cd8e3106: Pull complete
4b4424ee38d8: Pull complete
0b4eb4cbb822: Pull complete
43443b026dcf: Pull complete
Digest: sha256:fc738db1cbb81214be1719436605e9d7d84746e5eaf0629762aeba114aa0c28d
Status: Downloaded newer image for node:20.3.0
Welcome to Node.js v20.3.0.
Type ".help" for more information.
>

I can reproduce the assertion failure on an Ubuntu 16.04 host with node:20.3.0 but not with node:20.3.0-bullseye:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it node:20.3.0 node
Unable to find image 'node:20.3.0' locally
20.3.0: Pulling from library/node
bba7bb10d5ba: Pull complete
ec2b820b8e87: Pull complete
284f2345db05: Pull complete
fea23129f080: Pull complete
9063cd8e3106: Pull complete
4b4424ee38d8: Pull complete
0b4eb4cbb822: Pull complete
43443b026dcf: Pull complete
Digest: sha256:fc738db1cbb81214be1719436605e9d7d84746e5eaf0629762aeba114aa0c28d
Status: Downloaded newer image for node:20.3.0
node[1]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xc8e4a0 node::Abort() [node]
 2: 0xc8e51e  [node]
 3: 0xd0a059 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
 4: 0xd0a17c node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xc4bbc4 node::V8Platform::Initialize(int) [node]
 6: 0xc49408  [node]
 7: 0xc497db node::Start(int, char**) [node]
 8: 0x7f6e8486218a  [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x7f6e84862245 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xba9ade _start [node]
root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it node:20.3.0-bullseye node
Unable to find image 'node:20.3.0-bullseye' locally
20.3.0-bullseye: Pulling from library/node
93c2d578e421: Already exists
c87e6f3487e1: Already exists
65b4d59f9aba: Already exists
d7edca23d42b: Already exists
25c206b29ffe: Already exists
599134452287: Pull complete
bd8a83c4c2aa: Pull complete
d11f4613ae42: Pull complete
Digest: sha256:ceb28814a32b676bf4f6607e036944adbdb6ba7005214134deb657500b26f0d0
Status: Downloaded newer image for node:20.3.0-bullseye
Welcome to Node.js v20.3.0.
Type ".help" for more information.
>

Our website build is actually broken running apt update with the default Node.js LTS image based on Debian 12 (bookworm) - - we've switched to the Debian 11 (bullseye) based image for now: nodejs/build#3382

targos · 2023-06-15T13:12:41Z

FWIW I opened an issue about this in the docker-node repo: nodejs/docker-node#1918

TLDR: this is not a problem with Node.js itself, but with the default base OS used by the Docker image, which was upgraded for v20.3.0.

ronag · 2023-06-15T13:23:24Z

bullseye works for me as well

ronag · 2023-06-15T13:25:50Z

Now I also get the file busy error:

test-srv3.hq.lan:~# docker run -it node:20.3.0-bullseye bash
root@ed020dd3f80e:/# yarn add bufferutil
yarn add v1.22.19
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /node_modules/bufferutil: Command failed.
Exit code: 126
Command: node-gyp-build
Arguments: 
Directory: /node_modules/bufferutil
Output:
/bin/sh: 1: node-gyp-build: Text file busy

EDIT: Works with UV_USE_IO_URING=0

ronag · 2023-06-15T13:34:03Z

So to summarize:

Node 20.3.0 crashes on start in non-bullseye docker container
In bullseye container install fails with "node-gyp-build: Text file busy"
In bullseye container + UV_USE_IO_URING=0 everything works

ronag · 2023-06-15T13:46:20Z

Should I split the uring problem into a separate issue?

bnoordhuis · 2023-06-15T15:18:05Z

Can someone post the result of strace -o trace.log -yy -f node app.js when it crashes with that uv_thread_create check? I expect to see a failing clone/clone2/clone3 system call but it'd be good to confirm.

richardlau · 2023-06-15T16:39:17Z

On the Ubuntu 16.04 infra machine I cannot run apt in the bookworm based node:20.3.0 or node:lts containers to install strace in them (it's not there by default).

Another datapoint, adding --security-opt=seccomp:unconfined makes this work on the Ubuntu 16.04 host:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run --security-opt=seccomp:unconfined -it node:20.3.0 node
Welcome to Node.js v20.3.0.
Type ".help" for more information.
>

bnoordhuis · 2023-06-15T17:18:15Z

Right, then I can predict with near 100% certainty what the problem is: docker doesn't know about the newish clone3 system call. Its seccomp filter rejects it with some bogus error and node consequently fails when it tries to start a new thread.

This docker seccomp thing is like clockwork, it always pops up when new system calls are starting to see broader use. It's quite possibly fixed in newer versions.

mcollina · 2023-06-16T08:38:21Z

Updating docker to the latest version fixed it (v24.0.2) for me.

A few notes:

Ubuntu 22/04 LTS ships with Docker v20.x, which does not support this.
I did not test any version in between, and I couldn't quickly identify what releases of Docker fixed it. From various comments in issues, it seems runc (a dependency of Docker) fixed it in v1.0.2.

Here is what I think we should do:

document this error and the UV_USE_IO_URING=0 solution for v20
disable io_uring when we backport libuv in LTS lines

This seems a future-proof solution while keeping the current functionality available.

bnoordhuis · 2023-06-16T08:40:46Z

document this error and the UV_USE_IO_URING=0 solution for v20

UV_USE_IO_URING is (intentionally) undocumented and going away again so don't do that.

mcollina · 2023-06-16T09:20:41Z

@bnoordhuis Would you just document this as "if you are hit by this bug, update docker"?

ronag · 2023-06-16T09:25:50Z

I think there are two different things here. I'm not sure updating docker will help with the uring problem. Or does it? Please confirm.

santigimeno · 2023-06-16T09:28:02Z

If I'm reading this correctly there are 2 separate issues here.

The crash in some instances. This seems to be directly related to a bug with docker and nothing to do with io_uring.
The Text file busy error, which might or might not be io_uring related but, at least, seems to be exacerbated by the use of io_uring.

I think we should try to understand better the 2nd issue before disabling it.

kamalkech · 2024-01-03T11:03:36Z

i have the same, what is the correct solution ?

LeoK80 · 2024-04-26T09:01:43Z

i have the same, what is the correct solution ?

Try upgrade to latest container runtime (docker, containerd, etc.) to latest. If nothing newer is picked up from your package manager, consider upgrading manually.

kode54 · 2024-05-18T01:38:42Z

Experiencing this with nodejs 22.0.0-1 from official Arch packages, on x86_64, as of kernel 6.9.1. (Also experienced it with -rc7 before that, so I downgraded to 6.8.9 at the time.)

getong · 2024-05-18T10:54:08Z

archlinux 6.9.1, nodejs 22.0.0-1, same error

node-gyp or node have a bug that prevents building with "text file busy" if the kernel is too fast, so we have to disable IO_URING support. This is cleary a hack and needs to be removed as soon as possible nodejs/node#48444 is the necro bumped thread originally from docker

colddegree · 2024-05-20T12:23:03Z

Same problem

➜  ~ uname -a
Linux tuf 6.9.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 17 May 2024 16:56:38 +0000 x86_64 GNU/Linux
➜  ~ pacman -Q | grep node        
node-gyp 10.1.0-2
nodejs 22.2.0-1
nodejs-nopt 7.2.0-1

daniluk4000 · 2024-05-20T12:29:48Z

Btw we still use bullseye, works pretty good

Vortetty · 2024-05-21T17:35:19Z

not to bring up an old issue again but this appears to be a reoccurring bug. reproducing it consistently with kernel 6.9.1-arch1 and node-gyp 10.1.0-2 nodejs 22.2.0-1 nodejs-nopt 7.2.0-1 packages on Crystal Linux (basically arch)

it is fixed with the UV_USE_IORING=0 tag, but i'd rather not have to use that if a fix can be found for newer kernels as well

given that that does fix it, it may also podentially be an issue with libuv-1.48.0-2

mcollina · 2024-05-21T17:50:01Z

cc @santigimeno can you take another look?

richardlau · 2024-05-21T17:57:05Z

it is fixed with the UV_USE_IORING=0 tag, but i'd rather not have to use that if a fix can be found for newer kernels as well

This is supposed to be the default for Node.js (since the February security releases).

https://nodejs.org/docs/latest-v22.x/api/cli.html#uv_use_io_uringvalue

io_uring is disabled by default due to security concerns.

santigimeno · 2024-05-21T19:13:25Z

A patch has just been sent to the kernel fixing this:
https://lore.kernel.org/io-uring/[email protected]/T/#u

It should land in stable shortly:
https://lore.kernel.org/io-uring/[email protected]/T/#t

santigimeno · 2024-05-21T19:30:12Z

This is supposed to be the default for Node.js (since the February security releases).

https://nodejs.org/docs/latest-v22.x/api/cli.html#uv_use_io_uringvalue

I just tested with 22.2.0 installed from nvm and as documented, io_uring is disabled there. Maybe is there a problem in the arch linux package?

gengjiawen · 2024-05-22T01:24:41Z

If you run into this on ubuntu 16 or 18, my fix is use ubuntu >= 20.04. The issue actually comes from docker for me.

kode54 · 2024-05-22T01:41:28Z

This is supposed to be the default for Node.js (since the February security releases).
https://nodejs.org/docs/latest-v22.x/api/cli.html#uv_use_io_uringvalue

I just tested with 22.2.0 installed from nvm and as documented, io_uring is disabled there. Maybe is there a problem in the arch linux package?

How is arch supposed to be disabling io_uring? It configures nodejs to use the system libuv, and builds its libuv with the default options.

santigimeno · 2024-05-22T05:14:32Z

How is arch supposed to be disabling io_uring? It configures nodejs to use the system libuv, and builds its libuv with the default options.

That's likely the problem. Due to the security reasons mentioned above node.js patched libuv to disable io_uring in the following commits: 42e659c and 6d14352. Maybe the arch packaging hasn't taken that into account?

mcollina · 2024-05-22T08:18:56Z

Thanks @santigimeno for the help debugging this.
This issue can indeed be closed.

felixonmars · 2024-05-22T09:46:01Z

How is arch supposed to be disabling io_uring? It configures nodejs to use the system libuv, and builds its libuv with the default options.

That's likely the problem. Due to the security reasons mentioned above node.js patched libuv to disable io_uring in the following commits: 42e659c and 6d14352. Maybe the arch packaging hasn't taken that into account?

I am the Arch packager and indeed I have missed this change.

Opened libuv/libuv#4416 to see if there is a better way forward.

https://aur.archlinux.org/cgit/aur.git/commit/?h=thelounge&id=fd50c63 node-gyp or node have a bug that prevents building with "text file busy" if the kernel is too fast, so we have to disable IO_URING support. This is cleary a hack and needs to be removed as soon as possible nodejs/node#48444 is the necro bumped thread originally from docker

For more info see nodejs/node#48444 (comment).

tniessen added the libuv Issues and PRs related to the libuv dependency or the uv binding. label Jun 13, 2023

ronag changed the title ~~Node 20.3 breaks bufferutil build?~~ Node 20.3 breaks builds? Jun 14, 2023

ronag changed the title ~~Node 20.3 breaks builds?~~ Node 20.3 Crashes Jun 15, 2023

mcollina added the confirmed-bug Issues with confirmed bugs. label Jun 15, 2023

mcollina changed the title ~~Node 20.3 Crashes~~ Node 20.3 Crashes all the time when executed inside docker Jun 15, 2023

ronag mentioned this issue Jun 15, 2023

Node 20.3 - node-gyp-build Text file busy #48468

Closed

thepetk mentioned this issue Oct 19, 2023

Devfile Registry deployment on minikube with helm keeps crash looping devfile/api#1295

Closed

parmentf mentioned this issue Nov 30, 2023

Use an older version of node in bases Inist-CNRS/web-services#11

Closed

mrvdot mentioned this issue Dec 26, 2023

supabase start fails to start supabase/cli#1780

Closed

lydell mentioned this issue Feb 17, 2024

Add workaround for io_uring bug on Linux lydell/run-pty#54

Closed

spljs mentioned this issue Apr 8, 2024

Upgrade Cube to NodeJS 20 cube-js/cube#8087

Closed

polyzen mentioned this issue May 20, 2024

just clone my project repo on my new pc and got bad interpreter error. typicode/husky#1444

Closed

1 task

mcollina reopened this May 21, 2024

mcollina closed this as completed May 22, 2024

archlinux-github pushed a commit to archlinux/aur that referenced this issue Jun 2, 2024

Fix for ongoing problem with yarn.

80d6135

For more info see nodejs/node#48444 (comment).

archlinux-github pushed a commit to archlinux/aur that referenced this issue Jun 2, 2024

Fix for ongoing problem with yarn.

40157e0

For more info see nodejs/node#48444 (comment).

yuft mentioned this issue Jul 8, 2024

Nodejs >v18 throws exception pulumi/pulumi#16596

Closed

Node 20.3 Crashes all the time when executed inside docker #48444

Node 20.3 Crashes all the time when executed inside docker #48444

Comments

ronag commented Jun 13, 2023 • edited Loading

ronag commented Jun 14, 2023

ronag commented Jun 14, 2023

bnoordhuis commented Jun 14, 2023

ronag commented Jun 14, 2023

bnoordhuis commented Jun 14, 2023

ronag commented Jun 15, 2023

ronag commented Jun 15, 2023

ronag commented Jun 15, 2023 • edited Loading

mcollina commented Jun 15, 2023 • edited Loading

mcollina commented Jun 15, 2023

richardlau commented Jun 15, 2023 • edited Loading

targos commented Jun 15, 2023

ronag commented Jun 15, 2023

ronag commented Jun 15, 2023 • edited Loading

ronag commented Jun 15, 2023 • edited Loading

ronag commented Jun 15, 2023

bnoordhuis commented Jun 15, 2023

richardlau commented Jun 15, 2023 • edited Loading

bnoordhuis commented Jun 15, 2023

mcollina commented Jun 16, 2023

bnoordhuis commented Jun 16, 2023

mcollina commented Jun 16, 2023

ronag commented Jun 16, 2023 • edited Loading

santigimeno commented Jun 16, 2023 • edited Loading

kamalkech commented Jan 3, 2024

LeoK80 commented Apr 26, 2024

kode54 commented May 18, 2024

getong commented May 18, 2024

colddegree commented May 20, 2024

daniluk4000 commented May 20, 2024

Vortetty commented May 21, 2024 • edited Loading

mcollina commented May 21, 2024

richardlau commented May 21, 2024

santigimeno commented May 21, 2024 • edited Loading

santigimeno commented May 21, 2024

gengjiawen commented May 22, 2024

kode54 commented May 22, 2024

santigimeno commented May 22, 2024

mcollina commented May 22, 2024

felixonmars commented May 22, 2024 • edited Loading

ronag commented Jun 13, 2023 •

edited

Loading

ronag commented Jun 15, 2023 •

edited

Loading

mcollina commented Jun 15, 2023 •

edited

Loading

richardlau commented Jun 15, 2023 •

edited

Loading

ronag commented Jun 15, 2023 •

edited

Loading

ronag commented Jun 15, 2023 •

edited

Loading

richardlau commented Jun 15, 2023 •

edited

Loading

ronag commented Jun 16, 2023 •

edited

Loading

santigimeno commented Jun 16, 2023 •

edited

Loading

Vortetty commented May 21, 2024 •

edited

Loading

santigimeno commented May 21, 2024 •

edited

Loading

felixonmars commented May 22, 2024 •

edited

Loading