[regression] NodeOS don't boot with Node.js v0.11.15

[piranna]

Continue discussion from nodejs/node-v0.x-archive#9097.

Since version v0.11.15 of Node.js that changed the version of v8, NodeOS can't boot entering an infinite loop instead of showing a REPL and consuming 100% of CPU. Recent tests showed that this still happen with Node.js v4.1.1

[ChALkeR]

Could you share a small reproducable testcase that shows that this is a bug in Node.js/v8?

[piranna]

You can be able to download NodeOS and generate yourself with npm install, by generating only the barebones is enought. You can check it with npm start and it will start NodeOS directly to a Node.js REPL.

Later, change the version to something bigger than v0.11.14, delete the deps and obj folders to force regeneration with the new version, and exec again npm install: the new one will enter an infinite loop consuming the 100% of the CPU and not showing the REPL.

[ChALkeR]

@piranna I am afraid that NodeOS doesn't fit into «small».
Could you make a self-contained testcase?

[piranna]

nodeos-barebones is the most minimal use case... Besides that, I only can think about upload somewhere a compiled version with v0.11.14 and another with v0.11.15 (or v4.1.1 if you want, but this is still in experimental status and still not merged in master due to the requeriment of GCC 4.8) so you can be able to check both.

[jbergstroem]

I'm not sure if this is a nodejs problem. What API in nodejs do you interface with? v8 is a very moving target which is why you should use NaN whenever possible.

Edit: I recall prior discussions about incompatibilities between musl and node (v8). Guessing that might be related. Even if you get it to compile I wouldn't assume it'll work as intended. It needs a lot of low level work [probably on both v8 and musl sides] and I'm not sure who would be interested in doing that at the moment.

[piranna]

Nodeos-barebones is only vanilla Node.js running directly on top of a Linux
kernel as PID 1, the only special thing is the usage of musl as C lib.
Besides that, code is untouched and are standard compilations.
El 06/10/2015 12:36, "Johan Bergström" [email protected] escribió:

I'm not sure if this is a nodejs problem. What API in nodejs do you
interface with? v8 is a very moving target which is why you should use NaN
whenever possible.

—
Reply to this email directly or view it on GitHub
#3204 (comment).

[jbergstroem]

@piranna I'm with @ChALkeR here; your case is specific enough to fall outside the scope of what is relevant to this project versus nodeos. If you can create a minimalistic test case which is debuggable it might be more relevant, but seeing how you're using musl I think thats too much to chew at the moment. Can you reproduce the same issue with glibc?

[ChALkeR]

@piranna nodeos-barebones compiles and builds a Linux kernel. It doesn't fit into my definition of «minimal».
Please, research that issue in your code and create a testcase.

[piranna]

We already checked, read the issue pointed on the first comment: the problem came on Node.js v0.11.15 with the change of version of v8, and a lot of people got problems and regressions then, just check old issues tracker. Seems the problem is due to a spin lock on v8:

nodejs/node-v0.x-archive#9113
nodejs/node-v0.x-archive#25137
nodejs/node-v0.x-archive#25295

[jbergstroem]

@piranna v8 will continue to be updated, and we make sure we ship a nodejs binary that works on our supported platforms. Your peculiar setup I would argue is out of that scope. Can you try building against glibc and report back?

[jbergstroem]

@piranna all linked issues has fixed closed to them. The last one seems to be related to using node.js --prof stuff. Do you use that in nodeos?

[piranna]

No, I can't, we are using musl on NodeOS because glibc is too much complicated to compile from scratch. Node.js supports musl and there's a lot of people using it already, and NodeOS uses a vanilla Linux kernel. You could get a similar environment just by copying the files from your Linux system by hand and put them together, since there are no modified code. Honestly, I don't know why you say it's a peculiar setup...

[piranna]

The last one seems to be related to using node.js --prof stuff

No, I'm not using it, I'm using regular Node.js without flags, but the problem seems to be related, since the results are similar.

[jbergstroem]

@piranna there's no official support of nodejs-baked binaries on top of musl as of yet. We're exploring this as part of making smaller docker images, but its still a work in progress. As per above, compile time doesn't mean runtime.

I don't have much to add from here. If anyone else wants to chip in and learn the innings of nodeos that would probably be a next step since we won't get a reduced case.

[piranna]

The Node.js binary used on NodeOS with musl works correctly on Ubuntu when pointing to the musl libraries, and as I told you, NodeOS uses a vanilla Linux kernel and Node.js v0.11.14 works but newer not. The only different thing between NodeOS and an Ubuntu environment that could explain this issue is the usage of /dev/console as stdio. Besides that, it's a plain regular Linux system.

And regarding NodeOS internals, honestly there's not too much to learn here...

[bnoordhuis]

I think the point @jbergstroem is trying to get across it that issues with nodeos are outside the scope of this project. If you can point to a single change or line that introduced the problem, fine, we can look into that. But in general, you should expect to have to support yourself.

[piranna]

If you can point to a single change or line that introduced the problem, fine, we can look into that

I've said it several times, the problem got introduced on v0.11.15, and I did a git bisect to find that the problem was the update of v8 to the 3.28.73 version.

[bnoordhuis]

That V8 upgrade may be a single commit but it's hardly a single change.

[piranna]

Are you proposing I should do a git bisect of v8, too?

[bnoordhuis]

Something like that, yes. If you suspect a change in V8 is causing you grief, you'll have to go in and find it. Don't expect someone else to do it for you.

[piranna]

I've review the history of changes of the directory deps/v8 to see if there's another previous one that could have caused the problem and all of them are integrated in the branch of v0.14.15, being the only one that's on v0.14.15 the one that introduced the problem. The problem is in that commit, but it has changes on 1715 files, so where to start to look for?

And besides the tedious part of doing manually the git bisect, what should I do with the cherry-picks and patches applied to the v8 source code? Or I don't have to worry about them and only need to focus on the changes on master from 3.26.33 to the conflictive 3.28.73 (seems it was already an abandoned branch...)?

[bnoordhuis]

You could start with bisecting V8 and see if the d8 shell runs as PID 1.

[seishun]

My humble opinion. As many people have already pointed out, we don't support NodeOS, so you should rephrase your issue in a way that doesn't involve NodeOS or any other third-party projects. What is it exactly that NodeOS was using that doesn't work anymore? If it turns out that NodeOS was using some internal undocumented feature of V8 or something then we probably can't help you and finding the exact V8 commit that broke it won't help.

[piranna]

@seishun, you are right, I was waiting to have some time to try to bisect V8, but until that I can be able to decribe my problem in a more generic way.

My problem is that trying to use Node.js on Linux as PID 1 (/init), up to Node.js v0.11.14 it gave me a REPL and also I was able to use Node.js apps as PID1, but starting at Node.js v0.11.15 and more exactly after the upgrade of the V8 version, I don't get anymore a Node.js REPL when using it as PID 1 nor use a Node.js app instead, and instead CPU gets at 100% without doing anything.

I would need to test if this use case is reproducible with a standard Ubuntu kernel and Node.js (It's just a matter of launching the Linux kernel setting /usr/bin/node as init), but basically that's the description of my problem.

[piranna]

Checked on Ubuntu Willy with Node.js 0.12.17 on a real (not virtualized) Dual-Core machine: I get a blinking Node.js REPL prompt, but don't respond to the keyboard.

[seishun]

Can you try with the latest Node.js?

[piranna]

Can you try with the latest Node.js?

Same Dual Core machine, new test with Linux 4.2.0-1 from Ubuntu Willy and Node.js 4.2.1 from NodeSource.

On a first test it crashed because the root filesystem was mounted as readonly (the default of Linux when set as kernel command line parameter) about not being able to create the .node_completion file, so it exitted and being the PID1, Linux throw a kernel panic. Setting the root filesystem as read-write it didn't crashed, but the REPL promt was not shown and the console didn't responded, and after some seconds the computer fan started to do some noise so seems the CPU got stuck at 100% too.

It's said: the problem is still there on Node.js 4.2.1 :-(

[piranna]

I've checkout the code just before the conflictive Node.js commit and checked that it worked, later I've changed the deps/v8 folder with a git clone of v8 and none of the commits between 3.26.33 (the same that was included with Node.js) or 3.28.73 worked, sometimes giving me an error about v8.h not being found and anothers about TRANSITION not being defined. Any clue about what could I have done bad or how could I be able to investigate what was the v8 commit that inserted this error? Besided changing the v8 folder with the v8 git repo, I'm out of ideas... :-(

[piranna]

I've done more checks and definitely there are some patches that need to be applied on v8, but don't know where to get them. In fact, If you checkout version 3.28.73 there are some differences with the same version of v8 used on Node.js... The list of commits show that there are several patches applied and backported. In any case, I'm not able to compile it and I'm not sure if this patches would help or if I have the problem in other place, because I've only changed the code of v8 for the one on the git repo and in fact it's being compiled... Honestly, I don't know to whom to ask for advice... :-(

[piranna]

Just checked with Linux 4.2 and Node.js 5.1, still not working :-/

[jbergstroem]

@piranna there's not much we as an upstream project can do until you give us the information people have asked you for (see for instance Ben's comments). I'm suggesting we close this and reopen when that is available.

[piranna]

I have tried to bisect v8 changes, but there's a lot of patches on top of it applied by Node.js team over the upstream version that make it almost impossible to do. Is there any repo with the Node.js patched v8? Maybe maintained by @isaacs? Seems commits on Node.js with updates on the v8 version are mostly pushed by him... This way I would be able to bisect them to try to find the one that is causing troubles... On the other hand, seems there were changes on Node.js outside v8 to support that particular version upgrade, so I'm not sure if by bisecting v8 alone would be enough... :-/

[jbergstroem]

@piranna bisecting it will most likely be enough. If I was in your position I wouldn't assume that anyone else would do it for me (ref isaacs), so I suggest that you use git bisect to identify which commit caused it, then spend an hour looking at why. It should be pretty quick since you always cut the commit log in half. I'm closing this now; feel free to update us once you have a reproducible result.

[piranna]

I'm not requesting that this get fixed by anybody nor asking for anybody to do nothing, what Iǜe said is that I already did a git bisect on Node.js and found that the problem surfaced when v8 was updated to 3.28.73, and that last week I did another git bisect in this case in v8 itself but I was not able to compile Node.js with it because it had several patched from Node.js team, so what I'm asking about is to know about if there's a repo with the patched v8 so I could do the git bisect over it, no more.

If you want some steps to reproduce my problem, they are easy, just boot a Linux system with the init=/usr/bin/node boot argument. Up to Node.js 0.12.14 this will boot Linux and give you a Node.js REPL, starting from Node.js 0.12.15 (and more exactly since v8 was updated to version 3.28.73) instead of getting a REPL the CPU gets hogged up to 100% indefinitely, and the same happens if instead of /usr/bin/node you use a Node.js script.

[piranna]

And regarding the v8 git bisect: the Node.js v8 3.28.79 is NOT the same as the upstream v8 3.28.79, that's why I ask for a v8 repo with the Node.js patches applied (if that thing exists...).

[jbergstroem]

@piranna: here's the commit history with floating patches for 0.11.15. Assumed you already checked it. https://github.com/nodejs/node/commits/v0.11.15/deps/v8

[piranna]

@piranna: here's the commit history with floating patches for 0.11.15. Assumed you already checked it. https://github.com/nodejs/node/commits/v0.11.15/deps/v8

Yes, I did so, in fact I was using the history up to the failling commit, that's when the already patched v8 3.28.73 was introduced. What I'm asking for if for a repo with the different patched v8 to see what change inserted the problem, as I was requested since that v8 upgrade commit has changes in 1700 files... :-/

In short: this Node.js commit updated v8 and inserted the problem, now I want to now what v8 commit did it, and bisect upstream v8 doesn't work because Node.js don't compile with it because it needs it's own patched one.