codecov exits with status code 4 if NODE_OPTIONS is set

[gclements-chwy]

Describe the bug

The pkg node module has an issue where if NODE_OPTIONS is set to --max-old-space-size=8192 (it doesn't need to be 8192, the issues mentioned below use 4096) the application will exit with status 4. It's literally this line in the pkg code:

https://github.com/vercel/pkg/blob/5.3.3/prelude/bootstrap.js#L1789

No error messages are printed. The app built with pkg just exits. This issue: vercel/pkg#1194 and this one: vercel/pkg#1217 mention the problem with no resolution.

To Reproduce
In a shell:

export NODE_OPTIONS=--max-old-space-size=8192
./codecov
echo $?

The output from the echo will be 4.

Expected behavior

Program just runs

Screenshots

N/A

Additional context

The circleci orb should unset NODE_OPTIONS before invoking codecov

[drazisil-codecov]

@gclements-chwy Thank you!

This explains that exit code, was getting flustered trying to troubleshoot it.

This can happen sometimes with the binary standalone on CircleCI as well. Not sure the fix there.

[drazisil-codecov]

This can happen sometimes with the binary standalone on CircleCI as well. Not sure the fix there.

Oh! 🤦‍♀️ I bet you and the other customer arerunningn the "small" size machine.

https://stackoverflow.com/questions/48387040/how-do-i-determine-the-correct-max-old-space-size-for-node-js

Ok, now the fun part. How to fix this. (talking to myself , mostly) 🤔

[drazisil-codecov]

@gclements-chwy

were you able to reproduce the error going away when this was cleared? It looks like NODE_OPTIONS is empty, by default.

[gclements-chwy]

@drazisil-codecov Yes, 100% an issue with NODE_OPTIONS set, 100% success after calling unset NODE_OPTIONS before invoking ./codecov.

We are running a large machine on circleci.

The fix may be to file a bug report with the pkg team to call delete process.env.NODE_OPTIONS; here: https://github.com/vercel/pkg/blob/main/prelude/bootstrap.js#L1787 before building and running the Script. But I am not familiar with how Script.runInThisContext works internally so it may or may not work.

I mostly filed this bug report so you could add a note to the README about clearing NODE_OPTIONS before invoking codecov and update the orb to clear NODE_OPTIONS too. 😄

[drazisil-codecov]

@gclements-chwy

Got it! We'll see about adding a note (@thomasrockhu-codecov can you update the Orb?)

Given I'm in sorta a deep dive about Node packing and compression and the like already I'll probably do some research on Script.runInThisContext prior to filing a bug. There's probably a reason why the node vm would crash when v8 memory limits are set. 🤷‍♀️

[gclements-chwy]

@drazisil-codecov

If I had to guess, I'd say it's trying to invoke a instance (vm?) with size max-old-space-size inside an instance with size max-old-space-size and it blows up. 💥

Setting NODE_OPTIONS is probably not the issue. It's the max-old-space-size setting that is the problem.

[drazisil-codecov]

Ah, yes. But what's stumping me is how that max-old-space-size is getting set. @mitchell-codecov and I did some deep diving in Node.js's codebase and found that it's always set to something

However, the Node.js memory garbage collection system appears very similar to Java's and is about as hard to understand :D

I ran a stack trace on both "good" and "bad" runs https://gist.github.com/drazisil/a144573590501cd737bac3d436d7cec5

futex(0x7fe9c6207910, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 348580, NULL, FUTEX_BITSET_MATCH_ANY) = -1 EAGAIN (Resource temporarily unavailable)
Is the error that it dies on. However, that can apparently be caused by a value mismatch as well as access issues, so I'm not sure where to go from here. @thomasrockhu-codecov , can you update the orb to remove NODE_OPTIONS prior to running the uploader?

[gclements-chwy]

Oh, we set it. 😄

We set NODE_OPTIONS=--max-old-space-size=8192 as part of our build to give webpack more RAM than the default of 4GB for a node application IIRC. Makes builds a bit faster than glacial, and webpack is less likely to fall over because it ran out of RAM.

[drazisil-codecov]

Well, I feel much less confused now! 🎉

[mitchell-codecov]

Perhaps the uploader could run in a step separate from the one wherein NODE_OPTIONS is set in order to isolate it from the flag while allowing Webpack to get the boost it needs.

By the way, if you're shopping around for a faster bundler, check out Snowpack.

[adolfov]

I'm running into the same issue with travis. We also set NODE_OPTIONS=--max_old_space_size=8192 and we also get exit code 4 with no output when running the uploader.

travis@travis-job-XX:~/build/X/scripts$ ./codecov
travis@travis-job-XX:~/build/X/scripts$ echo $?
4

[adolfov]

I can also confirm that unsetting NODE_OPTIONS fixes the issue in travis.
Should I create a separate issue?

[drazisil-codecov]

I can also confirm that unsetting NODE_OPTIONS fixes the issue in travis. Should I create a separate issue?

You can. I don't know how to fix it on Travis though. Where would you have gone to troubleshoot this that we can document unsetting NODE_OPTIONS, since this is an unfixable issue with node/pkg itself?

[adolfov]

You can. I don't know how to fix it on Travis though. Where would you have gone to troubleshoot this that we can document unsetting NODE_OPTIONS, since this is an unfixable issue with node/pkg itself?

I'll unset NOTE_OPTIONS before calling the uploader. Thank you

[drazisil-codecov]

This is a third-party issue.