[rfc] module: change load order

[bnoordhuis]

Consider the following:

$ mkdir -p node_modules/foo
$ echo 'module.exports = 0' > node_modules/foo.js
$ echo 'module.exports = 1' > node_modules/foo/index.js
$ echo '{"name":"foo"}' > node_modules/foo/package.json
$ node -p 'require("foo")'
0

In other words, node_modules/foo.js takes precedence over node_modules/foo/index.js. I have trouble understanding when this would ever be the desired behavior inside a node_modules/ directory.

Even more awkward, adding a package.json with a main property suddenly turns the precedence around:

$ echo '{"name":"foo","main":"./index.js"}' > node_modules/foo/package.json
$ node -p 'require("foo")'
1

I propose to harmonize this behavior and make node_modules/foo/index.js always take precedence over node_modules/foo.js.

If nothing else, the current behavior is woefully inefficient because of all the superfluous stat(2) calls:

stat("/app/node_modules/express", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
open("/app/node_modules/express/package.json", O_RDONLY|O_CLOEXEC) = 10
stat("/app/node_modules/express.js", 0x7ffe669cde70) = -1 ENOENT (No such file or directory)
stat("/app/node_modules/express.json", 0x7ffe669cde70) = -1 ENOENT (No such file or directory)
stat("/app/node_modules/express.node", 0x7ffe669cde70) = -1 ENOENT (No such file or directory)
stat("/app/node_modules/express/index.js", {st_mode=S_IFREG|0664, st_size=224, ...}) = 0
open("/app/node_modules/express/index.js", O_RDONLY|O_CLOEXEC) = 10

On my system, with a cold disk cache (echo 3 > /proc/sys/vm/drop_caches), this moderately large benchmark application:

• Takes about 1000 ms to start up, of which
• 450 ms is system time, of which
• 300 ms is spent on stat(2), of which
• 200 ms is wasted

Ergo, we could shave off 20% of the absolute start-up time and 45% of the total system time.

/cc @sam-github

[bnoordhuis]

The current behavior appears to be at odds with the pseudo-code algorithm from the documentation, that clearly says that foo.js should always be loaded: first LOAD_AS_FILE, and only then LOAD_AS_DIRECTORY.

It seems to be a regression in v0.12. v0.10 always prints '0', v0.12 and newer print either '1' or '0', depending on the value of the main property in package.json.

I guess I'll cook up a PR that restores the v0.10 behavior. I'm not wildly thrilled by the inefficiency of the algorithm but at least it's consistent.

[kzc]

If something as fundamental as module load order is going to be changed after many official releases, then I think it would be a good opportunity to address "#3402: Don't resolve symlinks when requiring" at the same time.

[zeusdeux]

If the regression has existed since 0.12 and there haven't been complaints then that means that the no., of people using the node_modules/foo.js use case is very low. In that case, I would definitely prefer the new consistent behaviour that you propose over the one seen in 0.10.

[cjihrig]

This is probably fine, since the inconsistency being described assumes that people are modifying node_modules by hand, instead of letting npm do what npm does. It's always a little unsettling to modify the module system though.

[sam-github]

Should definitely be consistent. I like the idea of optimizing it... I can't think of anybody using the setup described, where a package and a module exist side-by-side. The fact that we've already "betaed" the new behaviour by accident since 0.12 makes me think it should be proposed for the next major.

[jasnell]

@bnoordhuis ... still something you wanna do?

[bnoordhuis]

Yes, it is, but the question is: do we restore the v0.10 behavior or go with the efficient approach? I think the consensus is that it should be the latter but it wouldn't hurt for @nodejs/ctc to chime in.

[mscdex]

Efficiency! Is what is being proposed here a separate change from say #5689?

[bnoordhuis]

They're not quite the same. Issue #5689 is about require('./file') while this one is about require('file') (and also that having a main attribute in package.json changes the load order.)

[cjihrig]

+1 to making the change. Should we be using GitHub reactions for this?

[mscdex]

-1 to GitHub reactions

[jasnell]

I think I'm +1 on this change also.

[dlongley]

+1 for the more efficient approach, but also +1 to addressing other important module loading problems at the same time, if possible: #3402

[jbergstroem]

I'm +1 to changing the order.

[SystemParadox]

Oh yes please can we get #3402 fixed as part of this

[jasnell]

@bnoordhuis ... any reason to keep this open?

[bnoordhuis]

I still have this in my backlog but if we decide not to make the change, the documentation would need to be updated.

[bnoordhuis]

I don't think I'm going to get around to this so I'll go close it out.