Detect require vs. import based on the entry point, not package.json

[Pomax]

Is your feature request related to a problem? Please describe.

Node does not require a package.json file to run, unless you're writing modern import/export code, in which case you suddenly need to define a package.json even if you have no intention of creating a project out of the code you just wrote. You can't even use node --input-type=module because it will --for no reason that makes sense for users-- complain that you Cannot use import statement outside a module, the thing we're literally saying that's what we're doing by using that flag.

Describe the solution you'd like

Node should not need folks to tell it which parsing mode to use: it should scan the entry point it's being asked to run, and simply check whether or not the reserved JS keyword import is found in that file. If it is, done: run in ES module mode without needing folks to create a file that Node should not rely on to do its job.

• node is invoked with a file as run target
• node scans that files for the import keyword
• if it's found, node runs the target in ES module mode
  • any use of require(...) during the run is now a perfectly normal "this function is not defined in this scope" error.
• if it's not found, node runs the target in CJS mode
  • any instance of import during the run is now a perfectly normal "import is a reserved keyword" error.
  • any instance of the dynamic import(...) during the run is now a perfectly normal "this function is not defined in this scope" error.

And now Node does the correct thing, given perfectly normal code as run target. Will that very first step add to the startup time before code actually runs? Sure, but scanning for the import keyword takes on the order of nanoseconds, not milliseconds. We're not scanning the entire dependency tree to see if somewhere down the line we suddenly switch from import to require or vice versa: by default, without a package.json, Node doesn't need to mix and match: as default behaviour Node should run both legacy CJS and modern JS (either/or, not a mix, obviously) without runtime flags or needing files that have nothing to do with Node itself created.

Describe alternatives you've considered

There are no alternatives: Node should not rely on package.json just to run normal modern code, it should do the right thing automatically, with runtime flags and package.json only for folks who need it to do something else (and it should probably never need package.json, that file is so that NPM can do proper package management. Node should never need to consult it just to run normal, valid, modern or legacy, JS)

[aduh95]

Related discussion: https://github.com/nodejs/node/discussions/37857, nodejs/modules#300 (comment), nodejs/modules#296.

• node scans that files for the import keyword

I don't thing that's a reasonable solution: it would introduce a performance penalty on startup time if node has to parse the file for import (dynamic import() is a completely valid thing in CJS, are you suggesting you should remove that? And we would need to know if the import keyword appear inside a string or a comment, so it would be pretty expensive to determine that), and also not all ES module contains import so that would introduce some weird edge cases (AFAIK, there is no known algorithm to tell for sure if a file is written as a module or a script).

There are no alternatives:

You could use loaders to customize which format / parse goal for a given file: https://nodejs.org/docs/latest/api/esm.html#esm_loaders. It's not stable yet though.

Node should never need to consult it just to run normal, valid, modern or legacy, JS

Legacy code is exactly why changing the default behaviour of Node.js is a very difficult thing to do, it would be a problem if older code base stopped working. Note that Node.js looks for package.json only for .js file, if you use .mjs or .cjs files Node.js won't be looking for any package.json.

[Pomax]

dynamic import() is a completely valid thing in CJS, are you suggesting you should remove that

I'm suggesting changing the default behaviour, so if Node should allow mixed import/require when there's a package.json, then perfect: that is not at issue, I'm all for that. But Node should also do the right thing without a package.json.

it would introduce a performance penalty on startup time if node has to parse the file

Correct. And that would be very little time because it only has to scan one file, plus (and this is the big one): if you use a package.json, Node doesn't need to scan anything, it already knows which mode to run in, so it would do exactly what it's already doing right now, and no one following current necessary practices will be impacted. Only folks who invoke Node without a package.json would run into this performance hit, although I struggle to call it that: it'd be insignificant compared to the rest of Node's initialisation.

You could use loaders to customize which format / parse goal for a given file

While I see a lot of potential for loaders, they are antithetical to why I filed this issue: Node should do the "obvious" right thing out of the box (i.e. if asked to run a file, it should run that file, either in cjs mode, or esm mode, automatically. Not first be configured to do the right thing through runtime flags, or config files, or config code).

Note that Node.js looks for package.json only for .js file, if you use .mjs or .cjs files Node.js won't be looking for any package.json

Sure, but again: that's not doing the right thing, that's forcing people to use nonstandard extensions as workaround. Node should (and this is a long term "should", not "it must, now!") do the right thing when given a .js file, based on the content of that .js file (and also, only that .js file. There is no need to resolve the full tree, Node can do the "obvious" thing based on the fact that people who use mixed import/require will almost certainly be using a package.json already to make sure things work properly).

Legacy code is exactly why changing the default behaviour of Node.js is a very difficult thing to do

No it isn't? That's what major versions are for. I have no expectation of this getting changed in the current version, but I do kind of expect this to be seriously considered for Node 17 or 18. The way Node works as of v16, especially as LTS, should stay exactly what it is now, let's definitely not change that. People rely on it to work a specific way. But 17 or 18 are fair game.

[aduh95]

Hum so you'd like to see this behavior only for the entry point 🤔 In this case, I have more questions, how would you define parse goal for dependency of such modules:

// entry-point.js
import './submodule.js'; // is this ESM no matter its content?
import './submodule.cjs'; // is this CJS?

import 'some_package'; // can this be CJS?

// other-entry-point.js
require('./entry-point.js'); // is this CJS?

// another-entry-point.js

// no import, no export, no require, is this ESM or CJS?

console.log(this); // is this undefined?

var obj2 = { get x() { return 17; } };
obj2.x = 5; // should this throw?

I'm suggesting changing the default behaviour, so if Node should allow mixed import/require when there's a package.json, then perfect: that is not at issue, I'm all for that. But Node should also do the right thing without a package.json.

Why would the "right" thing to remove dynamic imports for CJS? Are they causing an issue, or is this just for convenience to tell if a file is CJS or MJS?

While I see a lot of potential for loaders, they are antithetical to why I filed this issue: Node should do the "obvious" right thing out of the box (i.e. if asked to run a file, it should run that file, either in cjs mode, or esm mode, automatically. Not first be configured to do the right thing through runtime flags, or config files, or config code).

Note that what is "obvious" to you may not be for everyone. While I understand loaders are not the long term solution you'd like, it can still be useful for you (or someone else) to build a POC.

[devsnek]

this is not a novel topic of discussion and I think the general sentiment is that this would be cool if someone builds something, but no one has built something because it's a very complex problem, and you have to make a lot of subjective calls which people disagree on.

[Pomax]

Good questions. In the absence of a package.json, keep it simple, and keep it naive:

// entry-point.js
import './submodule.js'; // is this ESM no matter its content?
import './submodule.cjs'; // is this CJS?

import 'some_package'; // can this be CJS?

would result in going "that's a normal import on line 1, stop checking and use ESM parsing" and then when it runs the file it tries to import submodule.cjs using ESM rules and it'll throw an error, and that's fine. Use a package.json if you need to mix modalities.

// other-entry-point.js
require('./entry-point.js'); // is this CJS?

Yes, it is, because the first import mechanism encountered uses require, not import.

// another-entry-point.js
// no import, no export, no require, is this ESM or CJS?
console.log(this); // is this undefined?

This is Node's default mode. If that's still CJS by v17 or v18, then it's CJS, unless Node's finally ready to switch to ESM parsing by default, then it's ESM.

Why would the "right" thing to remove dynamic imports for CJS? Are they causing an issue, or is this just for convenience to tell if a file is CJS or MJS?

Who said anything about removing dynamic imports? If the file starts as a normal, modern, plain, JS file (that is it starts with a bunch of imports of other files that end in .js) then it picks ESM parsing, and rolls with that, and if there are dynamic imports, no problem. Those work fine. If it starts with a bunch of requires then pick CJS parsing, and roll with that instead, and then if there are dynamic imports that are valid under the CJS model, then also no problem.

This feature request is entirely about what Node does when asked to run "plain files", in the absence of a package.json. That is, we're not dealing with a project, we're not dealing with external dependencies, someone just wrote a bunch of plain JS files using Node's API and they want to run them, they just happen to be more than a single file because the person doing the writing was taught proper code house keeping and sticks to that. Node should be able to run their code without anything other than "being run": in this scenario, Node can pick the correct parsing mode 100% of the time in these cases, because the examples you're showing just don't factor into that use-case.

This really is a matter of "if the entry file starts as a normal JS file with import, use ESM, if it starts with require, use CJS, and we're done". Any "oh hey you actually did something unusual" code like in those examples should, quite reasonably, halt with an error telling folks that they're mix-and-matching, and to use a package.json (or a runtime flag that explicitly tells Node which parser to use, which would most certainly be worth adding to 16, as that would be purely new functionality and not break any preexisting code).

Note that what is "obvious" to you may not be for everyone.

Hence the quotes, I'm talking about the kind of obvious that exists when looking in from the outside: "I have some JS files, I've written them following current standards, running node firstfile.js should works". The kind of expectation that someone who is new to Node would have.

[aduh95]

Thanks for the clarification, I think this makes a lot of sense; first we'd need to find or write an algorithm that can tell if a JS file is ESM or CJS, is this something you'd be interested in contributing?

Related (vaguely): TC39 proposal "use module";.

in the absence of a package.json [...], we're not dealing with a project, we're not dealing with external dependencies

If you allow me to be pedantic, you can have external deps without a package.json – Node.js doesn't use package.json to resolve local deps and there is an open PR to add support for HTTPS imports :)

[targos]

we'd need to find or write an algorithm that can tell if a JS file is ESM or CJS

Something that goes in this direction is being done in #39175

[Pomax]

If you allow me to be pedantic, you can have external deps without a package.json

Ah, yes that's true, good point.

Something that goes in this direction is being done in #39175

Nice!

we'd need to find or write an algorithm that can tell if a JS file is ESM or CJS, is this something you'd be interested in contributing?

While I wouldn't mind giving it a shot, given the number of projects I'm already involved with any promise to try to get to that soon would pretty much be a lie, I'm basically booked up for projects for at least a year, filing issues and discussing whether there's merit to it is the most I can do for larger projects for the foreseeable future =(

[aduh95]

#39508 didn't get much traction, and there was at least two outstanding objections for not implementing this proposal.

[Pomax]

These arguments seem to apply on the idea that "this makes it easier for beginners". That's an argument I don't buy into. Changing the behaviour as suggested in this issue (making Node use ESM parsing mode if the entry file and only the entry file is ESM) makes Node better as a tool. Tools should have sensible default behaviours, and a way to override any and all of that behaviour by specifying an explicit configuration (either throught runtime flags, or a config file, which for Node is the package.json file).

So if we're talking about sensible default behaviour: if someone writes plain, modern JS, which because of the JS spec can only be ES module based code (because that's what TC39 decided is the only import mechanism that spec-compliant JS can use) then Node should be able to execute that file, because Node's job is to run JS, which means at the very least it should run spec-compliant JS, even if it can also run its own flavour of JS. And if it's asked to run legacy CJS code, it should run that without any complains too, because it's been doing that for about a decade now; that should keep working. And if someone writes code and specifies a the configuration file that Node looks at, then obviously now Node should do whatever it does based on what's in that configuration file. This is basic tool design: if there's a config, that config kicks in.

And yes: that means that ESM code that works "on its own" suddenly won't run anymore when you use npm init to create a package.json file, because npm right now does not include the type field. That's npm's failing (and if there isn't an issue for this on their side already, I will be quite surprised), not Node's. In fact, that's exactly what you want: there is no problem here, Node will tell you that it can't run ESM unless you add "type": "module" to your package file, you follow Node's instruction and update package.json, and immediately your code works again.

"What if your ESM code has imports that somewhere down the line switchi to CJS style requires?"
Literally everyone doing that today already has a package.json file, because their code won't run if they don't. This is a non-issue, their code keeps working exactly as it already works right now.

"What if someone mixes ESM and CJS in their own (flat file) code?"
Then someone should tell them to fix that, because that's just nonsense. And that someone is the Node cli when it's asked to execute a script. It'll throw you an error right now if you try this, and it should keep throwing you that error.

"What if their ESM imports a Node API, which uses CJS?"
This would be the only valid argument against, if it weren't for the fact that Node can already expose its own API in ESM context just fine, making this a non-issue, too.

[bmeck]

CC: @nodejs/modules

I think we should close this as a duplicate of many older issues and discussions unless something novel comes up.

Per:

Something that goes in this direction is being done in #39175

This is only a heuristic to help debugging, it isn't 100% reliable or performant nor is it expected to become so to my knowledge.