esm: The `getPackageType` utility function is not exposed

[DerekNonGeneric]

Is your feature request related to a problem? Please describe.
Assist in determining module format by using internal machinery.

Describe the solution you'd like
Expose the getPackageType utility function.

Describe alternatives you've considered
The alternative is what I am currently doing, which is passing the --expose-internals flag (no bueno) and acquiring the utility function via internalBinding('module_wrap'). Exposing it would allow shaving off one dangerous flag and five lines from the following example.

// Flags: --expose-internals --experimental-modules --experimental-loader

import { createRequire /*, getPackageType*/ } from 'module';
import { dirname as pathDirname } from 'path';
import { fileURLToPath as pathFromFileUrl } from 'url';

const __filename = pathFromFileUrl(import.meta.url);
const __dirname = pathDirname(__filename);
const require = createRequire(__dirname);
const { internalBinding } = require('internal/test/binding');
const { getPackageType } = internalBinding('module_wrap');

// const TYPE_NONE = 0;
// const TYPE_COMMONJS = 1;
const TYPE_MODULE = 2;

/**
 * @param {string} specifier
 * @returns {Promise<{url: string, format: string}>}
 */
export async function resolve(specifier) {
  const type = getPackageType(specifier);
  if (type === TYPE_MODULE) {
    return {
      url: specifier,
      format: 'module',
    };
  }
}

As you may have noticed, one use-case is for custom loaders, which necessitate this info. Amongst others, one advantage of using the internal machinery (over writing it in JS) is that it is implemented in C++, which is important because this particular algorithm can be very expensive to perform and ( ! ) resolve hooks are on the hot code path of every module.

Additionally, exposing this function may also improve the viability of using loaders to fill the current gap in format autodetection until the --es-module-resolution flag's proposed auto mode is ready.

cc @nodejs/modules-active-members
bcc @GeoffreyBooth (please cc above)

[GeoffreyBooth]

@nodejs/modules-active-members

[devsnek]

I'm against exposing machinery like this. We are planning to expose functions to let you call up to the previous loader.

_{I think this is also a really good example of why we shouldn't be adding new fields to package.json but oh well}

[guybedford]

Package type does not imply the module format, but rather the module format database, so note that the usage example is wrong as it doesn't support eg .cjs.

The source of truth to follow for determining the module format of a file is the specification for the resolver (https://nodejs.org/dist/latest-v13.x/docs/api/esm.html#esm_resolver_algorithm), under the function ESM_FORMAT.

Exposing or having userland implementations of the spec would be very useful indeed. If Node core were to expose something here it should likely be a JS esmFormat implementation. Then how to handle caching and whether it should be sync or async come up. Both have pros and cons. As you can see that is not necessarily trivial from a development and consensus perspective. Contributors to this process welcome! Userland work would likely be a great starting point.

[GeoffreyBooth]

Does Node already do the package.json lookup before this loader is called? Presumably it must have, I would think, to know how to reach this file (such as if it were referenced via "exports"). If so, it would both improve performance and solve this particular need if Node could pass along the package.json type (or all package.json metadata) to the loader, so that the loader doesn’t need to look the data up again.

[bmeck]

I think exposing something is likely useful here since it is possible to sniff / can be looked up on disk and there isn't a clear workaround to make it easy to deal with. The main contention here for myself is what to expose; as @guybedford points out, just exposing the "type" might not be the best choice as it doesn't really expose the underlying mechanisms that the runtime is using when that is set and could lead to misunderstanding and misuse.

[GeoffreyBooth]

To give a concrete use case, I’m considering creating a CoffeeScript loader. So for a given file.coffee that this loader would process, I need to know the package type the same as if the file was file.js (since for CoffeeScript, we can’t store the parse goal metadata in the file extension and therefore I need to know the nearest parent package.json "type").

So for me, I’d want to know either what Node would determine the package scope type to be if my file were a .js file; or I’d want Node to provide to the loader the contents of the controlling package.json file.

[devsnek]

I feel like this points to an inherent problem that tooling needs to know way too much about the specific details of node's esm resolution system. In core we've been really careful to avoid adding new stuff like that to the cjs loader because of the burden on the ecosystem to support it and the node maintainers to keep it working for the ecosystem.

even if we expose all the needed APIs, that doesn't guarantee tooling uses them, or uses them at the correct times (matching the behaviour of bugs is important too).

[bmeck]

@GeoffreyBooth that sounds like you need to know the format of a file, not the package type.

[GeoffreyBooth]

@GeoffreyBooth that sounds like you need to know the format of a file, not the package type.

What do you mean? The format is CoffeeScript.

tooling needs to know way too much

Well, why does the loader API need to be given the type? Ideally that would be optional, and if it’s unspecified the regular Node logic would apply; and unknown types (e.g. '.coffee') could be treated the same as .js, or I would have some way of telling Node to treat them like .js. And by “treat as .js” I mean, use the same logic that Node’s ESM loader uses to determine whether to treat .js as CommonJS or as ESM.

[bmeck]

@GeoffreyBooth

I’d want to know either what Node would determine the package scope type to be if my file were a .js file; or I’d want Node to provide to the loader the contents of the controlling package.json file.

As you stated, you want to know what format a .js file is, not specifically the string value of the "type" field. Anyways, "package scope type" isn't done per file; a package has a set of formats mapping to file extensions within its boundaries. So, the package scope type of a file doesn't really make sense. You can see what format a package scope would treat a file to be though.

[guybedford]

Here is an implementation for getPackageType:

function getPackageType (checkPath) {
  const rootSeparatorIndex = checkPath.indexOf(path.sep);
  let separatorIndex;
  while (
    (separatorIndex = checkPath.lastIndexOf(path.sep)) > rootSeparatorIndex
  ) {
    checkPath = checkPath.slice(0, separatorIndex);
    if (checkPath.endsWith(path.sep + 'node_modules'))
      return 'commonjs';
    try {
      const pjson = JSON.parse(fs.readFileSync(checkPath));
      return pjson.type || 'commonjs';
    }
    catch (e) {
      if (e.code === 'ENOENT') continue;
      throw e;
    }
  }
  return 'commonjs';
}

This follows the specification (which is fully provided in the Node.js documentation exactly for this reason).

Because it is specified and relatively stable, there is nothing wrong with copy-pasting such a snippet. DRY does not apply when there are lots of decisions like API / async or async / caching concerns. Copy paste is good for this stuff. Or a userland packages with this code would be useful. Please be useful.

[GeoffreyBooth]

Thanks @guybedford, that certainly solves the need; though if this needs to happen for many/most loaders we might want to consider providing this as a utility, or reusing Node’s lookup rather than having this happen twice for each file.

This also makes me think of nodejs/modules#436; I wasn’t aware we treated a node_modules folder as a special case. It shouldn’t need to be, as every subfolder of node_modules should have a package.json; and we’re not really documenting this in the main docs, just in the resolver spec. I get that this is probably to cover some weird edge cases of files in node_modules that aren’t in subfolders, or something like that, but I wonder what the harm would be in removing this special casing.

[devsnek]

folders within node_modules aren't guaranteed to have package.json, that's just a side effect of package managers. it's really important to highlight that prior to our esm loader, package.json didn't infer a "boundary" to node.