Type Stripping in `node_modules/*/*.ts`

[karlhorky]

Copied from original discussion in nodejs/loaders#217 (comment)

---

I know that as a user I won't have much sway here, but I think I'm not alone in thinking that files in node_modules/*/*.ts should also be allowed --experimental-strip-types type stripping.

A few thoughts:

1) Node.js Alone Won't Prevent TS npm Publishing

There is a lot of movement in the "execute TypeScript, don't build it" ecosystem already, for example with the advent of many ecosystem projects such as:

• TypeScript execution CLIs like ts-node, tsm, tsx
• TypeScript-native runtimes like Bun and Deno
• modern TS-native registries like JSR
• somewhat related: ECMAScript proposal Type Annotations

Publishing .ts files to npm and other registries is already a growing pattern, which drastically reduces the effort and complexity of publishing a TypeScript package.

I think as these projects evolve and more people get to know about this possibility, there will be a greatly increased interest in this possibility.

It's my opinion that this banning of node_modules/*/*.ts in Node.js alone won't be able to hold back the energy in the ecosystem, especially with working alternatives. (not to mention users circumventing this by using any of the other projects, which do not have this limitation)

2) User Experience

My first experience with node --experimental-strip-types was creating an internal checking tool which was its own package, without a build step.

I immediately fell flat on my face with this, because of the banning of node_modules/*/*.ts files.

Even though I was already somewhat aware of the discussions and objections behind this (eg. the nodejs/TSC meeting notes from 2024-07-24), for my use case it felt like an arbitrary limitation imposed on my project.

I imagine that as type stripping becomes more widespread, this will be a common complaint from users.

3) Node.js Features Available Everywhere

@GeoffreyBooth wrote about supporting a Node.js feature everywhere:

If we support a file type, we support it everywhere. Just as module detection needs to apply within node_modules so that a package that works locally continues to work when published to npm, the same applies to TypeScript. We shouldn't be limiting our functionality to try to encourage best practices.

This resonates with me - it seems like supporting a Node.js feature in as many places as possible will be the best for users and cause the least amount of support complaints.

Banning node_modules/*/*.ts to try to avoid a pattern which the ecosystem may adopt seems like a weak reason. (not to mention that the ecosystem may end up adopting this pattern anyway by finding cowpaths around this)

4) Identifying and Addressing Downsides

Downsides as mentioned by Ryan and Daniel on the TypeScript team and vocal node_modules/*/*.ts opponents such as Matteo are important and should be considered.

But it feels like also:

1. Each problem should be carefully thought through, examined, clearly enumerated and documented (also asked for by Geoffrey)
2. Once clearly enumerated and documented, solutions should be also thought through, with each solution's tradeoffs being also enumerated and documented

This being done with a dispassionate, neutral stance, not trying to prove any one viewpoint.

In reading through the publicly-available documents, I don't think this has been done yet.

5) At Least Allow Users to Choose

Even if careful enumeration of problems and solutions leads to the final conclusion that the tradeoff is not worth it for Node.js, I think it would be good to allow users to choose for themselves and disable this node_modules/*/*.ts banning with a flag, eg:

node --experimental-strip-types-in-dependencies

6) Similarities to .ts Rewriting Limitation

Oh one more thing, more related of a feeling than fact-based:

This node_modules/*/*.ts banning feels like it has a similar vibe to the .ts -> .js rewriting decision in TypeScript, which was overturned after years of doubling-down on .js imports (ironically heavily inspired by Node.js type stripping).

[marco-ippolito]

Adding a flag to allow type stripping in node_modules is trivial.
For a real solution the issue is how does node recognize if its published npm package or a monorepo type project?
If there was a marker in the package.json that would be possible.
Immagine an entry in the package.json "local_node_modules": true that prevented publishing to npm (when you try npm publish it would throw) but if node finds it, it allows type stripping even in the node_modules.

[JakobJingleheimer]

If there was a marker in package.json that would be possible.

Isn't there? The namespace

---

I'm in favour of allowing publication of ts in general (as I'm also in favour of publishing jsx) because it allows the consumer to decide the output and avoids make-work steps and a game of telephone.

[aduh95]

At Least Allow Users to Choose

I think that's possible, with loader hooks, Marco have been working on exposing the type stripping utilities for this exact use-case IIRC.

[karlhorky]

how does node recognize if its published npm package or a monorepo type project?

@marco-ippolito I think maybe first needs to be identified:

1. What is the purpose of banning published npm packages with .ts? (What are all of the enumerated downsides and their symptoms?)
2. Should these downsides / banning also apply to packages published to other registries? (eg. private registries or TS-native registries?) If yes/no, why?
3. What are possible solutions or improvements to those downsides?

It seems like trying to ban users from publishing .ts files is maybe the wrong thing to be putting effort into?

Maybe better would be 4) Identifying and Addressing Downsides, listing out the real downsides which occur as a result of publishing .ts files, and then try to work on those problems / symptoms?

[JakobJingleheimer]

It is possible with a loader hook: https://github.com/jakobjingleheimer/nodejs-loaders#jsx--tsx

(This is about to be split into a monorepo and published separately as @nodejs-loaders/tsx)

[marco-ippolito]

Btw as @aduh95 mentioned, if parser.stripTypeScriptTypes nodejs/node#55282 lands, its possible to use an external loader that supports type_stripping in node_modules with the same output as node. amaro (should) already cover this use case when used as a loader

[mcollina]

I agree that we should document the downsides of not supporting .ts files inside downloaded npm packages are not documented, and it would be great if @DanielRosenwasser and the rest of the typescript team voiced their concerns. They are the same as mine, just expressed in a much more precise form. I could take a stab at writing those concerns if they prefer (for them to review). This is important.

Summing them up are:

1. TypeScript is designed so that only .d.ts files are shared across projects. They are interface files.
2. Assuming a significant portion of the npm ecosystem will be replaces by ts-only modules, it would slow down massively our IDEs, because the numbers of parsed files would increase (10-100x?).
3. Allowing only-TS dependencies will force a single tsconfig.json in the ecosystem.

[hverlin]

modern TS-native registries like JSR

jsr allows ts files, but has some restrictions (https://jsr.io/docs/about-slow-types#typescript-restrictions).
Following them might help to address the performance issues. (There is also the isolatedDeclarations option)

[jakebailey]

My first experience with node --experimental-strip-types was creating an internal checking tool which was its own package, without a build step.

I immediately fell flat on my face with this, because of the banning of node_modules/*/*.ts files.

Theoretically, this shouldn't happen for plain monorepo use; the determination of "is in node_modules" considers the realpath of a file. But if you aren't using a monorepo tool with symlinks, I can see how this would fail.

Publishing .ts files to npm and other registries is already a growing pattern, which drastically reduces the effort and complexity of publishing a TypeScript package.

The reduction in complexity here is from the tooling, though; something like jsr handles the downleveling and d.ts emit for you behind the scenes. Just publishing TS code itself doesn't reduce much complexity in a world which really does need to have/run .js files... (I think it has the potential to actually increase complexity, without some careful consideration.)

jsr allows ts files, but has some restrictions (jsr.io/docs/about-slow-types#typescript-restrictions).
Following them might help to address the performance issues. (There is also the isolatedDeclarations option)

These kinds of restrictions do not help published TS source; TypeScript will load all of the source code given to it, including function/class bodies, expressions, etc. Publishing and loading .d.ts files strips away all of those interior bits and allows TypeScript to do less work.

"Slow types" is a bit of a misnomer becuase it's talking about "slow" in terms of generating declaration files from the perspective that the "slow types" require running tsc to typecheck. isolatedDeclarations encodes a list of rules that prevent needing type checking, and is therefore "fast", but the speed at which declaration files are emitted does not change the fact that loading d.ts files is faster than actual source code. It'd be nice if JSR's docs were modified to not call them "slow types" anymore.

There are other concerns here as well:

• If users don't set --skipLibCheck (which, they arguably shouldn't in order to understand when their deps are broken), then they're going to be checking all of that extra source code too, including fully-annotated code.
• Additionally, a package's tsconfig may not have the same settings as the user; depending on the differences, new errors may appear that wouldn't be there if .d.ts files are consumed instead (e.g., exactOptionalPropertyTypes, noUncheckedIndexAccess, etc, in the consuming package). This is also why transforming someone else's TS code is dangerous; there are options that some other package may have set which would affect emit, e.g. decorator use (including uses that are not exported!), useDefineForClassFields, etc.
• Publishing TypeScript source may limit downstream users abiliy to not use the same TS version as you. Today, one runs tsc (or similar), leaving behind only JS (not read by tsc), and .d.ts without anything from expresion space. This means that TS is free to introduce a feature like satisfies and it'll be backward compatible for users of your package as the satisfies keyword does not appear in d.ts files. If you publish source, the version of TS you use to typecheck your code impacts downstream users.
  • This is technically no different than adding ES features, except that basically all TS syntax is erasable.
  • TS may also eventually remove something or be forced to change the emit of something altogether, e.g. to deal with enum, a new module keyword in JS, and so on.

All of these problems are handled by transforming your code before publishing it.

(My list is not exhaustive either; a proper combined list would of course be good.)

[marco-ippolito]

What if we supported running in node_modules only if the package.json containts the "private": true property https://docs.npmjs.com/cli/v10/configuring-npm/package-json#private which prevents a package being published.
This way you still use monorepos but wont disrupt the ecosystem

[karlhorky]

Downsides of Publishing .ts to npm

So if I may try to make a simplified list of problems:

1. Loading .ts source code is slower than loading d.ts files
2. Checking .ts source code in node_modules without --skipLibCheck is slow
3. Publishing newer / different TS syntax than the consumer tsconfig.json / TS version causes failures (similar to publishing newer ES syntax than consumer supports)

(happy to edit this or for someone to take over with a new list, just trying to make a simplified list)

Metrics

It would be cool to get some symptoms (metrics) on these things.

For example, with a small sized project that has 10 direct dependencies (and a number of transitive dependencies), and considering that 100% of the dependency graph converted to TypeScript, what would the concrete impacts be in latest VS Code on a 2020 mid-range computer?

By metrics I mean something like these fake numbers:

1. Loading .ts source code is slower than loading d.ts files
   • VS Code startup time: +1s (+80%)
   • TypeScript extension startup time: +2s (+20%)
   • RAM: +200MB (+10%)
   • Auto-completion response time: +0.5s (+15%)
2. Checking .ts source code without --skipLibCheck is slow
   • Full project build time: +2s (+15%)
   • Type checking for 10 dependencies: +1s (+10%)
   • RAM usage during type checking: +50MB (+5%)
   • Incremental builds time increase: +0.5s (+5%)

Update: @andrewbranch did some research and produced some similar metrics here:

.d.ts / .ts

Files:           268 / 317
LOC:             49k / 62k
Instantiations:  11k / 46k ❗
Memory:          92 kB / 118 kB
Parse time:      .76 s  / .91 s
Check time:      1.48 s / 3.02 s ❗
Total time:      2.58 s / 4.43 s ❗

• https://x.com/atcb/status/1705675339626909911

[karlhorky]

With clear metrics, it will be more clear what the tradeoffs are and what could be solved, optimized or worked around to minimize those downsides.

Some projects may decide that the tradeoff is worth it:

• 👎 decreased performance, compat considerations
• 👍 no build setup, configuration, maintenance, CI step