Slow compilation issues (40-50s) - --skipLibCheck not working

[NoPhaseNoKill]

TypeScript Version: 3.8.3

Search Terms: --skipLibCheck slow compilation tsc

Code

Running tsc --listFiles --generateCpuProfile profile --skipLibCheck from the command line results in having the following output - which appears to be slowing compilation significantly. It appears as though it's including multiple .d.ts files from my @types node_modules in the compilation. What sort of time should I be expecting this amount of files to compile in? Any help reducing the compilation time would also be greatly appreciated.

/home/user1/projects/projectName/node_modules/@types/semantic-ui-form/global.d.ts
/home/user1/projects/projectName/node_modules/@types/jquery/JQueryStatic.d.ts
/home/user1/projects/projectName/node_modules/@types/moment-timezone/moment-timezone.d.ts
.....etc

Extended diagnostics are:
Files:                         3022
Lines:                       410050
Nodes:                      1604064
Identifiers:                 555657
Symbols:                     804555
Types:                       254555
Memory used:               1220647K
Assignability cache size:     95587
Identity cache size:          13620
Subtype cache size:         4718477
Strict subtype cache size:     6378
I/O Read time:                0.10s
Parse time:                   1.78s
Program time:                 3.22s
Bind time:                    1.45s
Check time:                  39.98s
transformTime time:           1.39s
commentTime time:             0.56s
I/O Write time:               0.20s
printTime time:               5.02s
Emit time:                    5.02s
Total time:                  49.68s
 

my tsconfig is as follows:

{
  "compilerOptions": {
    "outDir": "./dist/",
    "extendedDiagnostics": true,
    "baseUrl": "src/",
    "module": "commonjs",
    "target": "es6",
    "jsx": "react",
    "allowJs": true,
    "allowSyntheticDefaultImports": true,
    "moduleResolution": "node",
    "esModuleInterop": true,
    "strict": true,
    "experimentalDecorators": true,
    "skipLibCheck": true,
    "lib": [
      "dom",
      "es2015",
      "scripthost",
      "es6",
      "es2016",
      "es2017"
    ]
  },
  "include": [
    "./src/**/*.ts",
    "./src/**/*.tsx"
  ],
  "exclude": [
    "node_modules",
    ".idea",
    ".storybook",
    ".git",
    "dist"
  ]
}

with my project folder structure being:

├── src
│   ├── folder1/....
│   └── folder2/.....
├── node_modules
│   ├── shouldNotBeCompiled1.d.ts
│   ├── shouldNotBeCompiled2.d.ts
│   └── shouldNotBeCompiled3.d.ts
├── dist

Expected behavior:
Should not type-check any of the files inside node_modules during compilation
Actual behavior:
Appears to be including node_modules/@types during compilation - which is causing a compilation time of ~50s for project.

Related Issues:
#37635
#36103

[NoPhaseNoKill]

@amcasey tagging you both @RyanCavanaugh in this as I saw you from other threads with similar/slow compilation issues

[MartinJohns]

You could try updating your TypeScript version: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-9.html#speed-improvements

[NoPhaseNoKill]

Upgrading to 3.9.7 results in basically the same speed:

Files:                         2698
Lines:                       380456
Nodes:                      1438657
Identifiers:                 499827
Symbols:                     772615
Types:                       237792
Instantiations:              763693
Memory used:               1705486K
Assignability cache size:     85249
Identity cache size:          13293
Subtype cache size:         9399787
Strict subtype cache size:     4043
I/O Read time:                0.12s
Parse time:                   1.58s
ResolveTypeReference time:    0.04s
ResolveModule time:           0.93s
Program time:                 2.96s
Bind time:                    1.83s
Check time:                  35.41s
transformTime time:           1.51s
commentTime time:             0.46s
I/O Write time:               0.14s
printTime time:               4.80s
Emit time:                    4.80s
Total time:                  44.99s

[NoPhaseNoKill]

As an FYI, if you need more information please let me know and I'll be happy to provide. The only problem is that I cannot post the project, as it's a private/institutional repo. So if there is any way of providing obfuscated data about the project that will help troubleshooting please let me know

[amcasey]

@NoPhaseNoKill I wrote up some instructions for how to gather a trace that will identify which parts of your program are taking the longest to compile: https://github.com/microsoft/TypeScript/wiki/Performance-Tracing

Ideally, you'd share the resulting logs, but one of our goals in adding this functionality was to be able to work with users who can't share source. I'm hoping it will be enough for you to identify the problem and either fix it on your end or provide us a reduced repro (e.g. calling the same npm module that's giving you problems). It's still a pretty new tool (and doc), so ping me if you get stuck and I'll do what I can to unblock you (except tomorrow, when I'll be away). Thanks!

[amcasey]

Anecdotally, from your huge subtype cache, I'm guessing you have a huge array of object literals somewhere (the trace should give you a file and position). If that's the case, my usual workaround is to add as any as YOUR_ARRAY_ELEMENT_TYPE to the first element.

I suspect the node_modules .d.ts files are a red herring. The short explanation is that the parts consumed by the code you're compiling will be checked (because the compiler needs to know about it to check your code) and the others won't. On their own, .d.ts files rarely contribute to check time - the cost is nearly always at the use-site.

[amcasey]

One last point: I didn't see anything in your comment about your findings from generateCpuProfile. If you looked through it, I'd be interested to know what the hotspots were. If you didn't (personally, I find them very difficult to read), you might want to skip that flag because it will just slow down your run.

[NoPhaseNoKill]

Yeah I didn't end up reading the CPU profile. Just currently running the performance tracing atm - so will get back to you ASAP.

What would an example of this be?

If that's the case, my usual workaround is to add as any as YOUR_ARRAY_ELEMENT_TYPE to the first element.

[NoPhaseNoKill]

When trying to open the trace.json file - nothing is displaying (even though it's got the fact it's opened that file). Any ideas?

edit: The trace file is like 1.5GB - got a feeling it's maybe crashing/too big?

[NoPhaseNoKill]

I'm more than happy to send the trace.json - as it doesn't include anything specific about the source code. How would you like me to upload this?

edit: I believe I can share these via OneDrive. Standby for links upon uploading

[NoPhaseNoKill]

There's actually a few pieces of data in here which the company considers 'sensitive'. So I'm going to go through and obfuscate it. This should be done next Monday - so I'll post here with the links when it's good to go :)

[NoPhaseNoKill]

Do let me know what you meant about the object literals though. We do have a large file which contains maybe 700/800 lines of object literals - so would be interested to see how we tackle this workaround considering we're assigning them to multiple consts etc.

Sorry for the constant posting, just brain dumping as I find out more info. Really appreciate the help :)