[rush] Ideas for Summarizer improvements

[witcher112]

Intro

Hi everyone!

I want to follow-up on the discussion from today's Rush Hour about potential Summarizer improvements.

Right off the bat, I can say I'm willing to help working on them, but beforehand I think it would be best to have a discussion on how to approach them.

I think we can start by sketching out different use cases and come up with various solutions.

Issue

Summarizer doesn't provide complete outputs of commands/phases making it hard to learn why particular project fails to build/test etc.

When working with editors like VSCode

Proposal #VSCode-1 - display path to *.log files

Since VSCode is able to detect file paths in the terminal as well as in the editor, it's possible to simply print the path to the generated log file so the developer would be able to access them just with Ctrl + Mouse Click.

Advantages:

• easy to develop
• could be opt-in (through argument like --print-log-paths)

Disadvantages: (I can't think of any)

Proposal #VSCode-2 - create a VSCode problem matcher

(not mine, proposed by someone else during Rush Hour, I'm just trying to start exploring this option)

We can introduce a custom VSCode problem matcher that would display errors in the "Problems" tab.

From my understanding of how these work, it could either:

• detect problems from the terminal
• detect problems from logs files

Advantages:

• uses built-in UI of VSCode made for reviewing problems

Disadvantages:

• (in case of detecting problems from the terminal) all errors would need to be printed anyway
• hard to develop (I guess it would need to support each tool individually as TypeScript, ESLint, etc. display errors in different formats)
• VSCode exclusive

When reviewing logs from CI

Proposal #CI-1 - display X last lines

Advantages:

• moderately easy to develop
• could be opt-in and configurable (through arguments like --print-log-last-lines and --print-log-last-lines-count

Disadvantages:

• hard to predict good amount of last lines to print as it varies between tools (while 10 should be enough for TypeScript Compiler, it could cut out results of some test runners)

Proposal #CI-2 - use AI summarizing

Just throwing out the idea, not sure if it's even possible/acceptable for tools like rush to leverage AI summarizing.

Advantages:

• if there are tons of errors, it could give a general idea of what's wrong
• could be opt-in (through argument like --print-log-ai-summarize)

Disadvantages:

• not consistent
• most probably hard to develop and introduces a new stack of dependencies
• could be slow and require an internet connection (depending on implementation)

My thoughts

As I don't heavily use CI, I want to limit my thoughts to the VSCode case only as I'm primarily interested in it.

The #VSCode-1 proposal could be the best way to start working on improvements here as it doesn't pollute the output too much but provides a quick and easy way to inspect all errors. Also, the fact that it could be opt-in makes it somewhat non-intrusive.

Please let me know what you think about it, I'm heavily motivated to push this topic forward :D

[witcher112]

Here's draft PR for #VSCode-1 proposal: #4999

[dmichon-msft]

I'm the primary advocate for integration with problem matchers; the request is not to implement a problem matcher into VSCode, but to integrate VSCode's problem matcher system into Rush, so that Rush can consume custom-defined problem matchers and use them to extract errors/warnings from the underlying tools' build logs.

[witcher112]

I'm the primary advocate for integration with problem matchers; the request is not to implement a problem matcher into VSCode, but to integrate VSCode's problem matcher system into Rush, so that Rush can consume custom-defined problem matchers and use them to extract errors/warnings from the underlying tools' build logs.

So once extracted, they should be provided to VSCode so they're displayed in "Problems" tab, do I get it right?

[dmichon-msft]

So once extracted, they should be provided to VSCode so they're displayed in "Problems" tab, do I get it right?

The extraction would then be used to have the final summary at the end of the build only list warnings and errors, rather than arbitrary chunks of the log. It could also be used to allow Rush plugins to interact directly with the warning and error messages from builds and perform custom actions, such as emitting them according to the specifications of the CI environment for them to be surfaced more readily (e.g. in Azure Pipelines, via ##vso[task.logissue ...]message here log lines, which result in the UI showing the messages).

With a VSCode plugin, they could be routed to the "Problems" tab via various built-in hooks as well, yes.

[kevin-y-ang]

Proposal #VSCode-1 - display path to *.log files
Since VSCode is able to detect file paths in the terminal as well as in the editor, it's possible to simply print the path to the generated log file so the developer would be able to access them just with Ctrl + Mouse Click.

This isn't necessary a feature that needs to be tied to VSCode, is it? It would be helpful inside a containerized environment as well.

Rather than assuming that *.log files are an inevitable technical constraint to work around, would it be possible to allow pipling the log outputs to multiple different custom utilities? Whether that's a VScode extension, or a fluentd or logstash service, or an AI summarizer.