-
Notifications
You must be signed in to change notification settings - Fork 390
Added InstrumentModulesWithoutLocalSources setting #1360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Hi. I've just created a similar PR: #1363 |
@searoz there are already include / exclude parameters available. I'd prefer the option to completely disable the implicit filtering (pdb without local sources), because it's not really intuitive - and use the existing filter possibilities instead to exclude or include certain modules. Are the existing filter options not sufficient for your case? |
@TFTomSun yeah, I guess you are right, the combination of BTW, I don't see any changes to |
@TFTomSun @searoz Thanks for both your PRs adressing this longstanding issue! I also think the approach of having a flag that controls the heuristic and then rely on normal include/exclude filtering is less complex both to understand as a user and to maintain in code. Have you seen #1164 which also discusses the topic? Does it make more sense to you to have a flag that overrides the automatic exclusion of supposed third-party assemblies, as in both your PRs, or have a flag that instead allows finetuning the behaviour of that behaviour as in that issue? |
@petli I found the default filter based on local sources confusing, We have for example build packages that link additional files into the project, which are not available if only the sources are cloned. Additionally we usually don't do test runs in dev environments. We want to avoid issues in production that are caused by dev environment expectations. Therefore turning off that local source based filter completely is what we need. We want to distinguish between internal and external assemblies upon their names, not upon the environment. |
@TFTomSun I think the current behaviour works well for straightforward projects out-of-the-box, but as soon as you start having a more complex build process there's a need for being able to turn it off. Since there is also the issue that this behaviour easily gets thrown off by generated sources, I was thinking it could be handled with a single parameter with multiple options, but I'm coming around to that it would be clearer for the user to have the option in this PR that completely disables the automatic filtering, and addressing the generated source files by relaxing the behaviour to include assemblies as long as there is at least one local source file. Any opinion, @MarcoRossignoli? |
Co-authored-by: Peter Liljenberg <[email protected]>
@petli I'm ok with the idea to have the option to exclude heuristic and move the "any" to "all". The heuristic was in place to automatically exclude for instance xunit, test adapters etc...that are close to the user dll test. @TFTomSun I'm oof for a pair of week I'll take a look when I'll be back from the vacation. @petli feel free to review and approve and I'll merge. For the "any" to "all" I expect some new "ticket issue" for unwanted included dll...but if are a lot we can add parameters to the new flag keeping current default. World go on, code generators changed the game here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the code looks good, but can you take a look at the comment #1360 (comment) about the cobertura generator and see if that needs addressing with this solution too.
It would also be golden if you can add this new flag to the documentation for the different drivers too. :)
@petli What's the expected behavior in the reporter? Do we expect the sources files to be available there? Our plan was to execute the tests in a close-to-production environment and create the report in a dev environment (with sources) and the source link case is already handled in there. As far as I know, the reporter needs the source files, right? So maybe that debug.assert still makes sense? |
@TFTomSun @petli AFAIU, cobertura reporter only uses sources to "fix" file paths in the report. IMHO, this is a pretty minor feature that could be safely ignored. The only reason I brought this up is the fact that cobertura reporter has
During my tests this |
Sounds like the |
removed obsolete Debug.Assert
Update CoberturaReporter.cs
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code looks good to me. It would be useful with adding the new parameters to the documentation for the three drivers, but I can also help out with that later in a separate PR.
@petli Do you know when can we expect new Coverlet version with these changes included? Our CI/CD depends on them (that's why I created my own PR in the first place) |
@searoz Once this is merged it would be included in the nightly builds, so you could target those until a release is published by @MarcoRossignoli. |
Updated [coverlet.collector](https://github.com/coverlet-coverage/coverlet) from 3.1.2 to 6.0.4. <details> <summary>Release notes</summary> _Sourced from [coverlet.collector's releases](https://github.com/coverlet-coverage/coverlet/releases)._ ## 6.0.4 ### Fixed - Fix empty coverage report when using include and exclude filters [#1726](coverlet-coverage/coverlet#1726) [Diff between 6.0.3 and 6.0.4](coverlet-coverage/coverlet@v6.0.3...v6.0.4) ## 6.0.3 ### Fixed - Fix RuntimeConfigurationReader to support self-contained builds [#1705](coverlet-coverage/coverlet#1705) by https://github.com/pfeigl - Fix inconsistent filenames with UseSourceLink after .NET 8 [#1679](coverlet-coverage/coverlet#1679) - Fix hanging tests [#989](coverlet-coverage/coverlet#989) - Fix coverlet instrumentation becomes slow after installing dotnet sdk 8.0.200 [#1620](coverlet-coverage/coverlet#1620) - Fix upgrading v6.0.1 to v6.0.2 increases instrumentation time [#1649](coverlet-coverage/coverlet#1649) - Fix Unable to instrument module - NET 8 [#1631](coverlet-coverage/coverlet#1631) - Fix slow modules filtering process [#1646](coverlet-coverage/coverlet#1646) by https://github.com/BlackGad - Fix incorrect coverage await using in generic method [#1490](coverlet-coverage/coverlet#1490) ### Improvements - Cache the regex used in InstrumentationHelper [#1693](coverlet-coverage/coverlet#1693) - Enable dotnetTool integration tests for linux [#660](coverlet-coverage/coverlet#660) [Diff between 6.0.2 and 6.0.3](coverlet-coverage/coverlet@v6.0.2...v6.0.3) ## 6.0.2 ### Fixed - Threshold-stat triggers error [#1634](coverlet-coverage/coverlet#1634) - Fixed coverlet collector 6.0.1 requires dotnet sdk 8 [#1625](coverlet-coverage/coverlet#1625) - Type initializer errors after updating from 6.0.0 to 6.0.1 [#1629](coverlet-coverage/coverlet#1629) - Exception when multiple exclude-by-attribute filters specified [#1624](coverlet-coverage/coverlet#1624) ### Improvements - More concise options to specify multiple parameters in coverlet.console [#1624](coverlet-coverage/coverlet#1624) [Diff between 6.0.1 and 6.0.2](coverlet-coverage/coverlet@v6.0.1...v6.0.2) ## 6.0.1 ### Fixed - Uncovered lines in .NET 8 for inheriting records [#1555](coverlet-coverage/coverlet#1555) - Fix record constructors not covered when SkipAutoProps is true [#1561](coverlet-coverage/coverlet#1561) - Fix .NET 7 Method Group branch coverage issue [#1447](coverlet-coverage/coverlet#1447) - Fix ExcludeFromCodeCoverage does not exclude method in a partial class [#1548](coverlet-coverage/coverlet#1548) - Fix ExcludeFromCodeCoverage does not exclude F# task [#1547](coverlet-coverage/coverlet#1547) - Fix issues where ExcludeFromCodeCoverage ignored [#1431](coverlet-coverage/coverlet#1431) - Fix issues with ExcludeFromCodeCoverage attribute [#1484](coverlet-coverage/coverlet#1484) - Fix broken links in documentation [#1514](coverlet-coverage/coverlet#1514) - Fix problem with coverage for .net5 WPF application [#1221](coverlet-coverage/coverlet#1221) by https://github.com/lg2de - Fix unable to instrument module for Microsoft.AspNetCore.Mvc.Razor [#1459](coverlet-coverage/coverlet#1459) by https://github.com/lg2de ### Improvements - Extended exclude by attribute feature to work with fully qualified name [#1589](coverlet-coverage/coverlet#1589) - Use System.CommandLine instead of McMaster.Extensions.CommandLineUtils [#1474](coverlet-coverage/coverlet#1474) by https://github.com/Bertk - Fix deadlog in Coverlet.Integration.Tests.BaseTest [#1541](coverlet-coverage/coverlet#1541) by https://github.com/Bertk - Add coverlet.msbuild.tasks unit tests [#1534](coverlet-coverage/coverlet#1534) by https://github.com/Bertk [Diff between 6.0.0 and 6.0.1](coverlet-coverage/coverlet@v6.0.0...v6.0.1) ## 6.0.0 Fixed -Could not write lines to file CoverletSourceRootsMapping - in use by another process coverlet-coverage/coverlet#1155 -Incorrect coverage for methods returning IAsyncEnumerable in generic classes coverlet-coverage/coverlet#1383 -Wrong branch coverage for async methods .NET Standard 1.x coverlet-coverage/coverlet#1376 -Empty path exception in visual basic projects coverlet-coverage/coverlet#775 -Align published nuget package version to github release version coverlet-coverage/coverlet#1413 -Sync nuget and github release versions coverlet-coverage/coverlet#1122 Improvements -Migration of the project to .NET 6.0 coverlet-coverage/coverlet#1473 Breaking changes New parameter ExcludeAssembliesWithoutSources to control automatic assembly exclusion coverlet-coverage/coverlet#1164. The parameter InstrumentModulesWithoutLocalSources has been removed. since it can be handled by setting ExcludeAssembliesWithoutSources to None. The default heuristics for determining whether to instrument an assembly has been changed. In previous versions any missing source file was taken as a signal that it was a third-party project that shouldn't be instrumented, with exceptions for some common file name patterns for source generators. Now only assemblies where no source files at all can be found are excluded from instrumentation, and the code for detecting source generator files have been removed. To get back to the behaviour that at least one missing file is sufficient to exclude an assembly, set ExcludeAssembliesWithoutSources to MissingAny, or use assembly exclusion filters for more fine-grained control. [Diff between 3.2.0 and 6.0.0](coverlet-coverage/coverlet@v5.7.2...v6.0.0) ## 5.8.0 ### Fixed -Fix TypeLoadException when referencing Microsoft.Extensions.DependencyInjection v6.0.1 [#1390](coverlet-coverage/coverlet#1390) -Source Link for code generators fails [#1322](coverlet-coverage/coverlet#1322) -Await foreach has wrong branch coverage when method is generic [#1210](coverlet-coverage/coverlet#1210) -ExcludeFromCodeCoverage attribute on local functions ignores lambda expression [#1302](coverlet-coverage/coverlet#1302) ### Added -Added InstrumentModulesWithoutLocalSources setting [#1360](coverlet-coverage/coverlet#1360) by @TFTomSun [Diff between 3.1.2 and 3.2.0](coverlet-coverage/coverlet@v5.7.2...v5.8.0) ## 5.7.2 Fixed -Fix CoreLib's coverage measurement is broken [#1286](coverlet-coverage/coverlet#1286) -Fix UnloadModule injection [1291](coverlet-coverage/coverlet#1291) [Diff between 3.1.1 and 3.1.2](coverlet-coverage/coverlet@v5.7.1...v5.7.2) ## 5.7.1 Fixed -Fix wrong branch coverage with EnumeratorCancellation attribute #1275 -Fix negative coverage exceeding int.MaxValue #1266 -Fix summary output format for culture de-DE #1263 -Fix branch coverage issue for finally block with await #1233 -Fix threshold doesn't work when coverage empty #1205 -Fix branch coverage issue for il switch #1177 -Fix branch coverage with using statement and several awaits#1176 -Fix CopyCoverletDataCollectorFiles to avoid to override user dlls for dotnet publish scenario #1243 Improvements -Improve logging in case of exception inside static ctor of NetstandardAwareAssemblyResolver #1230 -When collecting open the hitfile with read access #1214 by https://github.com/JamesWTruher -Add CompilerGenerated attribute to the tracker #1229 [Diff between 3.1.0 and 3.1.1](coverlet-coverage/coverlet@v5.7.0...v5.7.1) ## 5.7.0 Fixed -Fix branch coverage for targetframework net472 #1167 -Fix F# projects with unkown source #1145 -Fix SkipAutoProps for inline assigned properties #1139 -Fix partially covered throw statement #1144 -Fix coverage threshold not failing when no coverage #1115 -Fix partially covered await foreach statement #1107 by https://github.com/alexthornton1 -Fix System.MissingMethodException(TryGetIntArgFromDict) #1101 -Fix ExcludeFromCodeCoverage on props #1114 -Fix incorrect branch coverage with await using #1111 by https://github.com/alexthornton1 Added -Support deterministic reports #1113 -Specifying threshold level for each threshold type #1123 by https://github.com/pbmiguel Improvements -Implementation of Npath complexity for the OpenCover reports #1058 by https://github.com/benjaminZale [Diff between 3.0.3 and 3.1.0](coverlet-coverage/coverlet@v5.6.3...v5.7.0) ## 5.6.3 Fixed -Fix code coverage stops working if assembly contains source generators generated file [#1091](coverlet-coverage/coverlet#1091) [Diff between 3.0.2 and 3.0.3](coverlet-coverage/coverlet@v5.6.2...v5.6.3) ## 5.6.2 Fixed -Fix multi-line lambda coverage regression #1060 -Opt-in reachability helper to mitigate resolution issue #1061 [Diff between 3.0.1 and 3.0.2](coverlet-coverage/coverlet@v5.6.1...v5.6.2) ## 5.6.1 Fixed -Fix severe loss in coverage #1043 by https://github.com/daveMueller ## 5.6.0 Fixed -Attribute exclusion does not work if attribute name does not end with "Attribute" #884 by https://github.com/bddckr -Fix deterministic build+source link bug #895 -Fix anonymous delegate compiler generate bug #896 -Fix incorrect branch coverage with await ValueTask #949 by https://github.com/alexthornton1 -Fix switch pattern coverage #1006 Added -Skip autoprops feature #912 -Exclude code that follows [DoesNotReturn] from code coverage #904 by https://github.com/kevin-montrose -CoverletReport MSBuild variable containing coverage filenames #932 by https://github.com/0xced -Add Visual Studio Add-In #954 by https://github.com/FortuneN -Remove workaround for deterministic build for sdk >= 3.1.100 #965 -Allow standalone coverlet usage for integration/end-to-end tests using .NET tool driver #991 -Support .NET Framework(>= net461) for in-process data collectors #970 ## 5.4.0 Fixed -Fix for code complexity not being generated for methods for cobertura reporter #738 by https://github.com/dannyBies -Fix coverage, skip branches in generated MoveNext() for singleton iterators #813 by https://github.com/bert2 -Fix 'The process cannot access the file...because it is being used by another process' due to double flush for collectors driver #coverlet-coverage/coverlet#835 -Fix skip [ExcludefromCoverage] for generated async state machine #849 Added -Added support for deterministic build for msbuild/collectors driver #802 #796 with the help of https://github.com/clairernovotny and https://github.com/tmat Improvements -Refactore DependencyInjection #728 by https://github.com/daveMueller ## 5.3.1 Fixed -Fix ExcludeFromCodeCoverage attribute bugs #129 and #670 with #671 by https://github.com/matteoerigozzi -Fix bug with nested types filtering #689 -Fix Coverage Issue - New Using + Async/Await + ConfigureAwait #669 -Improve branch detection for lambda functions and async/await statements #702 by https://github.com/matteoerigozzi -Improve coverage, hide compiler generated branches for try/catch blocks inside async state machine #716 by https://github.com/matteoerigozzi -Improve coverage, skip lambda cached field #753 Improvements -Trim whitespace between values when reading from configuration from runsettings #679 by https://github.com/EricStG -Code improvement, flow ILogger to InstrumentationHelper #727 by https://github.com/daveMueller -Add support for line branch coverage in OpenCover format #772 by https://github.com/costin-zaharia ## 5.3.0 Added -Add log to tracker #553 -Exclude by assembly level System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage #589 -Allow coverlet integration with other MSBuild test strategies#615 by https://github.com/sharwell Fixed -Fix and simplify async coverage #549 -Improve lambda scenario coverage #583 -Mitigate issue in case of failure in assembly loading by cecil #625 -Fix ConfigureAwait state machine generated branches #634 -Fix coverage overwritten if the project has multiple target frameworks #636 -Fix cobertura Jenkins reporter + source link support #614 by https://github.com/daveMueller -Fix pdb file locking during instrumentation #656 Improvements -Improve exception message for unsupported runtime [#569](https://github.com/tonerdo/ coverlet/pull/569) by https://github.com/daveMueller -Improve cobertura absolute/relative path report generation #661 by https://github.com/daveMueller ## 5.2.0 [Feature][Collectors]Output multiple formats #533 [Bug fix] Fix exclude by files #524 [Feature] Skip instrumentation of module with embedded ppbd without local sources #510 -> with this today xunit will be skipped in automatic way. [Bug fix] Changed to calculate based on the average coverage of the module #479 [Bug fix] Fix property attribute detection #477 [Feature] Different exit codes to indicate particular failures #412 [Bug fix] Fix instrumentation serialization bug #458 [Bug fix] Fix culture for cobertura xml report #464 ## 5.1.1 #406 downgrades verbosity when hits file isn't found #409 Improves Coverage object to remove need to statically pass an instance of it #415 Improves validation of hits file existence ## 5.1.0 #367 ensures invalid PDBs do not error out instrumentation #368 fixes package layout adding support for .NET framework projects #375 ensures PDBs exist before trying to restore them and prevents errors for assemblies with embedded PDBs #376 makes inclusion of test assembly excluded by default and configurable #383 ensures Coverlet restores original assemblies if process exits unexpectedly #397 rounds down Coverage result ## 5.0.0 #303 fixes strong name validation errors #309 adds option to only record a single hit to improve performance #318 adds support for covering test files #322 reverts using memory mapped files to store hit counts #331 fixes open cover reporting errors #341 adds logging to Coverlet to improve debugging ## 4.1.1 #286 Fixes type loading issues with .NET Framework assemblies #291 Adds enhancements to multi-threaded scenarios #297 and #302 Fixes issues with instrumenting System.Private.CoreLib #276 switches to using memory mapped files instead of regular files for storing hit information #277 Enhances support for async/await branches after coverage merge ## 4.1.0 * #259 adds support for assemblies with embedded PDBs * #260 adds support for using SourceLink provided source file URLs in coverage results * #236 improves matcher when computing excluded source files * #256 fixes TeamCity reporter decimal separator ## 4.0.0 * #244 fixes error in TeamCity reporting * #252 updates version of Mono.Cecil * #233 adds support for specifying custom attributes on types/methods that you'd like to exclude * #253 allows Coverlet to probe for additional assemblies beyond the dependencies of the test assembly ## 3.2.2 * #220 ensure merging doesn't fail if existing results file doesn't exist * #132 adds support for computation of cyclomatic complexity * #227 adds support for TeamCity reporting format ## 3.2.1 * #158 supports branch coverage on asynchronous calls * #204 and #211 fixes coverage result merging algorithm * #209 Adds specific support for instrumenting System.Private.CoreLib ## 3.2.0 * #172 adds performance enhancements during instrumentation * #181 removes instrumented assemblies' dependency on the `coverlet.tracker` assembly * #186 adds support for merging results of current run with result from different run Commits viewable in [compare view](https://github.com/coverlet-coverage/coverlet/commits/v6.0.4). </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xkonti <[email protected]>
There are already several issues regarding unexpected empty coverage results.
#1089
#1121
We found out, that there's an implicit filter that checks whether a pdb source location can be mapped to a local file. However that logic is not optimal for some use cases, for example when the build and test run happens on a different machines or when the sources are embedded in the assembly. Since the source files are not used anywhere in the whole instrumentation logic except for that filter logic, we added an option to disable that implicit filter. The previous behavior stays the default behavior. With that setting activated, the user has full control over the filtering and can still filter using Include / Exclude to filter out 3rdParty or include only the assemblies he explicitly wants to measure.