Coverage results aren't stable

[ViktorHofer]

We (dotnet/runtime) recently switched from coverlet's global tool to dotnet test with the coverlet data collector. With the data collector driver I just noticed that the produce coverage results file isn't stable and varies between multiple invocations.

See the generated reports for five invocations of C:\git\runtime\src\libraries\System.Text.RegularExpressions\tests>dotnet test -f netcoreapp5.0 /p:Coverage=true: coverage-RegularExpressions.zip

Something that immediately points out is the difference in the summary:

  <Summary numSequencePoints="10557" visitedSequencePoints="7413" numBranchPoints="4982" visitedBranchPoints="3491" sequenceCoverage="70.21" branchCoverage="70.07" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="53" numClasses="55" visitedMethods="727" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="7431" numBranchPoints="4982" visitedBranchPoints="3516" sequenceCoverage="70.38" branchCoverage="70.57" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="53" numClasses="55" visitedMethods="727" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="8415" numBranchPoints="4982" visitedBranchPoints="4107" sequenceCoverage="79.71" branchCoverage="82.43" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="768" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="8181" numBranchPoints="4982" visitedBranchPoints="4029" sequenceCoverage="77.49" branchCoverage="80.87" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="758" numMethods="817" />
  <Summary numSequencePoints="10557" visitedSequencePoints="8376" numBranchPoints="4982" visitedBranchPoints="4092" sequenceCoverage="79.34" branchCoverage="82.13" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="768" numMethods="817" />

VSTest reports the same number of tests being discovered and executed for all the five runs:

Test Run Successful.
Total tests: 7204
     Passed: 7204

I'm wondering if hit points aren't correctly propagated back to the hits file?

cc @MarcoRossignoli @sharwell @petli @stephentoub

[MarcoRossignoli]

Could be related to other file lock issue, I'll take a look asap, bit busy week

[ViktorHofer]

I just compared the results with xunit.console and coverlet.console (1.7.1):

  <Summary numSequencePoints="10557" visitedSequencePoints="10258" numBranchPoints="4982" visitedBranchPoints="4652" sequenceCoverage="97.16" branchCoverage="93.37" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="55" numClasses="55" visitedMethods="807" numMethods="817" />

xunitconsole_coverage.zip

[ViktorHofer]

So the instrumentation is working as expected as the numSequencePoints, numBranchPoints, maxCyclomaticComplexity, minCyclomaticComplexity, numClasses and numMethods values are the same.

The visited ones differ largely...

[MarcoRossignoli]

Thanks for infos

[ViktorHofer]

Could be related to other file lock issue, I'll take a look asap, bit busy week

Unsure if that's really the reason. Just cleared my temp directory and test outdir and ran the command again and now get:

<Summary numSequencePoints="10557" visitedSequencePoints="8994" numBranchPoints="4982" visitedBranchPoints="4285" sequenceCoverage="85.19" branchCoverage="86" maxCyclomaticComplexity="5476" minCyclomaticComplexity="5476" visitedClasses="54" numClasses="55" visitedMethods="796" numMethods="817" />

The visited* numbers are still way too low. Any recommendation how to diagnose this further? How is coverlet.console behaving differently in respect to hitcount tracking than coverlet.collectors?

[MarcoRossignoli]

How is coverlet.console behaving differently in respect to hitcount tracking than coverlet.collectors?

The only difference is that collectors does not rely on process exit to flush hits but using in proc collector

https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L46

The out of proc collector do same thing as msbuild instrumentation task

https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L135

workflow

1. out of proc instruments file
2. test host run and in proc one is injected
3. test ends and in proc force flush hits file
4. out of proc get hits file and generates reports

[MarcoRossignoli]

Question, is possible that same instrumented sets of files run in more than one tests maybe in parallel?
Every test project should be used for "one test run" and not in parallel, because the hit filename is hardcoded in instrumented one so if it's used in different test one test could overwrite other.

[ViktorHofer]

The instrumented assemblies are only used throughout one "test session". But this might have to do with our RemoteExecutor tooling which loads the test assembly as well.

[MarcoRossignoli]

If the issue is RemoteExectutor we can try to find a way to avoid process exit flush in case of collectors usage because we can rely on in-proc flush at the end of tests.

Never mind, we cannot lose external run, we lose hits and is supported thanks to mutex and possible hits additions https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L86

The only way to lose hits is for parallel "generates report" with same instrumented assemblies where we remove hit file and another one fresh is created, but doesn't seem the case.

[ViktorHofer]

@MarcoRossignoli we currently support both xunit.console and VSTest at the moment. I plan to obsolete the former as soon as all features are on parity. Ideally we can fix the non-stable results before 7/6 (which is the dotnet/runtime infrastructure rollout date after the next one).

Do you believe #835 will fix this issue?

[MarcoRossignoli]

Do you believe #835 will fix this issue?

Unfortunately cannot be sure until try some repro with fix on my local, next days I'll try to repro on my local your sample and come back.
One question xunit.console run collectors?

cc: @tonerdo

[ViktorHofer]

One question xunit.console run collectors?

No. We are using coverlet.console in combination with xunit.console. That's what we want to get rid off in the future when the coverage results become stable.

[MarcoRossignoli]

Ok so the drift between xunit.console and direct dotnet test is always using same coverlet engine underneath