Massive disk usage when running on .NET Core / Ubuntu

[jamesmeneghello]

One of our build runners is Ubuntu-based currently, building a reasonably large project (~15 projects in a single solution). Using dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover, I'm seeing some files created in /tmp that are fairly enormous (naming scheme <project>_<hash>) - one of the files covering our largest project is nearly 90gb - about 180gb for all projects combined. Disabling coverlet, the files don't get created at all, so I'm assuming they're cache files of some kind.

Should they be this big? Our build runner doesn't have that much disk space allocated, so it keeps bringing it down :|

[hunterjm]

@Murodese - That would be the hits file. It adds a new line for every sequence point hit, soon to be branch targets as well. Those must be some massive projects. The hits file for this project is 133MB on my machine, and that's on my branch which adds additional instrumentation for better branch coverage. So 15 of them would be ~2GB in total.

@tonerdo - It might be worthwhile looking into reading and writing from a GZipStream. The markers should compress pretty well.

[tonerdo]

@hunterjm @Murodese this has already been fixed in #58. New NuGet package with the fix going out in a couple of hours

[tonerdo]

On closer look, this looks like more of a disk usage thing than a RAM usage thing. Definitely need to consider your approach @hunterjm

[jamesmeneghello]

Yeah, RAM usage is fine, it's just literally filling up our disk.

[OneCyrus]

yes, the memory problem comes later. but should be fixed in the next release. otherwise you would need 90-180GB memory as well to process this file.

the GZipStream could be a good option to solve this problem. shouldn't be too hard to implement.

[hunterjm]

@tonerdo @OneCyrus @Murodese - Initial take on using GZipStream is on my fork. If I merge hunterjm#1, it will become a part of #69. Otherwise, I can wait for #69 to be merged and re-target hunterjm#1 to master.

[tonerdo]

@hunterjm thanks for the fix, will take a look in a bit. Let's get #69 in first and then we can proceed with the branch fixes. There's also no harm in opening up a separate PR to this repo

[hunterjm]

@tonerdo - Agreed. I just made the changes locally off of the #69 branch, and since the fix touches Coverage.cs there would be merge conflicts if I tried to pull it out and open a separate PR to master.

[tonerdo]

New release with disk usage fix - https://www.nuget.org/packages/coverlet.msbuild/2.0.0