Code coverage with Coverlet

I’ve trying to get some code coverage results from our tests and blocking on a percentage threshold. Start by googling and reading the docs in respect to .NET. I’ve done this before with older versions of .NET and MSBuild.

https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=windows

Finds a syntax problem and open up an github issue.

We had used Coverlet in the past with other projects and the code base I am working on already has a reference to coverlet.collector in our test projects. I want to be able to fail the build upon coverage percentage falling below a specific percentage.

Noticed that the /p:Threshold parameter was not failing the build. Opened up a github issue. After reading the documentation noticed that there were two integrations:

• VSTest Integration (coverlet.collector).

• MSBuild Integration (coverlet.msbuild).

It’s noted not to reference both.

The threshold MSBuild property is only supported by the MSBuild Integration driver. Duh.

I replaced with the appropriate MSBuild Integration package.

dotnet remove package coverlet.collector
dotnet add package coverlet.msbuildThis successfully fails the build:

dotnet test --no-restore --no-build  -p:CollectCoverage=true -p:Threshold=100 -p:ThresholdStat=Total

Yay!

This does not:

dotnet test --no-restore --no-build  -p:CollectCoverage=true -p:Threshold=99,99,99 `
-p:ThresholdStat=Total

MSBUILD : error MSB1006: Property is not valid.
Switch: 99

Boo!

This also fails with double or single quoted value -p:Threshold:”99,99,99”. After some using the error message (useless other than finding a similar comma issue with coverlet.msbuild). Searching for “MSBuild and Commas” led me to this open issue.

Here is the working command:

dotnet test --no-restore --no-build  -p:CollectCoverage=true -p:Threshold=99%2C99%2C99 `
-p:ThresholdType=line%2Cbranch%2Cmethod -p:ThresholdStat=Total

For ease of testing, I created these helper functions to be sourced:

. .\CcHelper.ps1

Initially, I had run dotnet test with the solution (.sln) directly. This ran coverage for all projects in the solution and the threshold values were applied to each project and not the total coverage percentage. In the Invoke-CodeCoverageWithThreshold function we run each project found explicitly and generate a coverage output to the same location. Using ./TestResults for the --results-directory is relative to each project directory. The once the last .csproj is hit we can use the /p:MergeWith property to merge all report paths.

Terminal Output

If the output needs to be parsed or there is an issue with the MSBuild output, try /p:VSTestUseMSBuildOutput=false

Passed!  - Failed:     0, Passed:    40, Skipped:     0, Total:    40, Duration: 181 ms - Code.Chronicles.Wind.Tests.Client.UnitTests.dll (net8.0)
  [coverlet]
  Calculating coverage result...
  [coverlet] MergeWith: 'C:\git\SomeProject\TestResults\CodeCoverageResults.json'.
   Generating report 'C:\git\SomeProject\TestResults\CodeCoverageResults.json
  '

+---------------------------------------+--------+--------+--------+
| Module                                | Line   | Branch | Method |
+---------------------------------------+--------+--------+--------+
| Code.Chronicles.Wind.Client.Contracts | 100%   | 100%   | 100%   |
+---------------------------------------+--------+--------+--------+
| Code.Chronicles.Wind.Client           | 43.28% | 51.06% | 61.53% |
+---------------------------------------+--------+--------+--------+
| Code.Chronicles.Wind.Common           | 100%   | 81.25% | 100%   |
+---------------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 55.81% | 58.73% | 66.66% |
+---------+--------+--------+--------+
| Average | 81.09% | 77.43% | 87.17% |
+---------+--------+--------+--------+

Exclusions

I had a PR suggestion to use a .runsettings for exclusion of assemblies and namespaces. This sounded reasonable since it cuts down on having to specify the /p:Exclude and /p:ExcludeByAttribute properties in more than one place (helper script and build definition).

I added this support in with —settings flag, but after testing and seeing why certain assemblies were not excluded, realized this is only relevant for the VSTest Integration which makes sense for runsettings.

I added additional exclusions for Exceptions and the URL encoded comma is relevant as we pass this in via the MSBuild property argument:

/p:Exclude="[*]*.Program%2C[*]*.Controllers.*%2C[*]*.Models.*%2C[*]*.Exceptions*"

In generally, I try to include the [ExcludeFromCodeCoverage] attribute when I write model, contract, exception, test classes. This is safer as a blanket exclusion of assemblies and namespaces could exclude code that should be covered.

Python

In contrast things seemed much simpler with code coverage for Python tests with pytest-cov:

#!/bin/bash

# Normally just run:  pytest .

arg=$1
filters='not (e2e or arima or integration)'
test_dir='unit_test'
number_of_workers=4

# Check if an argument is provided
if [ $# -eq 0 ]; then
    pytest $test_dir -p no:warnings -m "$filters" -n $number_of_workers
    exit 0
fi

case $arg in
  cover)
    coverage run -m pytest $test_dir -p no:warnings -m "$filters" -n $number_of_workers --doctest-modules --junitxml=junit/test-results.xml
    ;;
  x)
    pytest $test_dir -x -p no:warnings -m "$filters" -n $number_of_workers
    ;;
*)
  echo "Unrecognized argument=$arg"
  ;;
esac

-------------------- coverage: ... ---------------------
Name                 Stmts   Miss  Cover
--------------------------------------------------------
code_chronicles/__init__            2      0   100%
code_chronicles/wind              257     13    56%
code_chronicles/common             94      7    94%
--------------------------------------------------------
TOTAL                             353     20    83%

Summary

It makes sense why there are multiple drivers. We could use the VSTest Integration, but we would have to fail the build manually by doing the aggregation of the percentage ourselves from the code coverage output.

• Coverlet drivers have different functionality.

• URL encoding for , (%2C) if your MSBuild property contains them.

• RTFM

References

• https://github.com/dotnet/docs/issues/45356

• https://github.com/coverlet-coverage/coverlet

• https://github.com/coverlet-coverage/coverlet/issues/1738

• https://github.com/coverlet-coverage/coverlet/blob/e106d0d701f37e5b38144509165189ad38d61baa/README.md?plain=1#L79

• https://github.com/dotnet/msbuild/issues/471

• https://github.com/coverlet-coverage/coverlet/issues/170#issuecomment-2744716382

• https://stackoverflow.com/questions/53255065/dotnet-unit-test-with-coverlet-how-to-get-coverage-for-entire-solution-and-not

• https://github.com/dotnet/sdk/issues/39127

• https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022

• https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.codeanalysis.excludefromcodecoverageattribute?view=net-9.0

• https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

• https://en.wikipedia.org/wiki/RTFM#:~:text=RTFM%20is%20an%20initialism%20and,forum%2C%20software%20documentation%20or%20FAQ.

• https://pypi.org/project/pytest-cov/