Code Chronicles
Inconsistent Test Runs

Inconsistent Test Runs

Between Visual Studio and dotnet test

Jeff Tong's photo
Jeff Tong
·

2 min read

Typically an test failures occur when there is some static variable that retains some sort of state. This could be from the TestClassInitialize.

There was an issue occurring with some changes that wouldn't have led to any failures. The tests all pass consistently in Visual Studio, but fail on our build pipeline. Looking deeper into the issue, the only difference is the running of dotnet test explicitly:

dotnet test -c Release

I was able to reproduce the issue right away relating to an expected exception that should have been thrown, but was not:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed InvokeAsync_When_NoCorrelationId_LogsAndThrows [100 ms]
  Error Message:
   Expected a <Starbucks.Payments.Common.CorrelationId.Exceptions.CorrelationIdException> to be thrown, but no exception was thrown.
  Stack Trace:
     at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
   at FluentAssertions.Specialized.DelegateAssertions`1.Throw[TException](Exception exception, String because, Object[] becauseArgs)
   at FluentAssertions.Specialized.DelegateAssertions`1.Throw[TException](String because, Object[] becauseArgs)
   at Payments.Common.CorrelationId.Tests.CorrelationIdEnforcementMiddlewareTests.InvokeAsync_When_NoCorrelationId_LogsAndThrows() in C:\git\Starbucks\Wallet\tests\Common\Payments.Common.CorrelationId.Tests\CorrelationIdEnforcementMiddlewareTests.cs:line 156

The code makes use of stevejgordon/CorrelationId.

Adding some diagnostic tracing:

System.Diagnostics.Trace.WriteLine(correlationContextAccessor.GetCorrelationId());

I could verify that a correlationId was returned from the correlationContextAccessor.GetCorrelationId(); call when run with dotnet test vs Visual Studio.

Debug Trace:
8bc57ef3-db44-4ce4-acff-58b60a2a2eae

Interesting that the same code while run in Visual Studio will yield a null for the CorrelationId.

Since the issue I opened for the ability to mock the CorrelationContextAccessor has been closed: github.com/stevejgordon/CorrelationId/issue..

We can now just directly mock with ICorrelationIdContextAccessor to enforce the expected behavior.

var accessor = new Mock<ICorrelationContextAccessor>();

or return a CorrelationContext with explicit null for CorrelationId.

var accessor = new CorrelationContextAccessor()
{
      CorrelationContext = new CorrelationContext(null, "header")
};

Tests now pass consistently via dotnet test and Visual Studio.

These tests were passing in the main branch so this behavior must have been introduced by a new package update.

dotnetcorevisual studiodebugging