Flakiness in unit tests refers to the tendency of a test to sometimes pass and sometimes fail, even when there has been no change to the code being tested. This can happen for a variety of reasons, but the result is that the test is not reliable, and cannot be trusted to accurately reflect the behavior of the code.
Flakiness in unit tests is bad because it can make it difficult to determine whether a failure is due to a genuine problem with the code, or just a flaky test. This can lead to wasted time and effort as developers try to diagnose and fix problems that don't exist. In addition, flaky tests can undermine confidence in the overall test suite, making it harder to trust the results of the tests. Finally, flaky tests can make it difficult to make meaningful progress on a project, as developers may be hesitant to make changes to the code for fear of breaking a flaky test.
Tests that rely on external systems, such as databases or web services, can be flaky if those systems are not always available or behave differently between test runs. For example, if a test depends on a database and the database is not running, the test will fail. What "External systems" means depends on your context. It can be perfectly ok to have a test that depends on a database if the database is running on the same machine as the test. But, if the database is running on a different machine, it's an external system. The same goes for web services. If the web service is running on the same machine as the test, it's ok. But, if the web service is running on a different machine, it's an external system.
In this post, I'll describe how to prevent http requests to external services in .NET unit tests. This will allow you to detect and fix tests using external resources, so you can fix them and make them more reliable.
#Code
I've already written about observing all http requests in a .NET application. The blog post describes different ways to observe http requests made by a .NET application whether you are listening to requests from the current application or an external application. You can use the DiagnosticListener class to detect http requests from unit tests. Indeed, you only need to detect http requests from the current application. This method is simpler than other described methods and gives more data if needed.
Then, you need to register the DiagnosticListener to start listening http requests before the first test run. You could use what the test framework provides to do that. But, C# provides a way to execute a static method when the assembly is loaded. This way, you can be test-framework agnostic. The blog post "Executing code before Main in .NET" explain how you can declare a ModuleInitializer to execute code when the assembly is loaded.
Let's combine both technics to detect and prevent http requests to external services from the unit tests!
HttpRequestsDetector.cs (C#)
using System.Diagnostics;
using System.Runtime.CompilerServices;
internal static class HttpRequestsDetector
{
// Configure the list of allowed domains
private static readonly HashSet<string> AllowedHosts = new(StringComparer.OrdinalIgnoreCase)
{
"localhost",
};
// Methods decorated with [ModuleInitializer] are executed when the DLL is loaded,
// so before the first test is executed. This is test-framework agnostic!
// No need to read the documentation of your favorite test framework :)
[ModuleInitializer]
public static void Initialize()
{
var eventListener = new DiagnosticSourceSubscriber();
eventListener.Subscribe();
}
// Register a DiagnosticListener to listen to all http requests.
// When a request is sent to an external service, an exception is thrown and the test fails.
private sealed class DiagnosticSourceSubscriber : IDisposable, IObserver<DiagnosticListener>
{
private IDisposable? _allSourcesSubscription;
private IDisposable? _subscription;
public void Subscribe()
{
// Register the listener
_allSourcesSubscription ??= DiagnosticListener.AllListeners.Subscribe(this);
}
public void OnNext(DiagnosticListener value)
{
if (value.Name == "HttpHandlerDiagnosticListener")
{
_subscription = value.Subscribe(new HttpHandlerDiagnosticListener());
}
}
public void OnCompleted() { }
public void OnError(Exception error) { }
public void Dispose()
{
_subscription?.Dispose();
_subscription = null;
_allSourcesSubscription?.Dispose();
_allSourcesSubscription = null;
}
private sealed class HttpHandlerDiagnosticListener : IObserver<KeyValuePair<string, object?>>
{
private static readonly Func<object, HttpRequestMessage?> GetRequestPropertyValue = CreateGetRequestPropertyValue();
public void OnCompleted() { }
public void OnError(Exception error) { }
public void OnNext(KeyValuePair<string, object?> value)
{
if (value.Key is "System.Net.Http.Request")
{
var request = GetRequestPropertyValue(value.Value!);
if (request?.RequestUri?.Host is not string host)
return;
if (!AllowedHosts.Contains(host))
throw new InvalidOperationException($"Requesting external resource '{request.RequestUri}' from unit tests is forbidden");
}
}
// The object sent by .NET to the listener is not public, so you need to
// use reflection to get it.
private static Func<object, HttpRequestMessage?> CreateGetRequestPropertyValue()
{
var requestDataType = Type.GetType("System.Net.Http.DiagnosticsHandler+RequestData, System.Net.Http", throwOnError: true);
var requestProperty = requestDataType!.GetProperty("Request");
return (object o) => (HttpRequestMessage?)requestProperty!.GetValue(o);
}
}
}
}
If you run a test that requests an external service, you will get an exception like this:
#Additional resources
Do you have a question or a suggestion about this post? Contact me!
