Caching http requests for faster debugging

When working with an http service, you have to do lots of http requests to the server. This may slow down your tests/debugging experience. There are multiple ways to handle that issue. The first one is to mock the calls and work with fake data. You can easily mock the HttpClient class in C# (see Mocking an HttpClient using an HttpClientHandler), but this requires to do that for each projects and to create fake responses for each request. The other solution is to create a global proxy on the machine that can automatically cache the responses and serve them when possible.

How Http caching with FiddlerCore works

To create a proxy in .NET, you can use Telerik FiddlerCore, the library behind Fiddler.

Progress® Telerik® FiddlerCore Embedded Engine is a .NET Class library you can integrate into your .NET Framework applications. There is also a .NET Standard 2.0 flavor of FiddlerCore that allows using it on any platform implementing .NET Standard.

Telerik FiddlerCore allows you to capture and modify HTTP and HTTPS traffic just like Fiddler, without any of the Fiddler UI.

Once you have downloaded the zip, you can add a reference to FiddlerCore45.dll. Then, you can start the proxy:

private static Proxy _proxyEndpoint;

public static void StartProxy()
{
    // Register for the FiddlerCore events
    FiddlerApplication.BeforeRequest += OnBeforeRequest;
    FiddlerApplication.AfterSessionComplete += OnAfterSessionComplete;

    // Start the proxy
    var startupSettings = new FiddlerCoreStartupSettingsBuilder()
        .ListenOnPort(8877)
        .RegisterAsSystemProxy()
        .DecryptSSL()
        .ChainToUpstreamGateway()
        .MonitorAllConnections()
        .OptimizeThreadPool()
        .Build();

    FiddlerApplication.Startup(startupSettings);
    _proxyEndpoint = FiddlerApplication.CreateProxyEndpoint(48721, true, "localhost");
}

Now we can handle the AfterSessionComplete event to save the responses in the local cache:

private static readonly object FromCache = new object();

private static void OnAfterSessionComplete(Session session)
{
    // We'll tag the session where the response comes from the cache, so we do not recache them...
    if (session.Tag == FromCache)
        return;

    // Allows to filter requests we want to cache
    if (!ShouldProcess(session))
        return;

    // Save the response to the disk
    var cacheFile = GetCacheFile(session);
    Directory.Create(Path.GetDirectoryName(cacheFile));
    session.SaveResponse(cacheFile, bHeadersOnly: false);
}

private static string GetCacheFile(Session session)
{
    using (var sha1 = SHA1.Create())
    {
        var hash =  Convert.ToBase64String(sha1.ComputeHash(Encoding.UTF8.GetBytes(url)));
        return Path.Combine(Path.GetTempPath(), "HttpCache", "Cache", hash + ".cache")
    }
}

private static bool ShouldProcess(Session session)
{
    if (session.RequestMethod.EqualsIgnoreCase("CONNECT"))
        return false;

    // TODO add your own logic here
    // You can filter on the method, the url, the headers or the body
    return true;
}

The last part is to use the cache to send the response. This is the time to handle the OnBeforeRequest event:

private static void OnBeforeRequest(Session session)
{
    if (!ShouldProcess(session))
        return;

    var file = GetCacheFile(session);
    if (File.Exists(file))
    {
        // Indicate FiddlerCore to not send the request to the server, we'll fill the response
        session.utilCreateResponseAndBypassServer();

        // Set the response from the file
        session.LoadResponseFromFile(file);

        // Tag the session to indicate the response comes from the cache
        session.Tag = FromCache;
    }
}

Finally, you'll have to stop the proxy when you leave the application:

public static void Stop()
{
    if (_proxyEndpoint != null)
    {
        _proxyEndpoint.Detach();
        _proxyEndpoint.Dispose();
    }

    FiddlerApplication.BeforeRequest -= OnBeforeRequest;
    FiddlerApplication.AfterSessionComplete -= OnAfterSessionComplete;
    if (FiddlerApplication.IsStarted())
    {
        FiddlerApplication.Shutdown();
    }
}

#Supporting https traffic

If you want to handle https requests, you have to install the root certificate of Fiddler. Everything is provided in the NuGet package to automate the process:

public static bool InstallCertificate()
{
    if (!CertMaker.rootCertExists())
    {
        if (!CertMaker.createRootCert())
            return false;

        if (!CertMaker.trustRootCert())
            return false;
    }

    return true;
}

#Conclusion

You'll find the complete application with UI on GitHub:

• Binaries: https://github.com/meziantou/HttpCache/releases
• Source code: https://github.com/meziantou/HttpCache

Http Cache demo

Do you have a question or a suggestion about this post? Contact me!