Caching http requests for faster debugging

When working with an HTTP service, repeatedly sending real requests to the server can slow down your tests and debugging workflow. One option is to mock the calls with fake data. You can easily mock the HttpClient class in C# (see Mocking an HttpClient using an HttpClientHandler), but this requires a separate setup for each project and a fake response for every request. A better alternative is to create a local proxy that automatically caches responses and serves them on subsequent requests.

How Http caching with FiddlerCore works

To create a proxy in .NET, you can use Telerik FiddlerCore, the library behind Fiddler.

Progress® Telerik® FiddlerCore Embedded Engine is a .NET Class library you can integrate into your .NET Framework applications. There is also a .NET Standard 2.0 flavor of FiddlerCore that allows using it on any platform implementing .NET Standard.

Telerik FiddlerCore allows you to capture and modify HTTP and HTTPS traffic just like Fiddler, without any of the Fiddler UI.

Once you download the zip, add a reference to FiddlerCore45.dll. Then start the proxy:

private static Proxy _proxyEndpoint;

public static void StartProxy()
{
    // Register for the FiddlerCore events
    FiddlerApplication.BeforeRequest += OnBeforeRequest;
    FiddlerApplication.AfterSessionComplete += OnAfterSessionComplete;

    // Start the proxy
    var startupSettings = new FiddlerCoreStartupSettingsBuilder()
        .ListenOnPort(8877)
        .RegisterAsSystemProxy()
        .DecryptSSL()
        .ChainToUpstreamGateway()
        .MonitorAllConnections()
        .OptimizeThreadPool()
        .Build();

    FiddlerApplication.Startup(startupSettings);
    _proxyEndpoint = FiddlerApplication.CreateProxyEndpoint(48721, true, "localhost");
}

Handle the AfterSessionComplete event to save responses to the local cache:

private static readonly object FromCache = new object();

private static void OnAfterSessionComplete(Session session)
{
    // We'll tag the session where the response comes from the cache, so we do not recache them...
    if (session.Tag == FromCache)
        return;

    // Allows to filter requests we want to cache
    if (!ShouldProcess(session))
        return;

    // Save the response to the disk
    var cacheFile = GetCacheFile(session);
    Directory.Create(Path.GetDirectoryName(cacheFile));
    session.SaveResponse(cacheFile, bHeadersOnly: false);
}

private static string GetCacheFile(Session session)
{
    using (var sha1 = SHA1.Create())
    {
        var hash =  Convert.ToBase64String(sha1.ComputeHash(Encoding.UTF8.GetBytes(url)));
        return Path.Combine(Path.GetTempPath(), "HttpCache", "Cache", hash + ".cache")
    }
}

private static bool ShouldProcess(Session session)
{
    if (session.RequestMethod.EqualsIgnoreCase("CONNECT"))
        return false;

    // TODO add your own logic here
    // You can filter on the method, the url, the headers or the body
    return true;
}

The final step is to intercept requests and serve cached responses using the BeforeRequest event:

private static void OnBeforeRequest(Session session)
{
    if (!ShouldProcess(session))
        return;

    var file = GetCacheFile(session);
    if (File.Exists(file))
    {
        // Indicate FiddlerCore to not send the request to the server, we'll fill the response
        session.utilCreateResponseAndBypassServer();

        // Set the response from the file
        session.LoadResponseFromFile(file);

        // Tag the session to indicate the response comes from the cache
        session.Tag = FromCache;
    }
}

Finally, stop the proxy when the application exits:

public static void Stop()
{
    if (_proxyEndpoint != null)
    {
        _proxyEndpoint.Detach();
        _proxyEndpoint.Dispose();
    }

    FiddlerApplication.BeforeRequest -= OnBeforeRequest;
    FiddlerApplication.AfterSessionComplete -= OnAfterSessionComplete;
    if (FiddlerApplication.IsStarted())
    {
        FiddlerApplication.Shutdown();
    }
}

#Supporting https traffic

To handle HTTPS requests, you need to install Fiddler's root certificate. The NuGet package includes everything needed to automate this:

public static bool InstallCertificate()
{
    if (!CertMaker.rootCertExists())
    {
        if (!CertMaker.createRootCert())
            return false;

        if (!CertMaker.trustRootCert())
            return false;
    }

    return true;
}

#Conclusion

You'll find the complete application with UI on GitHub:

• Binaries: https://github.com/meziantou/HttpCache/releases
• Source code: https://github.com/meziantou/HttpCache

Http Cache demo

Do you have a question or a suggestion about this post? Contact me!