Filter Application Insights events in ASP.NET Core

Application Insights ingests lots of data: requests, traces, events, metrics, etc. If your web site has lots of users, the amount of data can be huge. This means you pay for this data. While Application Insights is cheap, you may want to reduce the bill. One way is to sample the data. In short, you send only xx% of the events. This is the simplest solution. The other solution is to filter low-value events for your usage. For instance, if you use Application Insights to diagnose errors, you may want to filter success requests.

The Application Insights SDK provides an interface to manipulate the telemetry events: ITelemetryProcessor. This interface allows changing the events, for instance, to add a property or remove sensible data. If also allow filtering the event. Telemetry processors are chained. Each telemetry processor handles the event and passes it to the next processor. A telemetry processor can also choose to discard the event by not sending it to the next processor. We'll use this to filter the events.

First, create a class that implements ITelemetryProcessor. This class will filter request events with status code 200.

public class MyTelemetryProcessor : ITelemetryProcessor
{
    private ITelemetryProcessor _next;

    public MyTelemetryProcessor(ITelemetryProcessor next)
    {
        // Next TelemetryProcessor in the chain
        _next = next;
    }

    public void Process(ITelemetry item)
    {
        if (item is RequestTelemetry request)
        {
            if (request.ResponseCode == "200")
            {
                // Filter the event
                return;
            }
        }

        // Send the item to the next TelemetryProcessor
        _next.Process(item);
    }
}

Then, you have to configure Application Insights to use the MyTelemetryProcessor. In the Startup.cs file, add the following code in the Configure method:

public void Configure(IApplicationBuilder app)
{
    var configuration = app.ApplicationServices.GetService<TelemetryConfiguration>();
    configuration.TelemetryProcessorChainBuilder.Use(next => new MyTelemetryProcessor(next));
    configuration.TelemetryProcessorChainBuilder.Build();

    // ...
}

Your filter is now configured and active. Now you can customize the filter to remove other kinds of events. For instance, you can remove some specific traces based on their category:

public void Process(ITelemetry item)
{
    if (item is TraceTelemetry trace)
    {
        if (trace.Properties.TryGetValue("CategoryName", out var category) &&
            category == "Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware")
        {
            return;
        }
    }

    _next.Process(item);
}

If you have a public website, you may have lots of requests to the WordPress login page /wp-login.php. If you are not using a WordPress website, you can filter those events:

public void Process(ITelemetry item)
{
    if (item is RequestTelemetry request)
    {
        // The url is not handled, so 404
        if (string.Equals(request.ResponseCode, "404", StringComparison.Ordinal))
        {
            var path = request.Url?.AbsolutePath;
            if (path != null && path.EndsWith("/wp-login.php", StringComparison.OrdinalIgnoreCase))
            {
                return;
            }
        }
    }

    Next.Process(item);
}

#Conclusion

The Application Insights SDK allows you to control the data sent to Azure. You can augment events with custom data, edit the events to remove sensitive data, or filter unneeded events. So, you can better detect & diagnose issues and understand usage for your web apps.

#Additional resources

• Filtering and preprocessing telemetry in the Application Insights SDK

Do you have a question or a suggestion about this post? Contact me!