Extending BenchmarkDotNet - Custom Exporters, Metrics & CI/CD

Ever wondered if your application is running as efficiently as possible? Benchmarking is the key to unlocking performance insights, and BenchmarkDotNet is the ultimate tool for .NET developers. But what if the built-in features don’t quite fit your needs? That’s where extending BenchmarkDotNet comes in!

By customizing BenchmarkDotNet, you can tailor performance reporting, track specialized metrics, and integrate benchmarking into CI/CD pipelines to catch performance regressions before they impact users.

In this post, we’ll explore how to extend BenchmarkDotNet in three key areas:

• Custom Exporters – Formatting benchmark results to fit your needs.
• Custom Metrics – Tracking and reporting specific performance indicators.
• CI/CD Integration – Automating performance tests and detecting regressions.

Let’s dive in!

Creating Custom Exporters

Why Use Custom Exporters?

By default, BenchmarkDotNet provides multiple exporters (Markdown, CSV, JSON, etc.). However, sometimes you need to format benchmark results in a way that suits your reporting or dashboarding needs.

Implementing a Custom Exporter

A custom exporter in BenchmarkDotNet implements the IExporter interface. Here’s a basic example:

using BenchmarkDotNet.Exporters;
using BenchmarkDotNet.Loggers;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;
using System;

public class CustomJsonExporter : IExporter
{
    public string Name => "CustomJsonExporter";
    
    public void ExportToLog(Summary summary, ILogger logger)
    {
        var json = System.Text.Json.JsonSerializer.Serialize(summary.BenchmarksCases);
        logger.WriteLine(json);
    }
}

// Registering the exporter in a benchmark class
[Config(typeof(CustomBenchmarkConfig))]
public class MyBenchmark
{
    [Benchmark]
    public void TestMethod() { /* Your benchmark code here */ }
}

public class CustomBenchmarkConfig : ManualConfig
{
    public CustomBenchmarkConfig()
    {
        AddExporter(new CustomJsonExporter());
    }
}

This exporter serializes the benchmark results to JSON and logs them. You can modify it to write data to a file, a database, or an API.

Custom Metrics

Why Custom Metrics Matter

BenchmarkDotNet collects standard metrics (execution time, memory allocation, GC pressure), but sometimes you need additional performance indicators such as CPU usage, I/O operations, or cache misses.

Implementing a Custom Metric

To define a custom metric, implement the IMetricDescriptor interface and use it in a IColumnProvider:

using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;
using System.Diagnostics;

public class CpuUsageMetric : IMetricDescriptor
{
    public string Id => "CPU_Usage";
    public string DisplayName => "CPU Usage (%)";
    public string Legend => "Tracks CPU usage during benchmarking";
    public bool TheHigherTheBetter => false;
    public UnitType UnitType => UnitType.Dimensionless;
}

public class CpuUsageColumn : IColumn
{
    public string Id => "CPU_Usage";
    public string ColumnName => "CPU (%)";
    public bool AlwaysShow => true;
    
    public string GetValue(Summary summary, BenchmarkCase benchmarkCase)
    {
        var cpuUsage = GetCpuUsage(); // Custom method to measure CPU usage
        return cpuUsage.ToString("F2");
    }
    
    private double GetCpuUsage()
    {
        using (var process = Process.GetCurrentProcess())
        {
            return process.TotalProcessorTime.TotalMilliseconds / process.StartTime.ToUniversalTime().Millisecond;
        }
    }
}

// Registering the custom metric
public class CustomMetricConfig : ManualConfig
{
    public CustomMetricConfig()
    {
        AddColumn(new CpuUsageColumn());
    }
}

Now, when running benchmarks, you’ll see CPU usage reported alongside execution time and memory allocations.

Integration with CI/CD

Why Automate Benchmarking?

Integrating BenchmarkDotNet into CI/CD helps detect performance regressions early. By running benchmarks in a controlled environment and comparing results over time, you can prevent unintended slowdowns from making it into production.

Automating Performance Tests in CI/CD

You can configure your CI/CD pipeline to execute benchmarks and compare results using GitHub Actions or Azure DevOps Pipelines. Here’s a simple example for GitHub Actions:

name: Benchmark Tests

on:
  pull_request:
    branches:
      - main

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup .NET
        uses: actions/setup-dotnet@v2
        with:
          dotnet-version: '8.0'
      - name: Run Benchmarks
        run: dotnet run --project MyBenchmarkProject -c Release

Regression Detection in CI/CD

To track performance changes, you can store benchmark results and compare them across runs. One approach is to:

• Store JSON results as artifacts.
• Compare current results with previous runs.
• Fail the pipeline if a performance regression is detected.

Conclusion: Supercharge Your Performance Benchmarking

Extending BenchmarkDotNet gives you full control over performance testing, helping you fine-tune reporting, track advanced metrics, and automate benchmarking in CI/CD. Whether you’re looking to prevent performance regressions, collect insightful metrics, or generate custom reports, these techniques will supercharge your benchmarking workflow.

Now, it’s your turn! Have you implemented any custom extensions in BenchmarkDotNet? Share your experiences in the comments—let’s learn together!