*"How I Fixed a 300ms API Latency Spike—Without Touching the Database"*

# How I Fixed a 300ms API Latency Spike—Without Touching the Database

A 300ms spike in API response times after deploying a new validation rule? No database changes, no external calls—just a sudden performance hit. The culprit turned out to be a combination of **`IAsyncEnumerable` leaks in middleware** and **`CancellationToken` misuse** in background tasks. Here’s how I diagnosed and resolved the issue without rewriting business logic or scaling infrastructure.

---

## The Mystery: Where Did the Latency Come From?

After deploying a minor validation tweak, response times under load increased from ~80ms to ~380ms. No database queries or external API calls were modified. The issue was subtle but critical:

- **Middleware was leaking `IAsyncEnumerable` streams**, causing async operations to persist even after the request completed.
- **A background task was ignoring `CancellationToken`**, preventing timely cleanup of resources.

This created a cascading effect: middleware operations lingered, background tasks ran unnecessarily, and response times degraded silently.

---

## Diagnosing the Issue: `dotnet-trace` to the Rescue

To pinpoint the root cause, I followed these steps:

1. **Reproduced the issue** under load using Playwright, simulating concurrent requests.
2. Ran `dotnet-trace collect --providers Microsoft-AspNetCore.Hosting:Verbose` to capture detailed middleware execution.
3. Analyzed the trace for:
   - Unexpected async stream operations (`IAsyncEnumerable`) lingering after request completion.
   - Background tasks that weren’t respecting `CancellationToken`.

The trace revealed:
- Middleware wasn’t properly disposing of async streams, causing them to remain active.
- A background task was performing work even after the request was canceled, due to ignored `CancellationToken`.

---

## The Solution: Cleanup and Proper Cancellation Handling

### 1. Fixing the Middleware Leak
The middleware was using `IAsyncEnumerable` to process validation rules but wasn’t ensuring streams were properly disposed. I updated it to:
- Use `await foreach` with explicit disposal or ensure streams were awaited to completion.
  ```csharp
  // Before: Potential leak
  await foreach (var item in asyncEnumerable)
  {
      // Process item
  }

  // After: Ensure cleanup
  await foreach (var item in asyncEnumerable.ConfigureAwait(false))
  {
      // Process item
  }

2. Respecting CancellationToken

The background task was performing cleanup but wasn’t checking for cancellation. I modified it to:

• Pass the CancellationToken from the request context.

• Check for cancellation periodically.

  // Before: Ignored cancellation
  await Task.Run(() => PerformCleanup());
  
  // After: Respect cancellation
  await Task.Run(async () =>
  {
      while (!cancellationToken.IsCancellationRequested)
      {
          await PerformCleanup();
          await Task.Delay(100, cancellationToken);
      }
  }, cancellationToken);
  

3. Restoring Performance

After implementing these changes:

• Middleware no longer leaked streams.

• Background tasks respected cancellation and terminated promptly.

• Response times dropped back under 100ms under load.

Lessons Learned

1. Middleware leaks are silent performance killers: Always validate that async streams (IAsyncEnumerable) are properly disposed or awaited. Use dotnet-trace to detect lingering operations.

2. CancellationToken must be respected: Background tasks must check for cancellation to avoid unnecessary work and resource leaks. Pass the token explicitly and monitor it.

3. Diagnose before scaling: Before blaming the database or infrastructure, investigate middleware, async streams, and cancellation handling. Tools like dotnet-trace can uncover hidden bottlenecks.

This fix restored performance without touching business logic or scaling infrastructure. The takeaway? Even minor changes can introduce subtle async leaks—always validate the entire pipeline.