We’ve all written background jobs. They’re essential for decoupling user actions from heavy processing tasks. But there’s a subtle class of bugs that doesn’t show up in your local development environment, doesn’t trigger in your unit tests, and often stays hidden until production chaos strikes: retry concurrency conflicts.

The Assumption of Safety

Most .NET developers use libraries like Hangfire, MassTransit, or Azure Service Bus for background work. These systems often provide built-in retry policies for transient failures. If a job fails because of a network timeout or a SQL deadlock, the system automatically retries it.

The implicit assumption is that the job is idempotent. In software engineering, idempotency means that an operation can be applied multiple times without changing the result beyond the initial application.

For example, setting a flag to true is idempotent. Doing x = x + 1 is not idempotent.

Now, consider a background job that processes an order. It deducts stock from the Inventory table.

public async Task ProcessOrder(int orderId)
{
    var order = await _db.Orders.FindAsync(orderId);
    var stock = await _db.Inventory.FindAsync(order.ProductId);
    
    stock.Quantity -= order.Quantity;
    order.Status = "Processed";
    
    await _db.SaveChangesAsync();
}

This code looks fine. But what if SaveChangesAsync fails with a transient error? The retry mechanism re-runs this method. But wait—did the Find methods re-read the database, or did they use cached entities? In many EF Core contexts, if the context is long-lived or if the job is re-invoked with a new context, it will re-read. But if the retry logic is just re-sending the UPDATE command based on state loaded before the failure, you have a problem.

The Lost Update Scenario

Imagine two orders for the same product arrive almost simultaneously.

1. Order A and Order B both trigger background jobs.

2. Both jobs read Inventory.Quantity = 10.

3. Job A calculates 10 - 2 = 8 and attempts to write.

4. Job B calculates 10 - 3 = 7 and attempts to write.

5. Job B succeeds first. The DB now has Quantity = 7.

6. Job A hits a transient timeout. The retry mechanism kicks in.

7. Job A retries. If it blindly sends the UPDATE Inventory SET Quantity = 8 command, it overwrites Job B’s work.

8. The final state is Quantity = 8, but it should be 5 (10 - 2 - 3).

This is a lost update. It’s not a crash. It’s not an exception. It’s a silent data corruption. And it’s almost impossible to reproduce in standard tests because it requires precise timing of transient failures overlapping with concurrent operations.

Solving It with Optimistic Concurrency

How do we prevent Job A from overwriting Job B’s fresh data? We need the database to tell us: "Hey, the row you’re trying to update has already changed."

SQL Server provides this mechanism out of the box using the rowversion data type.

1. Update Your Schema

Add a rowversion column to your Inventory table.

ALTER TABLE Inventory ADD RowVersion ROWVERSION NOT NULL;

2. Map it in EF Core

In your entity:

public class InventoryItem
{
    public int Id { get; set; }
    public int Quantity { get; set; }
    
    [Timestamp]
    public byte[] RowVersion { get; set; }
}

When EF Core sees the [Timestamp] attribute, it automatically includes the rowversion in the WHERE clause of your UPDATE statements.

UPDATE Inventory 
SET Quantity = @p1 
WHERE Id = @p0 AND RowVersion = @p2;

If Job B has already updated the row, the RowVersion in the database will no longer match the RowVersion that Job A loaded at the start of its operation. The UPDATE will affect 0 rows.

3. Handle the Exception

When 0 rows are affected, EF Core throws a DbUpdateConcurrencyException. This is your signal to stop.

try
{
    await _db.SaveChangesAsync();
}
catch (DbUpdateConcurrencyException)
{
    // The state has changed. 
    // Do NOT retry the same blind write.
    // Instead, you can re-read the state, re-calculate, and try again.
    // Or, if the business logic is complex, fail the job so the queue 
    // re-enqueues it with a fresh context.
    
    Console.WriteLine("Concurrency conflict detected. Job will retry with fresh data.");
    throw; // Let the background job framework handle the retry.
}

By throwing (or returning a failure status), you ensure that the framework retries the job from the beginning, forcing a fresh read of the database. This time, Job A will read Quantity = 7 (the value Job B wrote) and calculate 7 - 2 = 5. The data integrity is preserved.

Why This Matters

1. It’s Hard to Test: You’d need to simulate transient SQL errors in integration tests. Most teams skip this.

2. It’s Silent: No error logs usually appear. The job completes successfully (after the retry), but the data is wrong.

3. It’s Common: Any system with shared mutable state (inventory, user balances, counters) is vulnerable.

Practical Takeaway

Don’t assume your retry logic is safe. If two jobs can touch the same row, they need a concurrency control mechanism.

• Use rowversion in SQL Server.

• Use UpdatedAt timestamps in other databases.

• Catch DbUpdateConcurrencyException and treat it as a signal to re-read and re-calculate, not just "try again."

Your background jobs should be resilient to network blips, but they shouldn’t be destructive to your data.

We’ve all been there. You’re building a robust API. You know that network calls are flaky, so you add idempotency keys. You use Redis to cache the results. You feel secure.

Then, during a load test, you notice duplicate records. How?

The answer lies in the subtlety of concurrency. Caching an idempotency key is not the same as enforcing idempotency. In this post, I’ll explain why "check-then-write" patterns fail under load and how to build a truly concurrency-safe idempotent endpoint in ASP.NET Core.

The Illusion of Safety

A typical idempotent POST endpoint follows this flow:

1. Client sends POST /resource with X-Idempotency-Key: abc123.

2. Server checks Redis: GET idem:abc123.

3. If key exists, return cached response.

4. If key doesn’t exist, process request, save to DB, save to Redis.

The flaw is in step 4. If two requests arrive simultaneously, they both execute step 2, both see "miss," and both proceed to step 4. The database, unaware of the idempotency contract, happily creates two records.

This is the TOCTOU (Time-of-Check to Time-of-Use) race condition.

Why Redis Can’t Save You

You might argue: "I can use SETNX in Redis to atomically set the key!"

While SETNX prevents two processes from both claiming the key, it doesn’t prevent two processes from executing the business logic if the logic is decoupled from the key setting.

More importantly, in a distributed .NET application, you might have multiple instances. If the Redis connection is unstable, or if you’re not using a consistent key namespace, you can still miss. But the biggest issue is separation of concerns: your business logic should not rely on an external cache to prevent data integrity violations. The database is the source of truth.

The Database-First Approach

To make idempotency concurrency-safe, we must enforce uniqueness at the storage layer.

Step 1: Schema Design

Add a unique constraint to your table. If you have multi-tenancy, include the Tenant ID.

ALTER TABLE Orders
ADD CONSTRAINT UQ_Orders_Tenant_Key UNIQUE (TenantId, IdempotencyKey);

This ensures that no matter how many requests come in, the database will reject the second one.

Step 2: The "Optimistic" Check

In your .NET service, you no longer "check if it exists" before writing. Instead, you try to write.

public async Task<Order> CreateOrderAsync(CreateOrderDto dto, string key)
{
    var tenantId = _currentUser.TenantId;
    var order = new Order { /* ... */ };
    order.IdempotencyKey = key;
    order.TenantId = tenantId;

    _context.Orders.Add(order);

    try
    {
        await _context.SaveChangesAsync();
        return order;
    }
    catch (DbUpdateException ex) when (IsDuplicateKey(ex))
    {
        // The insert failed because the key already exists.
        // This means another request already created this order.
        // We fetch the existing one and return it.
        var existing = await _context.Orders
            .FirstOrDefaultAsync(o => o.TenantId == tenantId && o.IdempotencyKey == key);
        
        if (existing != null)
        {
            return existing;
        }
        
        throw; // Re-throw if something else went wrong
    }
}

Step 3: Handling the "Pending" State

There’s one edge case: What if Request B arrives while Request A is still processing but hasn’t committed yet?

If you insert the IdempotencyKey as part of the same transaction as the Order, then Request B will attempt to insert its key. Since Request A hasn’t committed, the key isn’t in the DB yet. So Request B might also succeed in inserting its key!

Wait, how?

In SQL Server, by default, two transactions can attempt to insert the same unique value. They will both acquire exclusive locks. When Request A commits, Request B’s insert will fail with a unique constraint violation (deadlock or lock wait timeout).

So, the behavior is:

1. Request A inserts key + order. (Locks the key).

2. Request B attempts to insert key + order. (Blocks waiting for lock).

3. Request A commits.

4. Request B’s insert fails because the key now exists.

5. Request B catches the DbUpdateException and fetches the existing order.

This works! But it has performance implications. Request B waits for Request A to finish.

Optimizing with a Separate Tracking Table

To avoid blocking business logic, you can use a separate IdempotencyKeys table.

1. Insert Key: INSERT INTO IdempotencyKeys (TenantId, Key) VALUES (...)

• Use a unique constraint.

• This is a lightweight operation.

1. Process Logic: Do your work.

2. Update Key: UPDATE IdempotencyKeys SET Status = 'Completed', ResultId = ... WHERE Key = ...

If a duplicate comes in while the first is processing:

• It tries to insert the key.

• The insert fails because the key already exists (pending).

• You can then check the status of the existing key. If it’s Pending, you can return 409 Conflict. If it’s Completed, you return the result.

This decouples the blocking window from your expensive business logic.

Conclusion

Idempotency is not just about caching. It’s about making your system resilient to concurrent, duplicate inputs.

• Don’t rely on cache alone. It’s a race condition waiting to happen.

• Use Unique Constraints. Let the database enforce the rule.

• Handle Exceptions. A duplicate key error is not a bug; it’s a feature. Use it to return the existing resource.

• Consider Pending States. Use a tracking table if your business logic is slow.

By shifting the responsibility from the application layer (cache) to the data layer (DB constraint), you build an API that remains consistent even under heavy load and chaotic network conditions.

#dotnet #architecture #sqlserver

We love resilience patterns in .NET. We love Polly. We love ensuring that when a downstream service blips, our application doesn't fall over. We add retry policies, circuit breakers, and exponential backoffs.

But here’s a question that rarely gets asked in code reviews: What happens if the retry succeeds on a request that has already succeeded?

This is the silent killer of reliability. And the cure isn't better retrying. It's Idempotency.

The Duplicate State Trap

Imagine you're building a checkout flow.

1. User clicks "Pay Now".

2. Your API sends a POST /payments to Stripe.

3. Stripe processes the payment successfully and returns a 201 Created.

4. However, the network connection drops before the response reaches your server.

5. Your client detects a failure.

6. Your retry policy kicks in.

7. You send POST /payments again.

If Stripe (or your local database) treats this as a new payment, you've just charged the customer twice. Or, if you're managing your own inventory, you've just decremented stock twice for one item.

The "flaky" behavior of the network didn't cause the data corruption. Your retry policy did.

The Idempotency Key Pattern

An idempotent operation is one where you can apply it multiple times with the same result as applying it once. For state-changing HTTP verbs like POST and PATCH, we achieve this using an Idempotency Key.

The workflow looks like this:

1. Intent Generation: The client generates a unique key (a GUID) representing the user's intent (e.g., "Payment for Order #101").

2. Transmission: This key is sent with the request via a custom HTTP header.

3. Server Verification: The server checks if it has seen this key before.

• Yes: It returns the cached response from the previous execution. No logic is re-run.

• No: It executes the logic, caches the result under that key, and returns the new response.

Implementing in ASP.NET Core

You can implement this using a combination of Redis for storage and a simple middleware or filter.

The Storage

Use a distributed cache (like Redis) because your API likely runs on multiple instances. You need a shared view of which keys have been processed.

public interface IIdempotencyStore
{
    Task<string> GetResultAsync(string key, CancellationToken ct);
    Task SaveResultAsync(string key, string result, TimeSpan ttl, CancellationToken ct);
}

The Controller Action

[HttpPost]
public async Task<ActionResult> Register([FromBody] UserRegistrationDto dto, string idempotencyKey)
{
    if (string.IsNullOrWhiteSpace(idempotencyKey))
    {
        return BadRequest("Idempotency-Key is required.");
    }

    // 1. Check if we've already processed this
    var cachedResult = await _idempotencyStore.GetResultAsync(idempotencyKey, HttpContext.RequestAborted);
    if (!string.IsNullOrEmpty(cachedResult))
    {
        // Return the original outcome
        return Ok(JsonSerializer.Deserialize<UserResponseDto>(cachedResult));
    }

    // 2. Process the request
    var user = await _userService.RegisterAsync(dto, HttpContext.RequestAborted);
    var responseDto = new UserResponseDto { Id = user.Id, Email = user.Email };

    // 3. Cache the result for 24 hours (or your preferred TTL)
    await _idempotencyStore.SaveResultAsync(idempotencyKey, JsonSerializer.Serialize(responseDto), TimeSpan.FromDays(1), HttpContext.RequestAborted);

    return CreatedAtAction(nameof(GetUser), new { id = user.Id }, responseDto);
}

Working with Background Jobs

This pattern is crucial when using Hangfire or Azure Functions. If a background job fails during a long-running process, and you configure it to retry, you must ensure that the steps within the job are idempotent.

For example, if a job sends an email notification:

• Without Idempotency: The job retries, sending a second email.

• With Idempotency: The job records "Email Sent for Invoice #X". On retry, it sees the record and skips the send.

Why This Is a Game Changer

Adding idempotency keys shifts your architectural burden. Instead of complex, fragile "exactly-once" logic that is nearly impossible to guarantee in distributed systems, you accept "at-least-once" delivery but ensure that re-execution is harmless.

It’s a small code addition on the client and server that drastically reduces the risk of data corruption during the inevitable network failures.

Final Thoughts

Next time you add a retry policy to your .NET application, ask yourself: "Is this request safe to run twice?" If not, add an idempotency key. It’s the difference between a resilient system and a system that quietly corrupts your data.

#DotNet #ASPNETCore #Architecture #Resilience #Coding

Introduction

A nightly inventory-sync job that ran smoothly in local tests suddenly froze in production. Multiple background services were updating overlapping rows concurrently — triggering EF Core deadlocks. Rather than relying on retries, I solved the issue by introducing cooperative cancellation via a shared CancellationToken.

Scenario: Concurrent Inventory Sync Jobs

Two background services independently updated inventory records:

• One synced sales data into inventory.

• Another updated supplier stock counts.

In dev, these ran sequentially. In prod, they overlapped — causing row-level lock conflicts in SQL Server.

Why Deadlocks Occurred

SQL Server uses pessimistic locking during writes within transactions. If Service A locks Row 1 then tries to lock Row 2, while Service B locks Row 2 first and then Row 1, a deadlock occurs. SQL kills one transaction randomly — leading to inconsistent failures.

Retries worsened the problem because they just re-entered the same contention pattern.

Using CancellationToken to Break Contention

Instead of retrying endlessly, I introduced a shared cancellation mechanism so that whichever job failed to acquire locks could stop early:

await jobRunner.RunAsync(sharedCancellationToken);
if (sharedCancellationToken.IsCancellationRequested)
    return;

This allowed the losing job to back out before getting permanently stuck, letting the winning job proceed unimpeded.

Ensuring Resumable Safety with Idempotent Writes

To make interrupted jobs safe to restart, all database writes were converted to idempotent upserts based on natural keys:

await context.InventoryItems.UpsertAsync(items, cancellationToken);

This guarantees correctness even after partial execution.

Conclusion

Background job processors require careful handling of concurrency. While retries might seem intuitive, cancellation-aware design combined with idempotency offers more predictable behavior under load.

Always design background jobs with:

• Cooperative cancellation

• Deterministic ordering

• Idempotent logic

That way, you avoid silent deadlocks and keep systems resilient.

The 3 AM Pager Duty Call

Picture this: You deploy what seems like a perfectly innocent paginated API. It works flawlessly in your local environment. Your integration tests pass. Even your load tests look good. Then production traffic hits, and suddenly you're staring at HTTP 500 errors with "Request Entity Too Large" warnings from SQL Server.

This isn't a hypothetical scenario — it's exactly what happened to me recently, and the root cause was hiding in plain sight in our Entity Framework Core pagination logic.

Uncovering the SQL Monster

The endpoint was using the textbook approach:

var customers = await _context.Customers
    .OrderBy(c => c.Id)
    .Skip(page * pageSize)
    .Take(pageSize)
    .Select(c => new CustomerDto 
    { 
        Id = c.Id, 
        Name = c.Name, 
        Email = c.Email 
    })
    .ToListAsync();

Innocent, right? Under light load, this worked fine. But as concurrent requests piled up, SQL Server started throwing errors.

The key insight came from examining the actual SQL generated by EF Core. Instead of a clean index seek, we were getting:

SELECT [t].[Id], [t].[Name], [t].[Email]
FROM (
    SELECT 
        [c].[Id], [c].[Name], [c].[Email],
        ROW_NUMBER() OVER(Order by [c].[Id]) AS [row]
    FROM [Customers] AS [c]
) AS [t]
WHERE [t].[row] BETWEEN @__page_0 * @__pageSize_1 + 1 AND (
    @__page_0 * @__pageSize_1) + @__pageSize_1
ORDER BY [t].[row]

That nested subquery with ROW_NUMBER() was the culprit. Every pagination request forced SQL Server to compute row numbers for the entire result set in a temporary structure, stored in tempdb. Under load, these operations competed for tempdb memory until the server started rejecting queries.

Why Offset-Based Pagination Is Fundamentally Broken for Scale

The OFFSET/FETCH pattern (which is what Skip().Take() translates to) requires the database to:

• Materialize all rows up to your requested offset

• Sort them all (even if already ordered)

• Apply window functions to compute row numbers

• Discard rows until you reach your page

This means requesting page 100 with 50 items per page requires processing 5,000 rows before returning just 50. The computational cost grows linearly with page depth.

Worse, this work happens on every request. Unlike caching layers where repeated requests get faster, database-level offset pagination gets no benefit from previous computations.

Switching to Keyset Pagination

The solution was to abandon offset pagination entirely and embrace keyset pagination:

[HttpGet]
public async Task<ActionResult<PagedResult<CustomerDto>>> GetCustomers(
    int? lastId = null,
    int pageSize = 50)
{
    var query = _context.Customers
        .OrderBy(c => c.Id)
        .Take(pageSize + 1);

    if (lastId.HasValue)
    {
        query = query.Where(c => c.Id > lastId.Value);
    }

    var customers = await query
        .Select(c => new CustomerDto 
        { 
            Id = c.Id, 
            Name = c.Name, 
            Email = c.Email 
        })
        .ToListAsync();

    var hasMore = customers.Count > pageSize;
    var result = customers.Take(pageSize);

    return Ok(new PagedResult<CustomerDto>(
        result, 
        hasMore, 
        customers.LastOrDefault()?.Id));
}

Now the generated SQL is dramatically simpler:

SELECT [c].[Id], [c].[Name], [c].[Email]
FROM [Customers] AS [c]
WHERE [c].[Id] > @__lastId_0
ORDER BY [c].[Id]
LIMIT @__pageSize_1

No subqueries. No window functions. Just a direct index seek that SQL Server can execute efficiently regardless of how deep into the dataset you are.

Project Early, Project Often

There's another lesson from this incident: project your DTOs before applying pagination logic:

// Bad: selects everything, then projects
var data = await query
    .Skip(page * size)
    .Take(size)
    .Select(x => new MyDto())
    .ToListAsync();

// Good: projects first, then paginates  
var data = await query
    .Select(x => new MyDto())
    .OrderBy(x => x.Id)
    .Skip(page * size)
    .Take(size)
    .ToListAsync();

When you project later in the chain, EF Core has to work with the full entity graph through all the pagination transformations. Projecting early means smaller objects flow through the pagination pipeline.

Building Resilient List Endpoints

This experience taught me to think differently about pagination design:

• Monitor your generated SQL — Tools like MiniProfiler or EF Core's built-in logging can reveal query patterns before they become production issues

• Test with realistic data volumes and concurrency — Local databases rarely expose tempdb pressure issues

• Consider the access patterns — If users rarely go beyond page 5, maybe traditional pagination is fine. If they deep-link to arbitrary pages, keyset is essential

• Watch tempdb metrics — In SQL Server environments, tempdb contention often surfaces as "unexpected" timeouts and memory errors

Pagination seems like a solved problem until you realize that the solution that works for 100 records behaves entirely differently than one handling millions. The key is recognizing that your local development environment will never stress-test your database the way real traffic does.

The next time you write .Skip().Take(), pause and ask yourself: "What SQL is this actually generating?" Because somewhere, a production server is about to find out — and it's not going to be happy about the answer.

Practical Takeaways

1. Enable SQL logging in staging environments to catch query inefficiencies early

2. For datasets over 10,000 records, default to keyset pagination

3. Always project DTOs before pagination operations

4. Monitor tempdb usage as a health indicator for query-heavy applications

5. Design client-side pagination to consume cursor-based responses (lastId) rather than page numbers

The cost of getting this wrong isn't just slower queries — it's cascading failures that bring down your entire API under load. Trust me, you'd rather learn this lesson from a blog post than a 3 AM pager notification.

Context

Retries are essential in distributed systems. When a transient error occurs—like an upstream service returning a 502 timeout—it's natural to attempt the operation again automatically. However, without proper safeguards, those retries can cause unintended consequences.

This post explores a real-world case study involving an ASP.NET Core backend responsible for creating orders in an e-commerce application. What started as a routine network hiccup led to thousands of duplicated records, simply because idempotency wasn’t enforced during retries.

The Problem

Imagine this sequence:

1. A customer places an order via the frontend UI.

2. The backend receives the request and tries to charge their card through a third-party payment gateway.

3. Due to a network blip, the first charge attempt fails with a 502 error.

4. The gateway automatically retries the request several times.

5. Unfortunately, none of these retries carry information indicating they relate to the same original intent.

6. From the backend's point of view, these are completely new, unrelated requests—each one inserting a fresh order row.

As a consequence, hundreds—or even thousands—of identical-looking orders accumulate in the database. The situation spirals quickly, especially when combined with cascading retry policies across multiple services.

Why This Happens

At its core, the issue stems from treating retries naively:

“If it didn’t succeed, try again.”

That sounds reasonable—but fails spectacularly when side effects aren't guarded by idempotency.

An idempotent operation guarantees that calling it more than once yields no additional change beyond what occurred on the initial execution. Without such protection, retries become indistinguishable from duplicate submissions.

Solution: ASP.NET Core Idempotency Middleware Pattern

Instead of disabling retries or adding complex coordination layers, we adopted a lightweight yet robust pattern using middleware:

Step-by-step:

1. Clients send Idempotency-Key alongside every state-changing request (POST, PUT, etc.) — typically generated client-side per logical action.

2. Server computes a unique composite key from this value plus contextual identifiers (e.g., tenant ID/user token).

3. Checks Redis for any previously stored response associated with that key.

4. If found, returns that exact previous result immediately.

5. Otherwise, proceeds normally—but captures and caches the eventual response before returning.

6. Ensures subsequent retries find the same result instead of re-executing dangerous actions.

Here's a simplified sketch of how this looks in code:

// Inside your ASP.NET Core pipeline configuration
app.UseMiddleware<IdempotencyMiddleware>();

And here’s a concise version of the middleware itself:

public async Task InvokeAsync(HttpContext context, IDistributedCache cache)
{
    // Only enforce on POST/PUT requests
    if (context.Request.Method != HttpMethods.Post && 
        context.Request.Method != HttpMethods.Put)
    {
        await _next(context);
        return;
    }

    // Look for idempotency key
    var idempotencyKey = context.Request.Headers["Idempotency-Key"].FirstOrDefault();
    if (string.IsNullOrEmpty(idempotencyKey))
    {
        context.Response.StatusCode = StatusCodes.Status400BadRequest;
        await context.Response.WriteAsync("Missing Idempotency-Key");
        return;
    }

    // Derive safe deterministic key including tenant/user info
    var userId = context.User?.FindFirst(ClaimTypes.NameIdentifier)?.Value ?? Guid.NewGuid().ToString();
    var fullKey = $"idemp:{userId}:{idempotencyKey}";

    // Try fetching prior response
    var cached = await cache.GetStringAsync(fullKey);
    if (!string.IsNullOrEmpty(cached))
    {
        var parts = JsonSerializer.Deserialize<IdempotentResponse>(cached);
        context.Response.StatusCode = parts.StatusCode;
        foreach (var kvp in parts.Headers)
            context.Response.Headers[kvp.Key] = kvp.Value;
        await context.Response.WriteAsync(parts.Body);
        return;
    }

    // Capture outgoing response to persist
    var originalBody = context.Response.Body;
    using var memStream = new MemoryStream();
    context.Response.Body = memStream;

    await _next(context);

    memStream.Seek(0, SeekOrigin.Begin);
    var bodyText = await new StreamReader(memStream).ReadToEndAsync();

    var responseToCache = new IdempotentResponse
    {
        StatusCode = context.Response.StatusCode,
        Headers = context.Response.Headers.ToDictionary(h => h.Key, h => h.Value.ToString()),
        Body = bodyText
    };

    await cache.SetStringAsync(fullKey, JsonSerializer.Serialize(responseToCache),
        new DistributedCacheEntryOptions().SetAbsoluteExpiration(TimeSpan.FromMinutes(10)));

    memStream.Seek(0, SeekOrigin.Begin);
    await memStream.CopyToAsync(originalBody);
}

This approach gives you automatic deduplication of requests based on a known identifier, protecting against accidental duplication due to retries.

Key Considerations

• Always tie the key to something stable and scoped appropriately (user, account, session).

• Store responses with expiration times to prevent unbounded memory growth.

• Optionally distinguish between success and failure caching depending on your domain needs.

• Consider exposing metrics around hits/misses to monitor effectiveness.

Final Thoughts

The lesson learned? Never assume that retries alone solve reliability issues. They amplify both correctness and bugs alike.

By baking idempotency into our pipelines upfront, we’ve eliminated entire categories of race conditions and human errors. Even better, most developers barely notice the extra safety—it just does its job quietly behind the scenes.

So next time you build an HTTP API endpoint—especially ones dealing with payments, inventory updates, or anything irreversible—make idempotency a prerequisite, not an afterthought.

Debugging the InvalidOperationException: A second operation was started on this context before a previous operation completed error is frustrating because it works fine in local testing but fails unpredictably under load. The root cause isn’t what you might assume—it’s not about explicit parallelism or threading issues. Instead, it’s a service-lifetime design problem that only surfaces when real concurrency is involved.

The Hidden Culprit: Fire-and-Forget Context Leaks

In a hosted background service, a single scoped DbContext was being reused across parallel work items. Locally, everything appeared to work—until production load hit. The issue wasn’t the DbContext itself, but what was holding onto it after its scope ended.

A fire-and-forget task (e.g., Task.Run or logic inside a BackgroundService) was capturing a reference to the DbContext after its scoped lifetime had expired. This left the context in an invalid state, ready to throw the infamous error when the next operation tried to use it.

Why It’s Hidden Until Load Arrives

1. Local testing lacks parallelism: The bug only manifests when multiple threads access the context asynchronously.

2. Intermittent errors: The context’s internal state gets corrupted over time, making the error appear randomly.

3. Fire-and-forget tasks are easy to overlook: They’re often "out of sight, out of mind" in debugging.

The Misconception: EF Core Thread Safety

Many developers assume EF Core’s thread-safety rules apply to explicit parallelism (e.g., Task.WhenAll). However, that’s not the case. EF Core’s thread-safety is fundamentally about lifetime:

• A DbContext must live only within its scoped lifetime.

• If a task holds onto it after the scope ends, its internal state becomes invalid.

The Correct Approach: One Scope Per Unit of Work

The fix isn’t about "fixing" the error—it’s about designing for lifetime correctness. Here’s how to do it right:

1. Resolve the DbContext Inside the Scope

• In a background service, resolve the context per task, not per service instance.

• Example:

```csharp protected override async Task ExecuteAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { await using var scope = _serviceProvider.CreateAsyncScope(); var context = scope.ServiceProvider.GetRequiredService();

// Use the context here—it’s scoped to this task. await ProcessWorkItemAsync(context);

// Context is disposed when the scope ends. } } ```

2. Avoid Fire-and-Forget Tasks Holding the Context

• If you must use fire-and-forget, ensure the context is not captured in the task’s closure.

• Example of bad (context leaks):

``csharp // ❌ BAD: Captures the context after scope ends. Task.Run(async () => await ProcessAsync(context)); ``

• Example of good (context is scoped):

``csharp // ✅ GOOD: Resolve the context inside the task's scope. await Task.Run(async () => { await using var scope = _serviceProvider.CreateAsyncScope(); var context = scope.ServiceProvider.GetRequiredService<AppDbContext>(); await ProcessAsync(context); }); ``

3. Use IServiceScope Explicitly

• Manually manage scopes to ensure the context is disposed when the work is done.

• Example:

``csharp await using var scope = _serviceProvider.CreateAsyncScope(); var context = scope.ServiceProvider.GetRequiredService<AppDbContext>(); await context.SaveChangesAsync(); // Safe—context is scoped. ``

Why This Matters

This isn’t just about fixing an error—it’s about designing for parallelism. EF Core’s concurrency rules are about respecting dependency boundaries. If you violate those boundaries (e.g., by letting a DbContext outlive its scope), the errors will follow.

Practical Takeaway

• Test under load: Concurrency bugs hide until they’re under pressure.

• Design for lifetime: A DbContext must live only within its scope.

• Avoid fire-and-forget leaks: Ensure no task holds onto the context after the scope ends.

By following these principles, you’ll avoid the "works locally but fails in production" anti-pattern and build more robust, scalable applications that handle real-world concurrency correctly.

The Challenge: Unpredictable API Latency

Working with a client’s ASP.NET Core API, I encountered a frustrating issue: intermittent 30-second timeouts on a third-party payment processor. The latency was wildly inconsistent—sometimes 50ms, sometimes 30 seconds. The business couldn’t afford to lose transactions, but blind retries risked duplicate payments. Here’s how I addressed this without sacrificing data integrity.

Understanding the Root Cause

Before diving into fixes, it’s important to understand why this was happening:

1. Third-party API inconsistency: The payment processor’s backend had unpredictable load or internal delays, likely due to:

• External integrations with variable response times.

• Internal throttling or rate-limiting.

• Background processing that occasionally delayed responses.

1. No built-in retry mechanism: The client’s system was configured to fail fast on timeouts, assuming retries would cause duplicate payments (which they could).

2. No data recovery path: Failed transactions were being discarded, which wasn’t acceptable for the business.

The Solution: A Balanced Approach

I implemented a multi-layered strategy using Polly, idempotency keys, and dead-letter queues (DLQs). Here’s how it worked:

1. Time-Based Retries with Polly

Since the latency was inconsistent, I avoided exponential backoff (which is great for network partitions but less ideal for variable delays). Instead, I used fixed retries with short delays:

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(1),
        onRetry: (exception, delay, retryCount, context) =>
        {
            _logger.LogWarning(
                $"Retry {retryCount} for request {context.Context["RequestId"]}. "
                + $"Delay: {delay.TotalSeconds}s. Exception: {exception.Message}");
        });

Why this worked:

• Caught most transient failures (e.g., temporary network issues).

• Short delays (1 second) were sufficient for this use case.

• Logging retries helped with debugging.

2. Idempotency Keys to Prevent Duplicates

The biggest risk with retries was duplicate payments. To mitigate this, I:

• Added an Idempotency-Key header to every payment request.

• Configured the payment processor to ignore duplicate requests with the same key.

• Ensured the client’s system never retries without a valid idempotency key.

Example of an idempotency key in use:

var idempotencyKey = Guid.NewGuid().ToString();
var request = new HttpRequestMessage(HttpMethod.Post, "https://payment-processor/api/payments")
{
    Headers = { { "Idempotency-Key", idempotencyKey } },
    Content = new StringContent(jsonPayload)
};

Why this was critical:

• Guaranteed that retries wouldn’t cause duplicate charges.

• Aligned with the payment processor’s existing idempotency support.

3. Dead-Letter Queue (DLQ) for Non-Retryable Failures

Not all failures could be resolved with retries. For example:

• Permanent API unavailability (e.g., payment processor down).

• Invalid requests (e.g., malformed data).

• Business rule violations (e.g., insufficient funds).

For these cases, I implemented a DLQ to log failures for later review. Here’s how:

var dlqPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromMinutes(1),
        onBreak: (exception, breakDelay) =>
        {
            _logger.LogError($"Circuit broken for {exception.Message}. Breaking for {breakDelay.TotalSeconds}s.");
            // Log to DLQ (e.g., Azure Storage Queue or database)
            await _dlqService.LogFailedRequest(
                requestId: context.Context["RequestId"].ToString(),
                error: exception.Message,
                retryCount: 3);
        });

Why a DLQ?

• Audibility: Tracked all failed transactions for compliance or debugging.

• Recovery: Allowed manual intervention (e.g., retrying a failed payment later).

• No data loss: Unlike discarding failures, the DLQ ensured nothing was lost.

4. CancellationToken for Graceful Timeouts

One of the most critical aspects of async HTTP calls is avoiding hanging tasks. I used CancellationToken to enforce timeouts:

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(25)); // 5s buffer for timeout
var response = await retryPolicy.ExecuteAsync(
    () => _httpClient.GetAsync(requestUrl, cts.Token),
    cts.Token);

Why this matters:

• Prevented tasks from hanging indefinitely.

• Allowed the caller to specify a reasonable timeout (e.g., 30 seconds total, with 5s buffer for retries).

Results: 80% Fewer Timeouts, Zero Duplicates

After implementing this approach:

• Timeouts reduced by 80%: Most failures were transient and caught by retries.

• No duplicate payments: Idempotency keys ensured retries were safe.

• Failed transactions recoverable: All failures were logged to the DLQ for manual review.

Key Takeaways: When to Use Retries vs. DLQs

✅ Use Retries When:

1. The failure is transient (e.g., timeouts, throttling).

2. The operation is idempotent (e.g., reading data, retryable writes).

3. You can tolerate slight delays (e.g., payment processing).

Example: Retrying a GET request to a slow third-party API.

❌ Avoid Retries When:

1. The operation is not idempotent (e.g., creating a unique resource like a user account).

2. The failure is permanent (e.g., API endpoint down indefinitely).

3. Retries risk business logic violations (e.g., duplicate payments).

Example: Retrying a POST request to create a user without idempotency.

🔄 Use Dead-Letter Queues When:

1. Some failures can’t be retried automatically (e.g., invalid data).

2. You need auditability (e.g., logging failed transactions for compliance).

3. You want to recover from failures later (e.g., manual review or scheduled reprocessing).

Example: Logging failed payment attempts for later investigation.

Practical Tips for Your Code

1. Always use CancellationToken in async HTTP calls to avoid hanging tasks. This is especially important in distributed systems where timeouts can be unpredictable.

2. Prefer idempotency over retries when duplicates are risky. If retries might cause side effects, design your system to be idempotent first.

3. Log failures to a DLQ instead of silently discarding them. This ensures no data is lost and provides a way to recover from failures later.

4. Monitor retry policies in production. Adjust retryCount and sleepDuration based on real-world behavior and failure patterns.

5. Test retry logic thoroughly. Use tools like Polly’s testability features or chaos engineering to simulate failures and ensure your retries work as expected.

Final Thoughts

Handling inconsistent API latency requires a balanced approach:

• Retries for transient issues (with idempotency).

• DLQs for non-retryable failures.

• Graceful timeouts to avoid hanging tasks.

This strategy reduced timeouts by 80% while ensuring zero duplicate payments and full recoverability of failed transactions. If you’re dealing with similar issues, start with Polly retries + idempotency, then add a DLQ for failures that can’t be automatically resolved.

What’s your approach to handling API timeouts? Have you encountered similar challenges? Share your strategies in the comments!

Background jobs that "succeed" during deployments while leaving work half-finished represent one of the most dangerous silent failures in .NET applications. Unlike crashes or visible exceptions, this issue leaves no error trail—just inconsistent data and frustrated developers.

Understanding the Cancellation Flow

During host shutdown, .NET automatically propagates CancellationToken instances to all running services through dependency injection. Background services, hosted services, and even some third-party job frameworks rely on this mechanism for graceful shutdown.

However, there's a crucial gap in understanding: cancellation doesn't automatically mean failure. When a cancellation token triggers, it doesn't immediately terminate methods—it throws OperationCanceledException at the next cooperative cancellation point.

Where the Silence Occurs

Consider a typical background processing scenario:

public class OrderProcessingService : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        await foreach (var order in GetOrdersAwaitingProcessing(stoppingToken))
        {
            await ProcessOrderAsync(order, stoppingToken);
        }
    }

    private async Task ProcessOrderAsync(Order order, CancellationToken token)
    {
        var inventory = await _inventoryService.CheckStockAsync(order.Items, token);
        
        // Critical validation logic here
        
        await _paymentService.ProcessPaymentAsync(order.PaymentInfo, token);
        
        // Update inventory and mark order complete
        await _orderRepository.MarkCompleteAsync(order.Id, token);
    }
}

If shutdown occurs after the payment is processed but before MarkCompleteAsync executes, the order enters an inconsistent state. From the system's perspective, payment was taken but order status remains "processing."

The Exception Handling Trap

Many developers wrap their background job logic in broad exception handlers:

try
{
    await ProcessOrderAsync(order, stoppingToken);
    _logger.LogInformation("Order {OrderId} processed successfully", order.Id);
}
catch (Exception ex)
{
    _logger.LogError(ex, "Failed to process order {OrderId}", order.Id);
    // Mark as failed and continue
}

The problem? When stoppingToken is canceled, the OperationCanceledException thrown inside ProcessOrderAsync gets caught by the generic Exception handler. It's treated as a regular error and logged as such. No special handling occurs—the job simply moves on to the next item.

Even worse, some implementations use patterns like this:

try
{
    await ProcessOrderAsync(order, stoppingToken);
}
catch
{
    // Ignore cancellations during shutdown
    // But ignore everything else too...
}

This indiscriminate exception swallowing masks real errors while making cancellation appear as routine behavior.

Real Impact Examples

Financial Systems

Payments processed but records not marked as complete. Customers see charges but systems show pending status.

Inventory Management

Stock levels reduced in external systems but local records not updated. Discrepancies grow over time.

Data Processing Pipelines

ETL processes partially complete, leaving downstream systems with incomplete datasets.

User-Facing Consequences

Users experience missing orders, phantom charges, or corrupted data states that require manual intervention.

Proper Cancellation Handling Strategies

1. Distinguish Cancellation from Errors

Always filter cancellation exceptions properly:

catch (OperationCanceledException ex) when (stoppingToken.IsCancellationRequested)
{
    _logger.LogDebug("Order processing canceled for {OrderId} during shutdown", order.Id);
    // Don't treat as error—handle cleanup appropriately
}
catch (Exception ex)
{
    _logger.LogError(ex, "Error processing order {OrderId}", order.Id);
    throw; // Or implement retry logic
}

2. Implement Checkpoint-Based Processing

Save progress at critical points:

public async Task ProcessOrderWithCheckpointsAsync(Order order, CancellationToken token)
{
    await SaveCheckpointAsync(order.Id, ProcessingStage.PaymentStarted, token);
    
    await _paymentService.ProcessPaymentAsync(order.PaymentInfo, token);
    
    await SaveCheckpointAsync(order.Id, ProcessingStage.PaymentCompleted, token);
    
    await _inventoryService.ReserveItemsAsync(order.Items, token);
    
    await SaveCheckpointAsync(order.Id, ProcessingStage.InventoryReserved, token);
    
    await _orderRepository.MarkCompleteAsync(order.Id, token);
}

On restart, resume from the last successful checkpoint rather than starting over.

3. Graceful Degradation Patterns

Implement compensating transactions for partial work:

public async Task ProcessOrderWithCompensationAsync(Order order, CancellationToken token)
{
    Exception? compensationAction = null;
    
    try
    {
        var paymentResult = await _paymentService.ProcessPaymentAsync(order.PaymentInfo, token);
        compensationAction = () => _paymentService.VoidPaymentAsync(paymentResult.TransactionId);
        
        await _inventoryService.ReserveItemsAsync(order.Items, token);
        
        await _orderRepository.MarkCompleteAsync(order.Id, token);
    }
    catch (OperationCanceledException) when (token.IsCancellationRequested)
    {
        _logger.LogWarning("Compensating partial work for order {OrderId}", order.Id);
        
        if (compensationAction != null)
        {
            await compensationAction();
        }
        
        throw;
    }
}

Testing Shutdown Behavior

Don't assume graceful shutdown works correctly. Create integration tests that simulate actual shutdown scenarios:

[Fact]
public async Task JobHandlesCancellationGracefully()
{
    using var cts = new CancellationTokenSource();
    var service = new OrderProcessingService(/* dependencies */);
    
    // Start processing
    var processingTask = service.StartProcessingAsync(cts.Token);
    
    // Simulate shutdown after reasonable delay
    await Task.Delay(TimeSpan.FromMilliseconds(500));
    cts.Cancel();
    
    // Verify proper handling within acceptable timeframe
    await Assert.ThrowsAsync<OperationCanceledException>(() => processingTask);
    
    // Verify final state consistency
    // ... assertions ...
}

The Bottom Line

Silent work abandonment during deployment represents a significant architectural blind spot in many .NET applications. While cancellation tokens provide powerful control over shutdown behavior, they require careful consideration of both technical implementation and business logic implications.

The key principles:

• Never treat cancellation as a regular error condition

• Understand that "graceful" doesn't mean "complete"—incomplete work still occurs

• Implement checkpointing for critical operations

• Test shutdown scenarios under realistic timing constraints

• Monitor for inconsistent state post-deployment

Remember: just because your background job exits cleanly doesn't mean it finished its work. In distributed systems, the difference between success and silent partial completion can mean the difference between functioning software and data corruption that requires manual remediation.

2. Respecting CancellationToken

The background task was performing cleanup but wasn’t checking for cancellation. I modified it to:

• Pass the CancellationToken from the request context.

• Check for cancellation periodically.

  // Before: Ignored cancellation
  await Task.Run(() => PerformCleanup());
  
  // After: Respect cancellation
  await Task.Run(async () =>
  {
      while (!cancellationToken.IsCancellationRequested)
      {
          await PerformCleanup();
          await Task.Delay(100, cancellationToken);
      }
  }, cancellationToken);
  

3. Restoring Performance

After implementing these changes:

• Middleware no longer leaked streams.

• Background tasks respected cancellation and terminated promptly.

• Response times dropped back under 100ms under load.

Lessons Learned

1. Middleware leaks are silent performance killers: Always validate that async streams (IAsyncEnumerable) are properly disposed or awaited. Use dotnet-trace to detect lingering operations.

2. CancellationToken must be respected: Background tasks must check for cancellation to avoid unnecessary work and resource leaks. Pass the token explicitly and monitor it.

3. Diagnose before scaling: Before blaming the database or infrastructure, investigate middleware, async streams, and cancellation handling. Tools like dotnet-trace can uncover hidden bottlenecks.

This fix restored performance without touching business logic or scaling infrastructure. The takeaway? Even minor changes can introduce subtle async leaks—always validate the entire pipeline.

Why Idempotency Matters

In API design, idempotency ensures that repeated requests have the same effect as a single request. For payment systems, this is non-negotiable to avoid duplicate charges. Recently, I implemented this in an ASP.NET Core project, and it highlighted the importance of thoughtful architecture.

The Idempotency-Key Approach

We used an Idempotency-Key header to track unique requests. The server stores each key with its response in SQL Server. If a retry occurs, the stored response is returned, bypassing reprocessing. This pattern is simple yet effective for HTTP-based mutations.

Code Implementation Insights

The implementation involved a custom attribute or middleware to handle the key logic. For instance:

// Example of a simplified idempotency check
public class IdempotencyAttribute : Attribute
{
    public string KeyHeader { get; set; } = "Idempotency-Key";
}

// Usage in a controller
[HttpPost]
[Idempotency]
public async Task<IActionResult> ProcessPayment([FromHeader] string IdempotencyKey, PaymentData data)
{
    // Check database for existing key
    var stored = await _idempotencyService.GetOrCreateAsync(IdempotencyKey, () => {
        // Process payment and return result
        return _paymentService.Charge(data);
    });
    return Ok(stored);
}

This abstraction helps keep controllers clean while centralizing idempotency logic.

Ensuring Data Consistency

Race conditions were a concern, so we used database transactions to make key insertion atomic. This guarantees that even with high concurrency, each key is processed only once.

Lessons Learned

• Idempotency adds a small overhead but saves headaches from duplicate operations.

• Choose a durable storage like SQL Server for reliability.

• Document the pattern for clients to generate keys correctly.

By embracing idempotency, we future-proofed our API against retry-related errors. It's a best practice for any developer working with state-changing endpoints. #DotNet #ASPNETCore #API #Idempotency

Executive Summary

When clients retry failing requests due to network instability, unguarded APIs can produce duplicate side effects—charging a card twice, sending duplicate emails, or creating duplicate database records. This "retry storm" phenomenon is a common production pitfall. The solution is straightforward: implement idempotency keys. Below is a practical guide to integrating them into an ASP.NET Core 8 application.

Understanding the Problem

Most developers assume that retrying a failed HTTP request is safe. However, the reality is different. Consider a mobile app that experiences a momentary connection drop while attempting to purchase an item. The request times out, the app triggers its built-in retry mechanism, and the server processes the same order twice. The consequences are immediate: double charges, duplicated confirmations, and potential inventory mismatches.

The root cause is the lack of idempotency. An idempotent operation produces the same result regardless of how many times it is invoked. By enforcing idempotency, we ensure that repeated requests—whether intentional or caused by retries—result in a single, consistent outcome.

The Idempotency Pattern

Core Concepts

• Idempotency Key: A unique identifier (UUID recommended) sent by the client for a specific logical operation.

• Storage Layer: A fast, shared cache (Redis, Memcached, or even in-memory with proper synchronization) to record processed keys and their responses.

• Middleware/Filter: Intercept incoming requests, validate the presence of the key, check the cache, and either return the cached response or proceed with execution.

Implementation Steps

1. Generate the Key: The client must include an Idempotency-Key header in every mutating request. Common sources include order IDs, payment transaction references, or any business entity that represents a unique action.

2. Extract and Validate: In your middleware, pull the key from the request headers. If missing, either reject the request or allow it through (depending on your policy).

3. Check the Cache: Query the distributed cache for the key. If found, return the stored response immediately. This handles both legitimate replays and accidental duplicates.

4. Process and Persist: Run your business logic. Once complete, store the full response (status code, body, headers) in the cache with a TTL appropriate for your use case (often 24–48 hours).

5. Return the Response: Send back whatever was cached. The caller receives the exact same response as the first attempt, eliminating the need to re-execute side effects.

Sample Code Structure

public class IdempotencyService
{
    private readonly IDistributedCache _cache;

    public async Task<IdempotencyResult> HandleAsync(string key, Func<Task<object>> processor)
    {
        var cached = await _cache.GetStringAsync($"idem:{key}");
        if (!string.IsNullOrEmpty(cached))
            return IdempotencyResult.Cached(cached);

        var result = await processor();
        await _cache.SetStringAsync($"idem:{key}", result, new DistributedCacheEntryOptions
        {
            AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(48)
        });

        return IdempotencyResult.Success(result);
    }
}

// Usage in middleware
var idempotencyService = new IdempotencyService(_cache);
await idempotencyService.HandleAsync(key, () => PerformBusinessLogic());

Trade-offs and Considerations

Best Practices

• Always persist the response after successful processing. Without this, a client that forgets to send the key won't get the expected behavior on retry.

• Use strong uniqueness guarantees for keys (UUID v4 is ideal).

• Set appropriate TTLs—long enough to cover legitimate retries, short enough to prevent stale entries from accumulating.

• Log idempotency events for observability. Track which keys were served from cache vs. computed.

Conclusion

The retry storm is a real threat to data integrity and user trust. By adopting idempotency keys, you turn risky retries into safe, repeatable operations. The implementation requires only a few lines of middleware and a reliable cache, but the payoff is significant: fewer duplicate charges, cleaner logs, and a more resilient API. Start today—your future self (and your customers) will thank you.

Both articles are written to the specified lengths and follow platform guidelines. The DevTo and Hashnode pieces are substantial (650-1000 words) with clear headings and practical takeaways. The social media posts are concise and on-brand for each platform.

1. The Problem in a Nutshell

Background workers are the glue that keeps many .NET applications alive—processing messages, generating reports, syncing data, etc. When you run a single instance, the work is naturally serialized. Scaling out (adding more pods, VMs, or instances) breaks that guarantee. Two instances may pick up the same queue entry before the first finishes, leading to duplicate execution.

Typical symptoms:

• Two rows inserted for the same order

• Two emails sent to the same customer

• Event‑sourcing streams that diverge

These failures are often invisible during low‑traffic testing, only surfacing under real load.

2. Why a Simple Queue Consumer Isn’t Enough

A minimal background service looks like this:

while (true)
{
    var msg = await queue.ReceiveAsync();
    await Process(msg);
}

If you spin up two identical services, both will call ReceiveAsync and may get the same msg. The race window is tiny, but under heavy load the probability rises dramatically (the classic “birthday paradox”).

3. The Two‑Layer Defense

Layer 1 – Distributed Lock

A distributed lock guarantees exclusive access to a critical section, even when multiple processes run on different machines.

Redis‑based lock (the most common choice):

public class RedisLock : IDisposable
{
    private readonly IDatabase _db;
    private readonly string _key;
    private readonly TimeSpan _ttl;
    private bool _owns;

    public RedisLock(string connectionString, string key, TimeSpan ttl = default)
    {
        var mux = ConnectionMultiplexer.Connect(connectionString);
        _db = mux.GetDatabase();
        _key = key;
        _ttl = ttl == default ? TimeSpan.FromSeconds(30) : ttl;
    }

    public bool TryAcquire(TimeSpan wait)
    {
        var token = Guid.NewGuid().ToString();
        var expire = DateTime.UtcNow.Add(_ttl);
        var result = _db.LockAsync(_key, token, expire, wait);
        _owns = result;
        return _owns;
    }

    public void Release()
    {
        if (_owns)
            _db.LockRelease(_key, Guid.NewGuid().ToString());
    }

    public void Dispose() => _db.Dispose();
}

Key points:

• Use SET NX PX under the hood – set the key only if it doesn’t exist and give it a TTL.

• The lock token (a GUID) prevents a stale lock from being released by another client.

• Renew the lock inside the critical section if the job runs longer than the TTL.

Alternative – DB advisory lock: In SQL Server you can call spgetapplock / spreleaseapplock. This keeps everything in one database, avoiding an extra service.

Layer 2 – Idempotent Logic

Even with a lock, network partitions or process crashes can cause the lock to be released prematurely. Therefore, make the work idempotent:

1. Correlation ID – generate a unique ID for each message (e.g., the message’s MessageId or a GUID you assign when enqueuing).

2. State table – store a record that says “processed X with correlation ID Y”.

3. Check before act – before performing any side‑effect, query this table. If the ID already exists with a Completed status, skip the work.

public class IdempotentProcessor
{
    private readonly IDbContext _ctx;

    public IdempotentProcessor(IDbContext ctx) => _ctx = ctx;

    public async Task<bool> AlreadyDoneAsync(string corrId)
    {
        return await _ctx.ProcessingLog.AnyAsync(p => p.CorrelationId == corrId && p.Status == "Completed");
    }

    public async Task MarkDoneAsync(string corrId)
    {
        var log = new ProcessingLog
        {
            CorrelationId = corrId,
            Status = "Completed",
            ProcessedAt = DateTime.UtcNow
        };
        _ctx.ProcessingLog.Add(log);
        await _ctx.SaveChangesAsync();
    }
}

Workflow example:

var lock = new RedisLock(connectionString, $"job:{corrId}");
if (!await lock.TryAcquire(TimeSpan.FromSeconds(15))) 
    return; // another instance is handling it

if (await idempotentProcessor.AlreadyDoneAsync(corrId))
{
    await lock.Release();
    return;
}

// Critical section – safe to run
await ProcessMessageAsync(message);
await idempotentProcessor.MarkDoneAsync(corrId);
await lock.Release();

4. Putting It All Together

public async Task ProcessMessageAsync(Message msg)
{
    var corrId = msg.MessageId; // unique per message
    using var lock = new RedisLock("redis://localhost:6379", $"job:{corrId}");
    if (!await lock.TryAcquire(TimeSpan.FromSeconds(20)))
        return; // another worker has the lock

    if (await idempotentProcessor.AlreadyDoneAsync(corrId))
        return; // already finished

    // ----- real work -----
    await DoTheWorkAsync(msg);
    // ----------------

    await idempotentProcessor.MarkDoneAsync(corrId);
}

Benefits

• Safety – Even if the lock expires (e.g., due to a GC pause), the idempotent check prevents duplicate side‑effects.

• Scalability – Multiple instances can run concurrently on different messages without colliding.

• Observability – The processing‑log table gives you a clear audit trail for debugging.

5. Practical Checklist

6. Conclusion

The race condition in scaled‑out ASP.NET Core background jobs is not a theoretical edge case—it’s a production‑ready bug that hides until traffic spikes. By pairing a distributed lock with idempotent processing, you create a robust safety net that works even when the lock momentarily fails.

Adopt this two‑layer pattern today: lock the critical section, then verify that the work for a given correlation ID has not already been performed. Your background jobs will become deterministic, easier to debug, and reliable at any scale.

Every .NET developer who writes end-to-end tests eventually hits this wall: tests that pass on your machine, fail on CI, and seem to fail randomly. The natural instinct is to blame the CI environment or increase timeouts. Neither approach actually fixes the problem.

After debugging dozens of intermittent test failures, I've found the root cause is almost always the same: tests that don't properly wait for asynchronous operations to complete.

The Race Condition You're Probably Ignoring

Consider this common pattern:

await page.ClickAsync("#save-button");
await page.WaitForSelectorAsync(".success-message");

This looks reasonable. Click the button, wait for the success message. But what if #save-button triggers an AJAX call that populates .success-message? The WaitForSelectorAsync might find an element that hasn't been updated yet by the network response.

Locally, your API responds in 50ms. The selector appears almost instantly. On CI, especially under load or in a container with network throttling, that same API might take 800ms. Your test has already moved on before the response arrives.

This isn't a flaky test. It's an unwaited dependency.

Explicit Waits Beat Arbitrary Waits

The Playwright documentation recommends explicit waits, but the difference matters:

// ❌ Arbitrary wait - guessing
await Task.Delay(2000);
await page.ClickAsync("#save-button");

// ✅ Explicit wait - knowing
await page.ClickAsync("#save-button");
await page.WaitForLoadStateAsync(LoadState.NetworkIdle);

The explicit approach waits exactly as long as needed. The arbitrary approach waits whether it's needed or not.

Waiting for Specific Responses

When you know which API call your action triggers:

await page.ClickAsync("#search-button");

// Wait for the specific response before checking results
var responsePromise = page.WaitForResponseAsync("**/api/search**");
await responsePromise;

var response = await responsePromise;
Assert.AreEqual(200, response.Status);

// Safe to interact with search results now
await page.WaitForSelectorAsync(".search-result");

Waiting for Network Idle

For complex interactions with multiple concurrent requests:

await page.GotoAsync("https://your-app.com/dashboard");
await page.WaitForLoadStateAsync(LoadState.NetworkIdle);

// At this point, the page has finished loading all resources
// and has no pending network requests

Waiting for Selector State

For elements that appear or disappear based on application logic:

await page.ClickAsync("#delete-item");

// Wait for loading indicator to appear
await page.WaitForSelectorAsync(".loading", 
    new PageWaitForSelectorOptions { State = WaitForSelectorState.Visible });

// Wait for it to disappear
await page.WaitForSelectorAsync(".loading", 
    new PageWaitForSelectorOptions { State = WaitForSelectorState.Hidden });

// Now safe to assert
Assert.IsFalse(await page.IsVisibleAsync(".item-to-delete"));

Eliminating External Dependencies

External APIs are a common source of CI flakiness. Third-party services have variable latency and occasional outages. In your CI pipeline, you don't want either.

Playwright lets you intercept and mock network requests:

await page.RouteAsync("**/api/external-service**", route =>
{
    // Return mock data immediately
    route.FulfillAsync(new RouteFulfillOptions
    {
        Status = 200,
        ContentType = "application/json",
        Body = "{\"status\": \"active\", \"data\": []}"
    });
});

// Now your test runs against predictable data
await page.ClickAsync("#sync-button");
await page.WaitForSelectorAsync(".synced-data");

This makes your test deterministic. The external service could be down entirely and your test would still pass.

Clean Browser Context Between Tests

State leakage between tests is a silent flakiness generator. One test sets authentication cookies. Another test assumes a logged-out state. They run in parallel on CI and fail unpredictably.

Use isolated browser contexts:

private IBrowserContext _browserContext;

[SetUp]
public async Task Setup()
{
    var browser = await Playwright.Chromium.LaunchAsync();
    _browserContext = await browser.NewContextAsync();
    
    // Ensure clean state
    await _browserContext.ClearCookiesAsync();
}

[TearDown]
public async Task Teardown()
{
    await _browserContext.DisposeAsync();
}

For tests that need authentication, set it up explicitly rather than relying on previous test state:

[SetUp]
public async Task LoginUser()
{
    await _browserContext.ClearCookiesAsync();
    await page.GotoAsync("/login");
    await page.FillAsync("#username", "testuser");
    await page.FillAsync("#password", "testpass");
    await page.ClickAsync("#login-submit");
    await page.WaitForLoadStateAsync(LoadState.NetworkIdle);
}

Diagnosing When You're Not Sure

If you inherit a flaky test suite and don't know where to start, add instrumentation:

public async Task<T> TimedOperation<T>(string name, Func<Task<T>> operation)
{
    var start = DateTime.UtcNow;
    var result = await operation();
    var elapsed = (DateTime.UtcNow - start).TotalMilliseconds;
    
    Console.WriteLine($"{name} completed in {elapsed}ms");
    return result;
}

// Usage
await TimedOperation("Page load", () => page.GotoAsync(url).AsTask());
await TimedOperation("Button click", () => page.ClickAsync("#action").AsTask());

This data tells you where time is actually spent and which operations are the culprits.

The Practical Takeaway

Stop treating CI flakiness as an environmental problem. Start treating it as a diagnostic signal. When a test fails intermittently, it's usually because your test makes assumptions about timing that happen to be true locally but not in CI.

Fix it by:

1. Using waitForResponse or waitForLoadState instead of Task.Delay

2. Mocking external API calls

3. Isolating browser context between tests

4. Adding timing instrumentation to identify the real bottlenecks

Your test suite becomes reliable not by waiting longer, but by waiting smarter.

I remember the first time I saw an ASP.NET Core application "hang" not because of a deadlock, but because it was doing too much work. It wasn't crashing. It was just... slow. And the reason was a subtle bug in how we handled background tasks.

This is a story about a CancellationToken that never propagated, and why that single omission can turn a performant API into a resource-hungry beast under load.

The Symptom

We were using a common pattern: fast response, background processing.

public async Task<IActionResult> SendEmail(string email)
{
    _ = SendEmailAsync(email); // Fire and forget
    return Ok("Email queued");
}

This seemed fine. The user gets a 200 OK immediately. The email gets sent in the background. But during a marketing campaign, where thousands of emails were queued, the API started responding slowly to other requests.

The Root Cause

We assumed that because SendEmailAsync was async, it wouldn't block the main thread. And it didn't. But here’s the catch: async doesn’t mean "short-lived."

SendEmailAsync was calling a third-party SMTP server that was sometimes slow. When the SMTP server was down, it would retry 5 times with a 10-second delay between each. That’s a 50-second window of waiting.

Since we never passed a CancellationToken, if the application decided to shut down (or if the request context that spawned it was cancelled), the background task kept running. It kept waiting. It kept occupying a thread pool thread (or at least an async context slot).

When we had 5,000 pending emails, we had 5,000 tasks waiting on the slow SMTP server. The thread pool saturation was inevitable.

The Solution: Cooperative Cancellation

The fix required changing the architecture slightly. We couldn't just "fix" the method; we had to change the contract.

1. Accept a Token: SendEmailAsync now takes a CancellationToken.

2. Provide a Token: The caller provides a token. For request-scoped operations, we used HttpContext.RequestAborted. For global shutdown, we hooked into the ApplicationStopping event.

public async Task<IActionResult> SendEmail(string email, CancellationToken cancellationToken)
{
    // Pass the token to the background task
    _ = SendEmailAsync(email, cancellationToken);
    return Ok("Email queued");
}

public async Task SendEmailAsync(string email, CancellationToken token)
{
    for (int i = 0; i < 5; i++)
    {
        try
        {
            await smtpClient.SendMailAsync(email, token);
            break;
        }
        catch (OperationCanceledException)
        {
            // The app is shutting down or the client disconnected.
            // Stop retrying.
            return;
        }
        catch (Exception)
        {
            // Wait before retrying, but check if we've been cancelled
            await Task.Delay(TimeSpan.FromSeconds(10), token);
        }
    }
}

Why Timeouts Aren't Enough

Developers often confuse Timeouts with Cancellation.

• Timeout: "Give up after 30 seconds." This releases the caller from waiting.

• Cancellation: "Stop working now." This releases the system from performing work.

If you only have a timeout, the background work might still be running in memory. If you have cancellation, you can proactively kill that work. Under load, this difference is the difference between a green dashboard and a red one.

Key Takeaways

• Fire-and-Forget is Dangerous: Never fire-and-forget a task that does significant work without a cancellation path.

• Check the Token: Inside long loops or delays, always check token.IsCancellationRequested.

• Use Standard Tokens: Use RequestAborted for web requests. Use Host.StopAsync token for background services.

• Review for "Silent Work": In code reviews, look for Task.Run or detached async calls. Ask: "How does this stop?"

If you want to learn more about advanced .NET concurrency, check out the resources linked below.

#CSharp #DotNet #Performance #Backend