Calling Other Services: HttpClient, Retries, Circuit Breaker – Web API | Extra 8
Introduction
Since Lesson 94, the SMS queues a text message for every fee receipt and reminder. But no text has ever left the building: the senders from Lesson 41 only build the words.
This lesson connects the SMS to an outside text-message service over HTTP. That service lives on someone else's servers, so it will sometimes be slow, sometimes busy, and sometimes down. The SMS must cope with all three.
By the end of this lesson, you will be able to:
- Call another service with a typed client from
IHttpClientFactory. - Decide which failures are worth a retry, and which are not.
- Add time limits, retries with backoff, and a circuit breaker with one call.
- Use an idempotency key, so a retry never sends the same text twice.
- Test all of this without a network, and use the same retry idea for a SQL deadlock.
Why this matters
Almost every real backend calls other services: payments, text messages, email, maps, identity checks. The network between them fails in small ways every day, and an API that assumes it never fails will fail too.
The obvious fixes cause new problems. Retrying everything can send a parent the same fee reminder three times. Waiting forever for a slow service blocks the worker that sends every other message.
This lesson shows the safe pattern, step by step, and proves each step with a stand-in service that fails on purpose.
The concept
Think of the school office phoning the bus company to book a trip. If the line is busy, the clerk calls again in a minute, then waits two minutes, then four. Calling every second would only make the line busier.
If the call drops after the clerk gave the details, the clerk calls back and says "this is about booking 42". The bus company sees it already has booking 42 and does not book a second bus.
If the line has been dead all morning, the clerk stops trying for half an hour and does other work. Each of these habits has a name in software, and this lesson builds all three.
Example: one HttpClient factory for the whole app
HttpClient sends HTTP requests from C#. It looks simple, but creating a new one for every request is a classic
mistake: each one opens new network connections, and a busy server can run out of them.
Keeping one HttpClient forever has the opposite problem: it never notices when the service's address changes.
IHttpClientFactory solves both. It reuses connections and renews them every two minutes by default.
// 1. Wrong: a new client, and new connections, for every message.
using HttpClient client = new HttpClient();
// 2. Risky: one client forever. It never notices a changed DNS address.
static readonly HttpClient Shared = new HttpClient();
// 3. Right: a typed client. The factory creates its HttpClient and manages the connections.
builder.Services.AddHttpClient<ITextMessageGateway, TextGatewayClient>();
A typed client is a class that takes an HttpClient in its constructor and knows one service's API. The rest
of the app only sees an interface.
Example: which failures deserve a retry
A retry only helps when the next attempt could give a different answer. A busy service may answer next second; a wrong request will be wrong forever.
| Answer | Retry? | Why |
|---|---|---|
| 500, 502, 503, 504 | Yes | The service has a problem that may pass |
| 408 Request Timeout, 429 Too Many Requests | Yes | Wait, then try again |
| No answer in time, or a broken connection | Yes | The network or the service was slow |
| 400, 401, 403, 404, 409, 422 | No | The request itself is wrong; it will fail the same way again |
This is exactly the list that .NET's standard resilience handler uses, so you rarely write it yourself.
Example: backoff and jitter
Backoff means waiting longer after each failure: about 1 second, then 2, then 4. A struggling service gets time to recover instead of more traffic.
Jitter adds a small random part to each wait. Without it, a thousand clients that failed at the same moment would all retry at the same moment too, and knock the service over again.
Example: the idempotency key
Here is the dangerous case. The SMS sends a text, the service sends it to the parent, and then the answer gets lost on the way back. The SMS sees a timeout and tries again.
Without help, the parent gets two identical texts. An idempotency key prevents this: every message carries a unique id in a header, and the same message keeps the same id on every attempt.
POST /messages Idempotency-Key: 3edaf435-... (answer lost: the SMS times out)
POST /messages Idempotency-Key: 3edaf435-... (the service: "already sent, not sending again")
An operation that gives the same result no matter how often it runs is called idempotent. The key turns a risky POST into an idempotent one.
Example: the circuit breaker
When a service is completely down, retries only waste time: each message waits seven seconds to fail. A circuit breaker counts failures and, once there are too many, stops calling for a while.
Closed is normal: calls go through. Open means every call fails at once, without touching the network. Half-open lets one trial call through to see if the service is back.
Apply it to the SMS
Step 1: a gateway that can wait for the network
INotificationSender from Lesson 41 has string Send(...). That was fine for building text, but a network call
must be async, so the thread is free while the service answers. So the SMS gets a new interface for texts.
using SchoolApp.Core.Notifications;
namespace SchoolApp.Core.Interfaces;
// Extra 8: sends a text message through an outside service. Unlike INotificationSender (Lesson 41), which
// only builds the text, this goes over the network, so it is async and it can fail.
public interface ITextMessageGateway
{
Task SendAsync(TextMessage message, CancellationToken cancellationToken = default);
}
namespace SchoolApp.Core.Notifications;
// Extra 8: one text message for the outside service. Key is the idempotency key: the same message always has
// the same key, so when a request is sent again after a failure, the service can tell it is not a new message.
public sealed record TextMessage(string Key, string To, string Text);
Every queued message also gets its own id, made once when it is queued. That id becomes the idempotency key.
public sealed record GuardianMessage(string InvoiceNumber, MessageReason Reason)
{
// Extra 8: a new id for every message queued; it becomes the text message's idempotency key.
public Guid Id { get; init; } = Guid.NewGuid();
}
GuardianMessageService from Lesson 94 now sends texts through the gateway. Email is still only built.
Show src/SchoolApp.Core/Services/GuardianMessageService.cs
using SchoolApp.Core.Interfaces;
using SchoolApp.Core.Notifications;
namespace SchoolApp.Core.Services;
// Lesson 94: sends one queued message. The pieces come from earlier lessons: the notices from Lesson 40
// write the text, the factory from Lesson 56 picks email or phone, and the senders from Lesson 41 send it.
// Extra 8: a text message now really leaves the SMS, through ITextMessageGateway. INotificationSender could not
// do that job: its Send is not async, and a network call must be. Email is still only built.
public class GuardianMessageService
{
private readonly IFeeReminderRepository _reminders;
private readonly ITextMessageGateway _texts;
public GuardianMessageService(IFeeReminderRepository reminders, ITextMessageGateway texts)
{
_reminders = reminders;
_texts = texts;
}
public async Task<MessageResult> SendAsync(GuardianMessage message, CancellationToken cancellationToken = default)
{
FeeContact? contact = await _reminders.GetContactAsync(message.InvoiceNumber, cancellationToken);
if (contact is null)
{
return new MessageResult(MessageOutcome.NoPrimaryContact);
}
// Checked again now: the guardian may have paid between queuing and sending.
if (message.Reason == MessageReason.FeeReminder && contact.Balance <= 0)
{
return new MessageResult(MessageOutcome.NothingDue);
}
INotificationSender sender = NotificationSenderFactory.For(contact.Guardian);
FeeNotice notice = sender.Channel == "Email" ? new EmailFeeNotice() : new PhoneFeeNotice();
string text = notice.Compose(contact.Guardian, contact.StudentName, contact.Balance);
if (sender.Channel == "Phone")
{
// The message's id is the idempotency key, so a retry can never text the guardian twice.
await _texts.SendAsync(new TextMessage(message.Id.ToString(), contact.Guardian.Phone, text), cancellationToken);
}
else
{
sender.Send(contact.Guardian, text);
}
return new MessageResult(MessageOutcome.Sent, contact.Guardian.Id, sender.Channel);
}
}
Step 2: the typed client
The client knows one thing: how to ask the service to send a text. It puts the key in the Idempotency-Key
header and throws if the final answer is an error.
using System.Net.Http.Json;
using SchoolApp.Core.Interfaces;
using SchoolApp.Core.Notifications;
namespace SchoolApp.Api.Clients;
// Extra 8: a typed client. It knows only the outside service's API: one POST per text message.
// The HttpClient it receives comes from IHttpClientFactory, already set up in TextGateway.cs with the
// service's address, time limits, and retries, so none of that is repeated here.
public sealed class TextGatewayClient : ITextMessageGateway
{
private readonly HttpClient _http;
public TextGatewayClient(HttpClient http)
{
_http = http;
}
public async Task SendAsync(TextMessage message, CancellationToken cancellationToken = default)
{
using HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, "/messages")
{
Content = JsonContent.Create(new { to = message.To, text = message.Text })
};
// The same key on every attempt: if an earlier attempt did reach the service, it will not send again.
request.Headers.Add("Idempotency-Key", message.Key);
using HttpResponseMessage response = await _http.SendAsync(request, cancellationToken);
// Still an error after every retry: throw, and the worker logs the message as failed (event 3003).
response.EnsureSuccessStatusCode();
}
}
Notice what is missing: no address, no timeout, no retry loop. Those belong to the HttpClient the factory
gives it, set up in one place.
Step 3: time limits, retries, and the circuit breaker
AddStandardResilienceHandler (package Microsoft.Extensions.Http.Resilience) wraps every request in five
layers. The SMS changes only the numbers.
using Microsoft.Extensions.Http.Resilience;
using Microsoft.Extensions.Options;
using SchoolApp.Api.Options;
using SchoolApp.Core.Interfaces;
namespace SchoolApp.Api.Clients;
// Extra 8: how the SMS talks to the outside text-message service.
public static class TextGateway
{
public static IHttpClientBuilder AddTextGateway(this IServiceCollection services)
{
services.AddOptions<TextGatewayOptions>()
.BindConfiguration(TextGatewayOptions.SectionName)
.ValidateDataAnnotations()
.ValidateOnStart();
// IHttpClientFactory creates the HttpClient for every TextGatewayClient. It reuses the network
// connections underneath, so many clients never run out of sockets, and it renews them now and then,
// so a changed DNS address is picked up.
IHttpClientBuilder client = services.AddHttpClient<ITextMessageGateway, TextGatewayClient>((provider, http) =>
{
http.BaseAddress = provider.GetRequiredService<IOptions<TextGatewayOptions>>().Value.BaseUrl;
});
client.AddStandardResilienceHandler(Configure);
return client;
}
// The standard handler wraps every request in five layers, outermost first: a rate limiter, a total time
// limit, retries, a circuit breaker, and a time limit per attempt. These are the SMS's numbers.
public static void Configure(HttpStandardResilienceOptions options)
{
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(30); // all attempts together
options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5); // one attempt
// Up to 3 more attempts after the first, waiting about 1, 2, then 4 seconds, with a random part
// (jitter) so that many senders do not all retry at the same moment.
options.Retry.MaxRetryAttempts = 3;
options.Retry.Delay = TimeSpan.FromSeconds(1);
// When at least half of the last 10 or more attempts in 30 seconds failed, stop calling the service
// for 30 seconds. Failing at once is better than making every message wait for a service that is down.
options.CircuitBreaker.FailureRatio = 0.5;
options.CircuitBreaker.MinimumThroughput = 10;
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);
}
}
builder.Services.AddTextGateway(); // Extra 8: texts go out through an outside service, over HTTP
"TextGateway": {
"BaseUrl": "http://localhost:5181"
},
The layers work from the outside in. The total limit (30 s) covers everything. Inside it, the retry layer repeats failed attempts, the circuit breaker counts them, and each single attempt gets 5 seconds.
Step 4: a text service that fails on demand
To see all this for real, extras/e8-http-client/text-gateway.cs stands in for the outside service on port
5181. It prints every attempt, remembers every key, and has switches that make it fail.
Show extras/e8-http-client/text-gateway.cs
// Extra 8: a stand-in for an outside text-message service, so you can watch the SMS API's retries for real.
// POST /messages sends a text (here: prints it). The Idempotency-Key header makes it safe to repeat: a key that
// was already sent is answered with the first message's id, and nothing is sent twice.
// Trouble on demand:
// POST /trouble/fail?next=2 the next 2 messages get 503 Service Unavailable
// POST /trouble/slow?next=1 the next message is sent, but its answer takes 8 seconds
// POST /trouble/down every message gets 503 until POST /trouble/up
// Run with: dotnet run extras/e8-http-client/text-gateway.cs
#:sdk Microsoft.NET.Sdk.Web
#:property PublishAot=false
using System.Collections.Concurrent;
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders(); // only this file's own lines, below
WebApplication app = builder.Build();
ConcurrentDictionary<string, string> sentByKey = new ConcurrentDictionary<string, string>();
Trouble trouble = new Trouble();
int attempts = 0;
void Print(string line) => Console.WriteLine($"{DateTime.Now:HH:mm:ss.f} {line}");
app.MapPost("/messages", async (TextRequest text, HttpRequest request) =>
{
int attempt = Interlocked.Increment(ref attempts);
string key = request.Headers["Idempotency-Key"].ToString();
if (key.Length == 0)
{
return Results.BadRequest(new { error = "Send an Idempotency-Key header." });
}
if (trouble.Down || trouble.TakeFailure())
{
Print($"#{attempt} key {key[..8]} 503 (pretending to be overloaded)");
return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
}
if (sentByKey.TryGetValue(key, out string? firstId))
{
Print($"#{attempt} key {key[..8]} already sent as {firstId}: not sent again");
return Results.Ok(new { id = firstId });
}
string id = $"MSG-{sentByKey.Count + 1:D4}";
sentByKey[key] = id;
Print($"#{attempt} key {key[..8]} SENT {id} to {text.To}: {text.Text}");
if (trouble.TakeSlow())
{
Print($"#{attempt} key {key[..8]} (holding the answer for 8 seconds)");
await Task.Delay(TimeSpan.FromSeconds(8));
}
return Results.Accepted(value: new { id });
});
app.MapPost("/trouble/fail", (int next) => { trouble.Fail(next); Print($"trouble: the next {next} message(s) get 503"); });
app.MapPost("/trouble/slow", (int next) => { trouble.Slow(next); Print($"trouble: the next {next} answer(s) take 8 seconds"); });
app.MapPost("/trouble/down", () => { trouble.Down = true; Print("trouble: DOWN, every message gets 503"); });
app.MapPost("/trouble/up", () => { trouble.Down = false; Print("trouble: UP again"); });
Print("Text gateway listening on http://localhost:5181");
app.Run("http://localhost:5181");
public record TextRequest(string To, string Text);
public class Trouble
{
private int _failNext;
private int _slowNext;
public volatile bool Down;
public void Fail(int next) => Interlocked.Exchange(ref _failNext, next);
public void Slow(int next) => Interlocked.Exchange(ref _slowNext, next);
// Each call uses up one planned failure (or slow answer), if any are left.
public bool TakeFailure() => Interlocked.Decrement(ref _failNext) >= 0;
public bool TakeSlow() => Interlocked.Decrement(ref _slowNext) >= 0;
}
We started both apps, signed in as the accountant, and pressed "send reminders" (POST /api/fee-reminders,
Lesson 94) four times, with a different switch each time.
Example: two failures, then success
First, POST /trouble/fail?next=2. The service's console:
19:21:39.7 #1 key c1fe6697 503 (pretending to be overloaded)
19:21:40.5 #2 key c1fe6697 503 (pretending to be overloaded)
19:21:42.5 #3 key c1fe6697 SENT MSG-0001 to 9876540103: Diya Patel: fee balance 25500.00. NexCoding Public School
19:21:42.6 #4 key 9772745e SENT MSG-0002 to 9876540105: Kabir Singh: fee balance 45500.00. NexCoding Public School
... (8 more, one attempt each)
Three attempts, one key, one text. The waits were 0.8 and 2.0 seconds: the backoff, with jitter. The SMS's log (shortened) shows the same story from its side:
warn: Polly[3] Execution attempt. Source: 'ITextMessageGateway-standard//Standard-Retry', Result: '503', Handled: 'True', Attempt: '0'
warn: Polly[3] Execution attempt. Source: 'ITextMessageGateway-standard//Standard-Retry', Result: '503', Handled: 'True', Attempt: '1'
info: Polly[3] Execution attempt. Source: 'ITextMessageGateway-standard//Standard-Retry', Result: '202', Handled: 'False', Attempt: '2'
info: SchoolApp.Api.BackgroundJobs.GuardianMessageWorker[3001] FeeReminder message for invoice INV-2026-000002 sent to guardian 3 by Phone
Example: the lost answer
Next, POST /trouble/slow?next=1. The service sends the first text, then holds its answer for 8 seconds:
19:22:03.1 #13 key 3edaf435 SENT MSG-0011 to 9876540103: Diya Patel: fee balance 25500.00. NexCoding Public School
19:22:03.1 #13 key 3edaf435 (holding the answer for 8 seconds)
19:22:09.5 #14 key 3edaf435 already sent as MSG-0011: not sent again
After 5 seconds the SMS gave up on attempt one (OnTimeout in its log) and tried again with the same key. The
service recognised the key, answered with the first message's id, and sent nothing.
This is the case that would text a parent twice. With a new key on each attempt, attempt #14 would have been
SENT MSG-0012.
Example: the service is down
Then POST /trouble/down, on a freshly started API:
19:23:22.9 #33 key 6ceb1888 503 (pretending to be overloaded)
19:23:23.8 #34 key 6ceb1888 503 (pretending to be overloaded)
19:23:25.2 #35 key 6ceb1888 503 (pretending to be overloaded)
19:23:27.8 #36 key 6ceb1888 503 (pretending to be overloaded)
19:23:27.9 #37 key de9f7920 503 (pretending to be overloaded)
...
19:23:33.8 #42 key a76f45d0 503 (pretending to be overloaded)
fail: GuardianMessageWorker[3003] FeeReminder message for invoice INV-2026-000002 failed System.Net.Http.HttpRequestException: ... 503
fail: GuardianMessageWorker[3003] FeeReminder message for invoice INV-2026-000003 failed System.Net.Http.HttpRequestException: ... 503
fail: Polly[0] Resilience event occurred. EventName: 'OnCircuitOpened', Source: 'ITextMessageGateway-standard//Standard-CircuitBreaker'
fail: GuardianMessageWorker[3003] FeeReminder message for invoice INV-2026-000004 failed Polly.CircuitBreaker.BrokenCircuitException: The circuit is now open and is not allowing calls.
... (6 more, all in the same second)
The first two messages used all four attempts each. During the third, the tenth failed attempt opened the circuit. The remaining seven failed at once, without calling the service at all.
Finally, POST /trouble/up, and after the 30-second break, the button again. The log showed OnCircuitHalfOpened,
then OnCircuitClosed, and all ten texts went out on the first attempt.
Example: what the retries do not solve
Look again at the seven messages that failed while the circuit was open. They are logged and gone. Worse, each
one's FeeReminders row was already written, so the evening job will not try them again for seven days.
Retries handle failures that last seconds. An outage that lasts an hour needs the messages to be stored and sent later.
Teams usually keep them in an "outbox" table with a status, and a background job retries the unsent ones. The SMS does not do this yet; it is the next step in any real project.
Step 5: tests without a network
The tests use the client exactly as Program.cs registers it. Only the network is replaced: a scripted handler
answers each attempt and remembers every request. The tests skip the waits between attempts, so they run in
milliseconds.
Show tests/SchoolApp.Tests/Unit/TextGatewayClientTests.cs
using System.Net;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly.CircuitBreaker;
using SchoolApp.Api.Clients;
using SchoolApp.Core.Interfaces;
using SchoolApp.Core.Notifications;
namespace SchoolApp.Tests.Unit;
// Extra 8: the text gateway client with its real retries, time limits, and circuit breaker. Only the network
// is replaced: a scripted handler answers each attempt and keeps a copy of every request it received.
public class TextGatewayClientTests
{
private static readonly TextMessage Reminder = new TextMessage("key-1", "9800000021", "Kiara Thomas: fee balance 39500.");
private sealed class ScriptedHandler : HttpMessageHandler
{
private readonly Func<int, CancellationToken, Task<HttpResponseMessage>> _answer;
public ScriptedHandler(Func<int, CancellationToken, Task<HttpResponseMessage>> answer)
{
_answer = answer;
}
public List<(string Key, string Body)> Received { get; } = [];
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
string body = await request.Content!.ReadAsStringAsync(cancellationToken);
lock (Received)
{
Received.Add((request.Headers.GetValues("Idempotency-Key").Single(), body));
}
return await _answer(Received.Count, cancellationToken);
}
}
private static Task<HttpResponseMessage> Answer(HttpStatusCode status) => Task.FromResult(new HttpResponseMessage(status));
// The client exactly as Program.cs registers it, except that the tests do not wait between attempts,
// and one attempt may take half a second instead of five.
private static ServiceProvider Create(ScriptedHandler handler)
{
ServiceCollection services = new ServiceCollection();
services.AddSingleton<IConfiguration>(new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["TextGateway:BaseUrl"] = "http://gateway.test" })
.Build());
services.AddTextGateway().ConfigurePrimaryHttpMessageHandler(() => handler);
services.PostConfigureAll<HttpStandardResilienceOptions>(options =>
{
options.Retry.Delay = TimeSpan.Zero;
options.AttemptTimeout.Timeout = TimeSpan.FromMilliseconds(500);
});
return services.BuildServiceProvider();
}
[Fact]
public async Task ServiceErrors_AreRetried_WithTheSameKey()
{
ScriptedHandler handler = new ScriptedHandler((attempt, _) =>
Answer(attempt <= 2 ? HttpStatusCode.ServiceUnavailable : HttpStatusCode.Accepted));
await using ServiceProvider services = Create(handler);
await services.GetRequiredService<ITextMessageGateway>().SendAsync(Reminder);
Assert.Equal(3, handler.Received.Count); // 503, 503, then 202
Assert.All(handler.Received, request => Assert.Equal("key-1", request.Key));
Assert.Single(handler.Received.Select(request => request.Body).Distinct());
}
[Fact]
public async Task ASlowAttempt_IsCancelled_AndRetried()
{
// The first attempt never answers, like a service that received the message but whose answer is lost.
ScriptedHandler handler = new ScriptedHandler(async (attempt, cancellationToken) =>
{
if (attempt == 1)
{
await Task.Delay(Timeout.Infinite, cancellationToken);
}
return new HttpResponseMessage(HttpStatusCode.Accepted);
});
await using ServiceProvider services = Create(handler);
await services.GetRequiredService<ITextMessageGateway>().SendAsync(Reminder);
Assert.Equal(2, handler.Received.Count);
Assert.Equal(handler.Received[0], handler.Received[1]); // same key, same text: the service can ignore the copy
}
[Fact]
public async Task ABadRequest_IsNotRetried()
{
ScriptedHandler handler = new ScriptedHandler((_, _) => Answer(HttpStatusCode.BadRequest));
await using ServiceProvider services = Create(handler);
HttpRequestException error = await Assert.ThrowsAsync<HttpRequestException>(
() => services.GetRequiredService<ITextMessageGateway>().SendAsync(Reminder));
Assert.Equal(HttpStatusCode.BadRequest, error.StatusCode);
Assert.Single(handler.Received); // the same wrong request would fail the same way again
}
[Fact]
public async Task AServiceThatKeepsFailing_OpensTheCircuit_AndLaterCallsFailAtOnce()
{
ScriptedHandler handler = new ScriptedHandler((_, _) => Answer(HttpStatusCode.ServiceUnavailable));
await using ServiceProvider services = Create(handler);
ITextMessageGateway gateway = services.GetRequiredService<ITextMessageGateway>();
// Keep sending until the circuit breaker refuses; every message tries up to 4 times.
for (int message = 1; message <= 5; message++)
{
try
{
await gateway.SendAsync(Reminder);
}
catch (BrokenCircuitException)
{
break;
}
catch (HttpRequestException)
{
}
}
int attemptsSoFar = handler.Received.Count;
await Assert.ThrowsAsync<BrokenCircuitException>(() => gateway.SendAsync(Reminder));
Assert.Equal(10, attemptsSoFar); // MinimumThroughput: 10 attempts, all failed
Assert.Equal(attemptsSoFar, handler.Received.Count); // the open circuit did not call the service
}
}
The API tests must never call a real service, so ApiFactory swaps in a fake that just keeps the messages:
// Extra 8: text messages are kept in Texts, never sent to the outside service.
builder.ConfigureTestServices(services => services.AddSingleton<ITextMessageGateway>(Texts));
Passed! - Failed: 0, Passed: 194, Skipped: 0, Total: 194, Duration: 8 s - SchoolApp.Tests.dll (net10.0)
Five new tests: four for the client, and one checking that the message id becomes the key. The circuit breaker test confirms the number we set: it opened after exactly 10 failed attempts.
Example: the same idea for the database
Retries are not only for HTTP. In Lesson 25 you met deadlocks: two transactions each waiting for the other. SQL Server ends one by cancelling it with error 1205, and running it again usually works.
The demo makes a deadlock on purpose in two temporary tables, first without a retry, then with a Polly pipeline (the library under the standard handler) that retries only error 1205.
Show extras/e8-http-client/deadlock-retry.cs
// Extra 8: the same retry idea, for the database. SQL Server ends a deadlock by cancelling one of the two
// transactions (error 1205, "chosen as the deadlock victim"). Running that transaction again usually works,
// so 1205 is worth a retry. Most other SQL errors, such as a broken rule (50000+) or a duplicate key, are not.
// The demo makes a deadlock on purpose: two transactions update the same two rows in opposite order.
// The tables are global temporary tables in tempdb, so SchoolDb is not changed.
// Run with: dotnet run extras/e8-http-client/deadlock-retry.cs
#:package Microsoft.Data.SqlClient@6.1.7
#:package Polly.Core@8.4.2
#:property PublishAot=false
using System.Diagnostics;
using Microsoft.Data.SqlClient;
using Polly;
using Polly.Retry;
string connectionString = Environment.GetEnvironmentVariable("SCHOOLDB_CONNECTION")
?? "Server=localhost;Database=SchoolDb;Trusted_Connection=True;TrustServerCertificate=True;";
Stopwatch clock = Stopwatch.StartNew();
void Print(string line) => Console.WriteLine($"{clock.Elapsed.TotalSeconds,5:0.0}s {line}");
// This connection stays open to the end: a global temporary table lives as long as the session that made it.
await using SqlConnection setup = new SqlConnection(connectionString);
await setup.OpenAsync();
await new SqlCommand("""
DROP TABLE IF EXISTS ##FeeCounter; DROP TABLE IF EXISTS ##Receipts;
CREATE TABLE ##FeeCounter (Id INT PRIMARY KEY, Total DECIMAL(10,2));
CREATE TABLE ##Receipts (Id INT PRIMARY KEY, LastNumber INT);
INSERT ##FeeCounter VALUES (1, 0); INSERT ##Receipts VALUES (1, 0);
""", setup).ExecuteNonQueryAsync();
// One transaction: update one table, wait a moment, then update the other.
async Task Update(string name, string first, string second)
{
await using SqlConnection connection = new SqlConnection(connectionString);
await connection.OpenAsync();
await using SqlTransaction transaction = connection.BeginTransaction();
await new SqlCommand($"""
UPDATE {first} SET {(first == "##FeeCounter" ? "Total = Total + 100" : "LastNumber = LastNumber + 1")} WHERE Id = 1;
WAITFOR DELAY '00:00:01';
UPDATE {second} SET {(second == "##FeeCounter" ? "Total = Total + 100" : "LastNumber = LastNumber + 1")} WHERE Id = 1;
""", connection, transaction).ExecuteNonQueryAsync();
await transaction.CommitAsync();
Print($"{name}: committed");
}
// Round 1: no retry. One of the two is chosen as the victim and its work is lost.
Print("Round 1, without a retry");
Task a = Update("A", "##FeeCounter", "##Receipts");
Task b = Update("B", "##Receipts", "##FeeCounter");
try
{
await Task.WhenAll(a, b);
}
catch (SqlException error)
{
Print($"failed: error {error.Number}");
}
// Round 2: each transaction runs inside a retry pipeline that handles only error 1205.
ResiliencePipeline retryDeadlocks = new ResiliencePipelineBuilder()
.AddRetry(new RetryStrategyOptions
{
ShouldHandle = new PredicateBuilder().Handle<SqlException>(error => error.Number == 1205),
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
OnRetry = arguments =>
{
Print($"deadlock victim, running the whole transaction again (retry {arguments.AttemptNumber + 1})");
return ValueTask.CompletedTask;
}
})
.Build();
Print("Round 2, with a retry for error 1205");
await Task.WhenAll(
retryDeadlocks.ExecuteAsync(async _ => await Update("A", "##FeeCounter", "##Receipts")).AsTask(),
retryDeadlocks.ExecuteAsync(async _ => await Update("B", "##Receipts", "##FeeCounter")).AsTask());
SqlCommand totals = new SqlCommand("SELECT (SELECT Total FROM ##FeeCounter), (SELECT LastNumber FROM ##Receipts);", setup);
await using SqlDataReader reader = await totals.ExecuteReaderAsync();
await reader.ReadAsync();
Print($"Total {reader.GetDecimal(0)}, LastNumber {reader.GetInt32(1)}: 3 transactions committed (1 in round 1, 2 in round 2)");
0.4s Round 1, without a retry
2.1s A: committed
2.1s failed: error 1205
2.1s Round 2, with a retry for error 1205
7.1s B: committed
7.1s deadlock victim, running the whole transaction again (retry 1)
8.3s A: committed
8.3s Total 300.00, LastNumber 3: 3 transactions committed (1 in round 1, 2 in round 2)
Two rules make this safe. Retry the whole transaction from the start, never one statement in the middle. And retry only 1205: a broken rule such as error 50013 will break the same rule again.
Common mistakes
new HttpClient()for every call. A busy server runs out of connections. Fix:IHttpClientFactoryand typed clients.- No time limit. The default
HttpClienttimeout is 100 seconds; one slow service stalls every message. Fix: a short limit per attempt and a total limit. - Retrying everything. A 400 or 404 fails again, slower. Fix: retry only errors that can pass.
- Retrying a POST without an idempotency key. Lost answers turn into duplicate texts or payments. Fix: one key per operation, the same on every attempt.
- Retries without a circuit breaker. During an outage every caller waits for every retry. Fix: open the circuit and fail fast.
- Treating retries as storage. A long outage still loses the work. Fix: an outbox table and a later retry.
In real projects
A fee system added retries to its payment-gateway client in a hurry, after a few failed payments. A week later, during a slow afternoon at the gateway, some parents were charged twice.
The first attempts had worked; only the answers had been slow. Each retry looked like a brand-new payment, and the gateway charged it.
The fix was an idempotency key made from the invoice and the payment attempt, which the gateway already supported. The duplicate charges stopped, and the team wrote a rule: no retry on a POST without a key.
Best practices
- Use
IHttpClientFactorywith a typed client per outside service, behind an interface. - Start from
AddStandardResilienceHandlerand tune its numbers to the service. - Set a time limit per attempt and a total limit, shorter than whatever waits for the answer.
- Retry only errors that can pass; use backoff and jitter.
- Send an idempotency key with every operation that changes something.
- Watch the circuit breaker's events in the logs, and keep outside services out of readiness checks (Lesson 95).
- Store work that must not be lost, and retry it later from a background job.
Recap
- Texts now leave the SMS through
ITextMessageGatewayand the typedTextGatewayClient. - The standard handler gives each request a 30 s total limit, up to 3 retries with backoff and jitter, a circuit breaker, and 5 s per attempt.
- The message id is the idempotency key, so the lost-answer case sent one text, not two.
- After 10 failed attempts the circuit opened, and later messages failed at once until the service came back.
- Retries do not replace storage: messages that fail during an outage are lost until the SMS gets an outbox.
- The same retry idea, limited to error 1205, recovers a SQL deadlock victim. 194 tests pass.
Homework
Library module task: When a librarian adds a book by ISBN, the library fills in the title from an outside book-lookup service. That service is unreliable.
- In one single-file app, run a lookup service on
http://localhost:5108withGET /isbn/{isbn}. It answers 503 to the first 2 requests for each ISBN, then the title; an unknown ISBN gets 404. - Write a typed client,
BookLookupClient, withFindAsync(isbn)that returnsnullfor 404. - Register it with
AddStandardResilienceHandler: 3 retries, 200 ms first delay. - Look up one known and one unknown ISBN, and print what the service received.
A correct result looks like:
Looking up 9780143039655
lookup service: request 1 for 9780143039655
lookup service: request 2 for 9780143039655
lookup service: request 3 for 9780143039655
-> Malgudi Days
Looking up 9780000000000
lookup service: request 1 for 9780000000000
-> not found
Show the solution
// Library homework, Extra 8: the library fills in a new book's title from an outside book-lookup service.
// The file runs both sides: a lookup service on http://localhost:5108 that fails the first 2 requests for
// every ISBN (503), and a typed client, BookLookupClient, with the standard resilience handler.
// A known ISBN: 503, 503, then 200: found on the third attempt.
// An unknown one: 404 at once, with no retries: asking again cannot change "this book does not exist".
// Run with: dotnet run library-assignment/e8-library-book-lookup.cs
#:sdk Microsoft.NET.Sdk.Web
#:package Microsoft.Extensions.Http.Resilience@10.10.0
#:property PublishAot=false
using System.Collections.Concurrent;
using System.Net;
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.WebHost.UseUrls("http://localhost:5108");
// The library's side: a typed client from IHttpClientFactory, with retries.
builder.Services.AddHttpClient<BookLookupClient>(http => http.BaseAddress = new Uri("http://localhost:5108"))
.AddStandardResilienceHandler(options =>
{
options.Retry.MaxRetryAttempts = 3;
options.Retry.Delay = TimeSpan.FromMilliseconds(200);
});
WebApplication lookup = builder.Build();
// The outside service.
Dictionary<string, string> titles = new Dictionary<string, string>
{
["9780143039655"] = "Malgudi Days",
["9788173711466"] = "Wings of Fire"
};
ConcurrentDictionary<string, int> requests = new ConcurrentDictionary<string, int>();
lookup.MapGet("/isbn/{isbn}", (string isbn) =>
{
int request = requests.AddOrUpdate(isbn, 1, (_, count) => count + 1);
Console.WriteLine($" lookup service: request {request} for {isbn}");
if (!titles.TryGetValue(isbn, out string? title))
{
return Results.NotFound();
}
return request <= 2 ? Results.StatusCode(503) : Results.Ok(new BookInfo(isbn, title));
});
await lookup.StartAsync();
BookLookupClient books = lookup.Services.GetRequiredService<BookLookupClient>();
foreach (string isbn in new[] { "9780143039655", "9780000000000" })
{
Console.WriteLine($"Looking up {isbn}");
BookInfo? book = await books.FindAsync(isbn);
Console.WriteLine(book is null ? " -> not found" : $" -> {book.Title}");
}
await lookup.StopAsync();
public record BookInfo(string Isbn, string Title);
public sealed class BookLookupClient
{
private readonly HttpClient _http;
public BookLookupClient(HttpClient http)
{
_http = http;
}
// null when the service says the book does not exist; an exception when it is still failing after the retries.
public async Task<BookInfo?> FindAsync(string isbn)
{
using HttpResponseMessage response = await _http.GetAsync($"/isbn/{isbn}");
if (response.StatusCode == HttpStatusCode.NotFound)
{
return null;
}
response.EnsureSuccessStatusCode();
return await response.Content.ReadFromJsonAsync<BookInfo>();
}
}
The unknown ISBN got one request and no retries: 404 is not on the retry list, and asking again cannot make the book exist.
Quiz
Why not create a new HttpClient for every request?
Show answer
Answer: B. Each one opens new connections, and a busy server can run out of them IHttpClientFactory reuses connections and renews them every two minutes by default.
Which answer is worth a retry?
Show answer
Answer: C. 503 Service Unavailable A busy or restarting service may answer next time; a wrong request fails the same way again.
The service sent the text, but its answer was lost and the SMS retried. Why did the parent get only one text?
Show answer
Answer: B. The retry carried the same Idempotency-Key, so the service did not send again The gateway answered 'already sent as MSG-0011'.
What does an open circuit breaker do?
Show answer
Answer: B. Fails calls at once, without calling the service, until the break is over In our run, seven messages failed in the same second instead of waiting seven seconds each.
Why retry only error 1205 in the deadlock demo?
Show answer
Answer: B. A deadlock victim usually succeeds when run again; other errors such as a broken rule fail again Retry what can pass, and always rerun the whole transaction.
Interview questions
Questions are grouped by experience level. Click a question to see a model answer. All questions: Web API interview questions.
Fresher (0-2 yrs)
F Why should you not create a new HttpClient for every request? concept
Each new HttpClient opens new network connections, and closed ones linger for a while, so a busy server can
run out of them. One HttpClient kept forever has the opposite problem: it never notices a changed DNS address.
IHttpClientFactory reuses connections and renews them every two minutes by default.
F Which HTTP failures are worth retrying, and which are not? concept
Retry errors that can pass: 5xx answers, 408, 429, timeouts, and broken connections. Do not retry 400, 401, 403, 404, 409, or 422: the request itself is wrong and will fail the same way again.
F What is a typed client? coding
A class that takes an HttpClient in its constructor and knows one service's API, registered with
AddHttpClient<TInterface, TImplementation>(). The factory creates its HttpClient with the address, time
limits, and retries set in one place, and the rest of the app only sees the interface.
Mid (3-5 yrs)
M What are backoff and jitter, and why use both? concept
Backoff makes each wait longer than the last (about 1, 2, 4 seconds), so a struggling service gets time to recover. Jitter adds a random part to each wait, so many clients that failed together do not all retry at the same moment and overload the service again.
M What is an idempotency key, and when do you need one? concept
A unique id for one operation, sent in a header such as Idempotency-Key, the same on every attempt. If the
service already handled that key, it returns the first result instead of doing the work again. You need it
whenever you retry a request that changes something, such as a POST that sends a text or takes a payment,
because a lost answer would otherwise cause a duplicate.
M Explain the three states of a circuit breaker. concept
Closed: calls go through and failures are counted. Open: after too many failures, every call fails at once without touching the service, for a break period. Half-open: after the break, one trial call goes through; success closes the circuit, failure opens it again.
Senior (6-9 yrs)
S How would you make an API's calls to a payment or messaging provider reliable? design
A typed client from IHttpClientFactory behind an interface; a time limit per attempt and a total limit;
retries only for transient errors, with backoff and jitter; an idempotency key on every changing request; a
circuit breaker; logs and metrics for retries and circuit events. Work that must not be lost goes into an
outbox table first and is sent by a background job that can retry later. Tests replace the network with a
scripted handler.
How to answer: Cover the client, time limits, retry rules, idempotency, the circuit breaker, durable storage, observability, and testing.
S Your retries work, but during a one-hour provider outage messages were lost. What do you change? scenario
Retries cover seconds, not hours. Store each message durably before sending, in an outbox table with a status, in the same transaction as the business change. A background job sends pending rows, marks them sent, and retries failures later with growing delays, keeping the idempotency key, and alerts after too many failures.
How to answer: Separate short retries from durable delivery, then describe the outbox, its job, and monitoring.
Architect / Lead (10+ yrs)
A Many services in your company call each other. How do you keep failures from spreading? design
Give every call a time budget shorter than the caller's own, and pass deadlines along. Retry only at one layer, so retries do not multiply through the chain, and cap them with retry budgets. Use circuit breakers and bulkheads (separate limits per dependency), idempotency keys for changing calls, and asynchronous messaging with durable queues where an immediate answer is not needed. Keep shared dependencies out of readiness checks, and watch retry and circuit metrics on dashboards.
How to answer: Cover time budgets, where to retry, retry budgets, circuit breakers and bulkheads, idempotency, async messaging, health checks, and monitoring.
Resources
- Code for this lesson: git tag
extra-8-http-client(code repository coming soon) - Files:
src/SchoolApp.Api/Clients/TextGateway.cs,src/SchoolApp.Api/Clients/TextGatewayClient.cs,src/SchoolApp.Core/Interfaces/ITextMessageGateway.cs,src/SchoolApp.Core/Services/GuardianMessageService.cs,tests/SchoolApp.Tests/Unit/TextGatewayClientTests.cs,extras/e8-http-client/text-gateway.cs,extras/e8-http-client/text-gateway.http,extras/e8-http-client/deadlock-retry.cs,library-assignment/e8-library-book-lookup.cs - Video: coming soon
What's next
The SMS can now talk to other services and survive their bad days. The next extra handles files: importing a class list from a spreadsheet and exporting reports. Until it is published, continue with the capstone: Lesson 97: From one school to 1,000
Want live sessions, code reviews, and interview preparation? Join the next batch at Sahasra Technologies.