How Much Retry Traffic Can Your Agent Safely Add?

Calculate spare capacity for replay-safe work and measure how many retries fit your deadline. Admit replays through one shared budget and reserve a separate quota for any independent fallback.

Give retries a deadline and a shared ceiling

For an operation already classified as safe to repeat, measure end-to-end time from the first request. Reserve the first attempt's timeout, then add the backoff and timeout for every proposed retry. With an illustrative 900 ms deadline, 200 ms per attempt, and waits of 50, 100, and 200 ms, two retries fit at 750 ms. The third would finish at 1,150 ms if every attempt used its timeout. Stop when the next complete attempt cannot fit. Put retry ownership at one layer and count attempted replays there. In a staging timeout test, compare that counter with downstream request counts. If one admitted retry produces several downstream calls, find the nested retry policy before raising the shared ceiling.

Timeouts, retries, and backoff with jitter | AWS Builder Center

Load-test the fallback with its own traffic quota

Reproduce a dependency outage in a staging load test with measured baseline traffic on both routes. For each route, record successful throughput, latency, and queue depth before and after moving traffic. For example, a fallback already handling 80 requests per second at a tested ceiling of 100 has room for at most 20 additional requests per second under those conditions. If 30 requests per second are eligible for replay, diverting all 30 puts the fallback above its measured ceiling. A fallback that shares the same constrained database should draw from the same budget. Checkpoint the deferred work and track its age. If successful throughput falls or the queue keeps growing during the test, lower the admission rate until both recover.

Google SRE - Cascading Failures: Reducing System Outage

Record the intended write before sending it

Create a durable record for each external write before dispatch. Store the run ID, a stable operation ID, the intended effect, and an initial pending state. If the destination supports replay protection, pass that same operation ID on every permitted attempt. The Claude Code Slack channel repository documents a durable reply obligation recorded before sending, which is one concrete example of this pattern.

In staging, stop the worker after the destination accepts the write and before the worker saves the response. Restart it and inspect the pending record. Pass if the new worker checks the destination and records the existing result. A second external write or a silently discarded pending entry both fail the test.

GitHub - jeremylongshore/claude-code-slack-channel: Slack-native governance substrate for Claude Code and peer agents: every tool call passes a tiered policy engine and lands in a hash-chained, Ed25519-signed audit journal you can verify offline. The governance kernel the companion agent-governance-plane (AGP) builds on. · GitHub

Walk a timed-out write to a decision

In an illustrative run, an agent submits a maintenance ticket with client reference M-184, and the request times out after five seconds. Assume the ticket system can look up that reference. If it returns one matching ticket with the intended fields, save its ticket ID and continue from the next unfinished step. If it returns conflicting matches, send the details to a human and pause ticket creation.

If the lookup itself has a transient error, retry that read under a short deadline with capped backoff and jitter. An empty result may reflect a delay in visibility. Replay only if the destination's contract makes the same-key request safe or you can confirm the original write never took effect.

Timeouts, retries, and backoff with jitter | AWS Builder Center

Stop retries when the dependency is overloaded

Watch both the success rate and the volume of attempts. Google SRE describes rising latency and elevated errors as possible symptoms of resource exhaustion. If agents are submitting more attempts while fewer operations complete, reduce concurrency and give queued work a later retry time. Carry a single deadline and attempt budget through the agent, tool wrapper, and client so each layer draws from the same allowance.

Exercise this with a staging dependency that injects sustained transient failures. Pass if attempts remain within budget, new work slows down, and pending writes stay available for reconciliation. If attempts multiply across layers, find the client with its own retries and make it honor the shared budget.

Google SRE - Cascading Failures: Reducing System Outage

Escalate with a checkpoint an operator can use

When the destination cannot confirm a write and provides no safe replay contract, hold the operation for review. The Claude Code Slack channel repository describes quarantining uncertain work during startup recovery. Apply the same decision to ambiguous writes by saving the intended action, destination, last verified state, and lookup result in the operator's queue.

Give the operator a way to confirm an existing result, authorize a new attempt, or cancel the pending action. Record their decision on the same operation record and resume from the first unfinished step. Test by making a staging lookup inconclusive. Pass if the agent produces one review item, makes no further write attempts while paused, and resumes from the recorded decision.

GitHub - jeremylongshore/claude-code-slack-channel: Slack-native governance substrate for Claude Code and peer agents: every tool call passes a tiered policy engine and lands in a hash-chained, Ed25519-signed audit journal you can verify offline. The governance kernel the companion agent-governance-plane (AGP) builds on. · GitHub

Sources