Stage 10 of 11 · Reliability Engineering · 3 min · Reviewed Aug 2026
Agent Reliability Engineering: Safe to Run 10,000 Times
Reliability engineering for agents is deterministic structure around probabilistic nodes: retries, idempotent tools, timeouts, circuit breakers, state machines, checkpoints, human gates, permissions, secrets, injection defense, sandboxes, budgets, and termination. This lesson is the one most 2024 tutorials skipped and 2026 production teams cannot.
- Retries
- Idempotency
- Timeouts
- Circuit breakers
- Checkpoints
- Human gates
- Prompt injection
- Sandboxes
- Budgets
- Termination
What you learn
How to keep agents safe, bounded, and repeatable in production — not only clever in a demo.
Deterministic vs probabilistic
Keep structure deterministic: which tools exist, which edge is legal, which schema is required, which budget remains. Keep generation probabilistic inside that fence. If the graph itself is sampled, you cannot replay an incident.
State machines and workflows are the deterministic skeleton. The model is a node, not the OS.
Visual
Deterministic Skeleton, Probabilistic Nodes
The graph structure, edges, and output schema are deterministic and replayable. Only the model nodes inside are probabilistic — fenced by typed inputs and outputs.
Retries, idempotency, timeouts
Retry transient 429s and 503s with jitter. Do not retry place_order unless the tool is idempotent (Idempotency-Key or a natural key). Every network tool gets a timeout. A hung browser is a leaked worker.
Count retries in the budget. Infinite retry is an infinite loop with extra steps.
Interactive
Retry with Exponential Backoff and Jitter
Initial call
Agent calls an external API tool.
Circuit breakers and degraded modes
If search is failing at 40%, stop calling it for 60 seconds and take a degraded path: cached results, or “search unavailable.” Circuit breakers protect you from amplifying an outage and from a model that keeps hammering a dead tool.