An AI agent that works in a demo falls apart on a failed tool call because nothing outside the model is responsible for handling the failure. Demos run on the happy path, where every API answers instantly and completely. In production, calls time out, return half a record, or get rate-limited, and an agent with no orchestration around it will either stop, guess, or retry until something breaks.
Handling those failures is not a model capability. It is an orchestration responsibility, and it has to be designed before the agent goes live.
The model will improvise, and improvisation is the problem
Ask a language model to “handle the error” and it will produce something plausible. That is exactly the risk. Given a timeout from an accounting system, a model might retry immediately, retry forty times, summarise the failure as a success, or fill in the missing fields from what it expects the data to look like.
None of these are reasoning failures. The model has no clock, no retry counter, and no memory of how many times it has already called this endpoint in the last minute. It also has no idea that a duplicate “create invoice” call is a different kind of mistake from a duplicate “get invoice” call.
Partial data is worse than an outright error. A 200 response with a truncated line-item list looks like success. An agent that treats it as complete will post an invoice that is wrong by exactly the amount it never saw, and nothing in the logs will flag it.
This is a large part of why most AI pilots never leave the demo stage. The pilot was never exposed to a bad afternoon in someone else’s API.
Retries, timeouts, and escalation belong to the orchestration layer
The fix is to take failure handling away from the model and put it in deterministic code around it. In the orchestration layer, every tool call is wrapped in rules the model cannot override:
Timeouts are set per tool, based on how long that system actually takes, so a slow call fails predictably instead of hanging a workflow for an hour.
Retries use backoff and a hard ceiling, and respect rate-limit headers. They also depend on whether the call is safe to repeat. Reading a record can be retried freely. Writing one needs an idempotency key, or the retry may create a second payment.
Validation checks that a response is complete before the agent sees it. A payload missing required fields is treated as a failure, not as data.
Escalation is a defined path: after a set number of failures, the task stops and goes to a named person with the context attached.
What the agent should do after the third failed call
Take an agent reconciling vendor invoices against an accounting system. The first call times out, and the orchestration layer retries after a short delay. The second returns a rate-limit error, so it waits for the interval the API specified. The third times out again.
At that point the agent should not try a fourth time, should not reason its way to an alternative data source, and should not mark the invoice as reconciled on the strength of what it already knows. It should stop.
The orchestrator then does four things. It parks the task in a recoverable state. It records the three attempts, the errors returned, and the timestamps. It routes the item to the finance operator who owns that queue, with the invoice, the failure history, and what remains to be done. And it keeps the rest of the workflow moving, so one stuck invoice does not hold the batch hostage.
This is policy as much as engineering. The number of retries, the point at which a human is pulled in, and what counts as safe to repeat are decisions someone in the business has to own. At WiseKeel we write those thresholds down per tool before deployment, and every failed attempt is recorded in the audit trail alongside the escalation.
Where it still gets hard
Thresholds are judgment calls. Escalate too early and the agent becomes a notification machine that hands people the same toil they had before. Escalate too late and a failing dependency is hammered for twenty minutes while the queue backs up.
Some failures are also ambiguous. A write that times out may or may not have been applied. Without an idempotency key or a way to read back the result, the orchestrator cannot tell, and the honest answer is a human check. Many legacy accounting systems offer neither.
Multi-agent setups add a further layer. When one agent’s failure becomes another agent’s input, error handling has to cross the boundary, which is the territory covered in what breaks when agents are chained without handoffs. Each added agent is another place a failure can be swallowed.
The question to sit with
Pick one system your agent writes to, and ask what it does on the third consecutive failure. Not what the model would say it does, but what the surrounding code does, who gets told, and whether you could reconstruct the sequence afterwards. If nobody can answer, that is the gap a Book a diagnostic call is built to find.