Who answers for an AI agent’s mistake?

The question that isn’t technical

If an agent approves a wrong invoice, makes a wrong payment or writes something to a customer it shouldn’t have, the issue is not whether the model was capable enough. It is contractual: who answers for it? Whoever configured it, whoever approved it, whoever signed with the model vendor, whoever was on call that day?

Until that question has a settled answer, the person stays in the process for reasons that have nothing to do with technology. A more capable model reduces mistakes. It does not answer for the ones that remain.

Why software gets there first

In software development, agents already work on their own for long stretches. Not because the models are better there, but because a mistake can be seen, tested and fixed before it ships, and getting it wrong is cheap: a branch of code to throw away. A payment that has gone out can’t be thrown away.

The difference between the two worlds is not the intelligence of whoever executes. It is how verifiable and reversible what they execute is. That is a more useful test than any forecast of model capability, because it can be applied to a real process, today.

The human check is not a stopgap

The person in the process, the so-called human in the loop, is often presented as a transitional phase while models improve. In most business processes it is instead the condition that makes the process usable: it moves responsibility onto someone who can take it.

Copilot or agent? A copilot proposes something and the person decides whether to keep it. An agent carries out a sequence of actions on its own, to an outcome. The difference is not model capability, which is the same: it is who makes the final decision.

Where you can already go: closed-perimeter agents

The realistic step is not total autonomy, but agents that run unattended inside a closed perimeter: recurring, verifiable tasks, run on a schedule. Reconciliations, consistency checks, drafts prepared before a meeting.

The leap is not autonomy itself. It is that the person’s work shifts from producing to validating. And it is available now: it is not frontier and it is not expensive, the brake is organisational.

Five questions before putting an agent into production

• Who is responsible? A role with a name, not “the department”. A shared agent with no owner is the worst case, because everyone uses it and nobody answers for it.
• What is the perimeter? Which systems it reads, which it writes to, with what permissions. Read-only until there is a concrete reason to do more.
• What is reversible? A draft can be discarded, a bank transfer can’t. Irreversible actions go through an approval.
• What gets recorded? Every call, who started it, on what data, with what outcome. Without a complete log there is no accountability, only suspicion.
• When does it stop? A threshold (an amount, a number of operations, an anomaly) beyond which the agent asks instead of acting.

The agent as an entity of the company

I think company culture has to evolve before the moment processes automate themselves. We will need to see agents as shared entities of the company, not as extensions of a person. As long as an agent “belongs to” someone, its accountability is lost when that person changes role.

Anyone predicting total automation within two years is answering a technical question. The real question is a different one.

A declaration of interest: we are a supplier, and we work on the principle that permissions and a call log come before intelligence. The topic is close to what we build with AI agents and middleware, where every call has to leave a trace.