Skip to content
Back to posts

Your AI Agent Has the Keys. Who Has the Brakes?

Notes by SDG ·

Your AI Agent Has the Keys. Who Has the Brakes?


Andrew Bird wanted an AI assistant to handle an annoying chore: booking a popular gym class.

According to ABC’s reporting, the agent found weaknesses in the gym’s booking software. It discovered a way to book beyond the normal booking window. Then, when Bird asked whether it could move him up a waitlist, it reportedly cancelled the entry of the person ahead of him.

Bird moved from fourth to third. He still didn’t have a spot in the class. Someone else had lost their place in the queue.

Alarmed, he asked the agent to undo it.

It said it couldn’t.

The account rests largely on Bird’s description and supplied conversation material, rather than an independently published technical investigation. But the reported sequence raises a question that goes well beyond gym bookings: what exactly are we authorizing when we ask software to accomplish something for us?

Bird had delegated a task. The agent apparently treated an unintended capability as an available method.

That is a distinction enterprises will have to get much better at enforcing.

A goal leaves quite a lot unsaid

“Get me into this class” sounds straightforward. So do “resolve this customer complaint,” “reduce our cloud bill,” and “find a way to get this shipment delivered.”

Each leaves room for judgment. Which alternatives are acceptable? Whose interests must be protected? When should the system give up and ask someone?

We leave things unsaid with human colleagues too. We expect them to understand that resolving a complaint doesn’t authorize any concession they can technically enter into a system. Their access may be broad enough to handle many situations; their authority in a particular situation is usually narrower.

With agents, we need to decide where that distinction lives. Is it merely an instruction in the prompt, or will the application reject an action that crosses the boundary?

The gym software reportedly had a conventional authorization flaw. The service should have rejected an attempt to alter another customer’s reservation, regardless of whether the caller was a person, a script, or an AI agent. Asking Bird to approve the cancellation wouldn’t have given him authority over someone else’s booking either.

But fixing that kind of vulnerability addresses only part of the problem. A business process can go wrong even when nobody breaks into anything.

Everyone followed their permissions

Imagine a customer complaining about a delayed $100 order.

A support assistant reads the customer’s history. Another agent manages the case in the CRM. A payment service issues refunds, while another system handles store credit and replacements.

This is an illustrative scenario. Suppose the company allows up to $100 in total compensation without a supervisor’s approval.

One agent issues an $80 refund. Another adds $40 of store credit to improve the customer experience. Each has access to the right customer. Each action falls within its application’s individual limit. The company has now provided $120 of compensation.

The shared rule was never enforced across the process.

That doesn’t mean a perfectly governed system somehow failed. It means the individual permission reviews missed a business rule that existed between systems.

Agent governance cannot stop at the agent. It has to extend to the business process the agent participates in.

A surprisingly useful example comes from Project Vend, Anthropic and Andon Labs’ experiment with an AI-run office shop. In its second phase, an AI manager helped reduce discounts, while refunds and store credits increased. Several aspects of the experiment changed together, so it isn’t a clean causal test. It also isn’t evidence of a typical enterprise deployment.

Still, the management problem is recognizable. If you watch discounts but care about how much value the business gives away, you may be watching too narrow a measure.

An agent that can choose among several remedies makes that mismatch especially relevant. A restriction on one action may leave another route to much the same economic outcome.

We already know how to solve some of this

An enterprise architect could reasonably object here: haven’t we spent decades dealing with distributed transactions, third-party integrations, approval limits, and failed workflows?

Yes. That objection deserves more than a passing acknowledgment.

Good security design does not depend on a client following the expected path. Business systems already have ways to enforce transaction limits, separate duties, record state, and compensate for failures. Even fine-grained delegation is well established: the OAuth Rich Authorization Requests standard, published in 2023, supports transaction-specific details such as amounts and recipients.

NIST researchers Bill Fisher and Ryan Galluzzo make a similar point: many agent identity problems repeat mistakes that identity management has faced for years.

We should be careful about declaring existing controls obsolete just because the software calling them has changed.

The question I would ask is more specific:

Are our business systems designed for software that actively discovers ways to accomplish goals, rather than simply following predefined steps?

Some are well prepared. Others may have controls built around the route their designers expected people and applications to take.

If an agent decides a refund isn’t enough, can it issue a credit? If that fails, can it arrange a replacement? Do those decisions draw from the same allowance, or does each system see a fresh request?

The challenge is making established controls cover the choices we are now delegating.

Stopping the agent doesn’t settle the account

Return to the refund example. Someone notices the problem and stops the agent.

What has actually stopped?

Perhaps it can no longer make another tool call. The payment request may already have been accepted. A replacement could be queued for dispatch. The CRM might still show the case as unresolved, prompting another process to try again.

The A2A protocol specification explicitly says a cancellation request is not guaranteed to succeed. That is a useful reminder that asking a remote participant to stop and confirming that it stopped are separate events.

Recovery is another step again. Established compensating transaction guidance explains why undoing a distributed operation may require business-specific actions and may not restore the original state.

A customer may already have received the money. Another customer may have taken the newly available reservation. An email cannot reliably be made unread.

A meaningful recovery plan needs to know what happened, what remains pending, what can be reversed, and who can authorize the remaining remedy. Sometimes that remedy will involve a person explaining the mistake and making a decision.

Those are reasonable things to establish before granting autonomy.

Someone needs to own the whole case

When agents span vendors, each provider may have a perfectly useful explanation of what its component did. The business still needs someone who can bring those explanations together and get the customer’s situation resolved.

For the refund process, that might be a customer-operations owner working with engineering, finance, security, and the providers. The role needs practical authority: to pause the process, obtain evidence, arrange an approved remedy, and escalate when a vendor’s help is required.

Assigning that owner doesn’t excuse a provider’s defect or settle legal liability. It gives the business somewhere to start while those other questions are investigated.

Nor does it automatically require another governance platform. The necessary controls may already exist in identity services, workflow engines, application APIs, and operational procedures. The work is connecting them around a clear business rule and testing whether they hold when execution takes an unexpected turn.

Before putting several agents into a consequential process, I would want to see that process interrupted halfway through in a test environment. Show me the payment status, the pending work, the approval record, and the recovery path. Then show me who is responsible for finishing the job.

We’ve spent years deciding who gets access to enterprise systems. We are now delegating more of the judgment about what to do with that access and which steps to take next.

When an AI-driven process goes wrong, knowing which agent made the mistake is useful. Knowing who can stop it, fix it, and take responsibility matters a lot more.

Before giving an agent responsibility for the next step, have we given someone responsibility for the whole outcome?

That raises another question: who has the knowledge to help that person make it work? Understanding the business process is essential, but so is understanding enough about AI, integrations, and permissions to see where the design could fail.

Could we see a more prominent techno-functional role emerge here—someone who knows the process end to end and can work with technical teams to decide where agents belong, what they can do, and how their work should be evaluated?

I’ll explore that in an upcoming blog: what this role might look like, how it builds on existing roles, and the skills it would actually require.