Designing Trust When an Agent Is the User

I work with agents daily. They read my repositories, edit my files, deploy my sites and talk to my APIs. That is not a demo, it is how the work gets done here.

So the question of when software should be allowed to act without asking is not theoretical for me. I have got it wrong, and getting it wrong is memorable.

The assumption every confirmation was built on

Every "are you sure" dialog ever shipped assumed a person was reading it. The whole design depends on friction landing on someone who can feel it. A moment of hesitation, a second look at the file name, the small physical reluctance to click the red button.

None of that exists for software. A confirmation an agent can dismiss is not a safeguard, it is a formality. If you have built your safety story on a modal, you have built it on the one thing that stops working here.

What I use instead

Two ideas, and they are not sophisticated.

Separate reversible from irreversible, and be honest about which is which. Reading is reversible. Writing a file is usually reversible. Sending an email is not. Deploying to production is not. Deleting data is not. Most products have never made this distinction explicit because they never had to, and now it is the most important line in the system.

Then require a human for the irreversible ones. Not a dialog. An actual stop, where the work is prepared up to the edge and nothing crosses it until a person says so. The difference matters: I want everything ready, and I want the last step to be mine.

My own rule is that approval does not carry forward. Saying yes to a deploy on Tuesday is not saying yes to a deploy on Wednesday. Permissions that persist quietly are how you end up surprised, and being surprised by your own tools is the failure state.

The thing I got wrong

I used to think the risk was capability. Give it less power and you are safer.

The real risk is ambiguity. An agent with narrow permissions and a vague instruction does more damage than one with broad permissions and a precise instruction, because the vague one fills in the gap with a guess and the guess looks like completed work.

That changed how I write instructions more than it changed my permissions. Say what done looks like. Say what is out of scope. Say which step is mine. Most of the safety is in the sentence, not in the settings file.

What this means if you are building the product

Three practical things I would want from any tool I let an agent use.

Tell me what it is about to do before it does it, in a sentence I can read quickly. Not a diff of everything, the summary of the consequence.

Make undo real, and say so. If a thing can be undone, tell me at the moment of the action, because that is when I am deciding how much to worry.

And put the irreversible actions behind something an agent structurally cannot do on its own, rather than something it is asked politely not to do. A request is not a boundary.

Why this is a design problem

Because all of it is about what gets communicated, when, and to whom. That is the job.

The engineering side of this is mostly solved and mostly dull. Scopes, tokens, allowlists. The unsolved part is the human one: how someone stays genuinely informed about work they are not watching, without being buried in notifications they will learn to ignore. We have decades of practice at that problem in every other context, and almost none of it has been applied here yet.

That is the most interesting design space I have worked in for a long time, and it is barely started.

Common questions

Is this the same as permissions?
Permissions are the floor. They answer what is possible. Trust is about what should happen by default and what deserves a stop, and that is a design decision.

Do agents actually need this, or is it about reassuring the human?
The human, mostly, and that is not a criticism. The person is accountable for the outcome. Their confidence is a real requirement, not a soft one.

What about audit logs?
Useful afterwards, useless during. A log tells you what happened. It does not stop it. Both matter and they are not substitutes.

How much should the agent explain?
Enough that you could catch a mistake, short enough that you actually read it. I would rather have one accurate sentence than a full transcript I skim.