We don’t prompt our way to safety

The labs work on the agent. We work on the gateway.

There is a good reason the frontier labs focus where they do. They ship a model to everyone, in every context, with no idea what it will be pointed at, so the only lever they have is the model’s own disposition. They spend enormous effort making it decline harmful instructions and resist manipulation, and they are getting better at it every release.

We are downstream of that work and glad to have it. But we build for one company at a time, doing known jobs, in named systems — which means we have a lever the labs do not: we can decide what is reachable at all. That is a different kind of control, and it composes with theirs rather than competing.

The practical difference is what happens on a bad day. If the control lives in the instructions, a bad day is a model that was talked around, and the action goes through. If the control lives at the gateway, the model can be fully convinced and completely wrong, and the action still does not go through, because nothing in its reach can perform it.

The mistake that hurts is the one that repeats

When people picture an agent failing, they usually picture it being wrong. It was told to refund a customer $12 and it refunded $20, because it decided sales tax was part of the order total. That is a context error. Human employees make it constantly, it is caught in the ordinary course of business, and it is not where the real exposure is.

The exposure is in scope. The failure that hurts is the one where the amount was perfectly correct and the agent applied it to all forty thousand customers. Where you approved deleting three records and the instruction was read as the table. Where “go find us more revenue” became outbound email to every address it could reach.

This follows directly from what computers are good at. We hand them the repetitive work because they will do the same thing ten thousand times without flagging, and that willingness does not distinguish between ten thousand correct executions and ten thousand copies of one misunderstanding. So the thing worth engineering is the size of the hole a single mistake can fit through. Accuracy is the model vendor’s problem, and they are good at it. Blast radius is ours.

A Busy teammate has its own accounts

This is the difference that does the most work, and it is the one most easily missed because it sounds like an implementation detail. Almost every AI product that touches your real systems uses delegated access: you authorize it, and from then on it acts inside your account, as you. Your Gmail. Your Dropbox. Your audit log, with your name on every line it wrote.

A Busy teammate gets a seat instead. Roan has his own Google Workspace account at your domain and his own Dropbox under it. People share folders with him the way they share with any colleague, and that sharing isthe grant. There is no scoping configuration to get right, because the account boundary is already the fence — the same fence that keeps your Dropbox and your colleague’s Dropbox separate without either of you thinking about it.

It also makes the vendor’s own permission system available to you. Seller Central has a long list of per-section toggles; Google and most SaaS have read-only, editor and admin; QuickBooks has a read-only accountant role. A teammate that logs in as itself is subject to every one of those the moment you set them, and you set them in the vendor’s own settings screen, not in ours.

We do support delegate access, because the genuine executive-assistant case exists — sometimes a teammate really does need to see the principal’s own inbox. When that happens it is additive and explicit: the teammate already has a seat, delegate access is granted on top of it, it is labelled as delegate on the connection, and it is the one case where scoping is required rather than optional. What we will not do is make delegation the default and call the resulting confusion a feature.

Which way the bolt goes when the power dies

“Failsafe” has a loose everyday meaning — a plan that cannot go wrong. In access control it means something exact, and the distinction is worth borrowing. Electric door locks come in two kinds. A fail-safe lock unlocks when it loses power, because the thing being protected is the people inside and they have to get out. A fail-secure lock stays locked, because the thing being protected is what is in the room.

Both are correct engineering. The question is only ever what you are protecting, and a building uses both: fail-safe on the stairwell, fail-secure on the server room. Every gateway a Busy teammate passes through is fail-secure. When a check cannot be completed, the action does not happen.

That sounds obvious and it is routinely built the other way around, usually by accident. A recipient check that is skipped when the allowlist happens to be empty reads exactly like a gate and behaves like one right up until the list is empty, which is the same day someone finishes configuring a new teammate. So the rules are stated as absolutes: an empty list permits nobody, an unreachable policy means the teammate acts on nothing, and a teammate whose stance was never configured gets the tightest one rather than the loosest.

Who a teammate may talk to is one setting, not four

Reach is the highest-stakes permission a teammate has, because it is the one that leaves the building. So it is a single value from a short list of named rungs, and each rung is a fixed answer to every reach question at once: who it may write to, what happens when a stranger writes in, whether it may send outside the company, and whether it may ever start a conversation nobody asked it to start.

It could have been four independent switches, and that would have been worse. Independent switches can be set to combinations nobody would ever choose on purpose — drafting every external reply for review while cheerfully answering cold email from strangers. A ladder can only express the postures that make sense together, which means a misconfiguration has to be a deliberate act rather than an oversight.

Most teammates sit at strict-internal or chaperoned, and Roan — who works on construction projects where a stray email can direct work or start a contractual clock — is at strict-internal permanently, by contract rather than by setting.

Two-factor override: how a teammate steps up a rung

Strict rungs are only livable if there is a way through them, and everyone already knows what that looks like. It is the moment on a support call when the agent says let me get my manager to authorize that. The rule did not bend and the agent did not decide anything. Someone with the authority to approve it, approved it.

That is what a two-factor override is, and the two-factor framing is literal rather than decorative. When you log into your bank, the second factor works because it arrives somewhere other than the login screen — a different device, a different channel, out of reach of whatever is happening in the browser. A 2FO request arrives the same way: a push to your phone, or an SMS, carrying the specific action, the specific recipient, and an expiry.

The distinction that matters most is what happens after you approve. The approval is not relayed back to the teammate as a message it then acts on. The approval executes the action. The gateway that was closed opens for that one recipient, the email goes, and the teammate is told it has been sent.

This is the whole reason we say a teammate asking “may I email Dan?” in a chat window is not a control. It is a courtesy, and a convincing one, but it runs entirely inside the system being controlled: the same component decides to ask, interprets your answer, and performs the action. An override runs outside it. And critically, an approval can be scoped — this one message, this one recipient forever, or this contact added to the oracle list, which is how a teammate permanently earns a rung rather than being handed one.

The email Vera would not send

Recently a friend emailed asking me for something, and I forwarded it to Vera to handle. She did the work well: read the thread, built the document, wrote the reply. Then she emailed it to me and only me, because he was not on her list. Which left me doing the last two steps by hand — sharing the doc, forwarding the note — and made her meaningfully less useful than an assistant who could just finish the job.

That is a real cost and it is worth being precise about it, because the obvious fix is a genuinely bad idea. Ninety-nine times out of a hundred I wanted her to send it. The temptation is to widen her reach so she can email anyone. But the failure that setting permits is not one unwanted email. It is the day an ambiguous instruction turns into outreach to every address she can find, sent from my domain, in my name. One design tolerates a small recurring annoyance; the other tolerates an unbounded one.

So there are three real answers, and the ladder is what lets us pick between them per teammate rather than once for the product.

Model the escalation on a person and the failures stay interpretable

There is a principle underneath all of this that we apply well beyond safety: the closer a teammate’s behavior maps onto how a person in that role would actually operate, the more predictable its failures become. Not because the mechanism is human — it is software all the way down — but because you already have accurate intuitions about how colleagues fail, and none about how a novel system fails.

Escalation is the clearest case. A clear rung, a request for an override, and an expiry produce a failure you can read at a glance and act on in seconds. It is annoying. Annoying is a good ceiling for a failure mode.

It is worth saying what the alternative looks like, because the industry has landed on a bad equilibrium. Consumer assistants are increasingly trained to refuse — sometimes to the point of declining to fetch a page you wrote yourself — and the refusal arrives with no stated rule, no way to authorize it, and no way to tell a policy from a bug. We would rather set an explicit boundary and then be genuinely permissive inside it. A teammate that knows precisely what it may do has no reason to hedge about the rest.

Hard boundaries are what make wide latitude affordable

The reason to build all of this is not caution. It is that a fence you actually trust is what lets you stop supervising everything inside it. If your kid is reliably home by eight, they get the car on a Friday. If the rule was “use good judgment,” nobody can tell whether it was followed, so nobody hands over the keys.

Every control on this page exists to buy latitude somewhere else. Because the card has a hard limit, the teammate does not have to ask before every purchase. Because the books are read-only, it can work in them all day unsupervised. Because reach is enforced at the gateway rather than requested in a prompt, it can be trusted with a real inbox instead of a sandbox.

That is the trade we are making, and we think it is the one that eventually makes agents genuinely useful at work. Fence the model well enough and you can hand it considerably more to do.