Stop shipping probabilistic agents

For weeks, a lender's support agent quietly handed out free credit. Nobody caught it, because nothing looked broken.

It was a buy-now-pay-later lender I worked with, for a fintech. The agent had real authority: it could pause a plan, waive a fee, restructure a schedule, on its own. Useful. Also quietly expensive. Every time it paused a plan sitting on a 0% promo, the pause broke the promo's conditions, and a broken promo defaulted to interest-free, so the customer kept the 0% the lender meant to end. Free credit nobody agreed to give. It ran for weeks, because every transcript looked fine.

Here's why that happens, and why a sharper model won't fix it.

An LLM is a probability engine.

It predicts the most plausible next token, and most of the time plausible is right. That's why it's so good at language: there are many good ways to say a thing, and it only has to find one.

Now give it the authority to act. An agent doesn't just talk, it does: it pauses an account, issues a refund, changes a term, grants access. The moment a probability engine starts deciding what's allowed, you've put a guess where you needed a rule.

Sometimes the agent is genuinely missing the rule. More often it isn't: the rule is sitting right there in its context, and it still gets the call wrong, because it answers over the rules the way it answers everything: probabilistically. The rule's in the room, and a confident story talks it out of it anyway.

This piece is about the fix. You don't need a sharper model. You need to put the model where guessing is an asset, and keep it out of the places where guessing is the whole problem.

What follows is that architecture, component by component.

Split the agent into 2 jobs

Start by noticing that “the agent” is doing 2 jobs that have almost nothing in common, and the trouble starts the moment one system owns both.

The 1st is interpretation: read the input, work out what's actually being asked, produce a fluent response. A probability engine is extraordinary at this. It's the whole reason to reach for one.

The 2nd is judgement: decide what's allowed, then act. 

Can this be paused. Is this fee waivable. Does this action quietly break a promise the company already made. Those have correct answers, and “most plausible” is not how you find them.

You hand both to 1 model because it can hold a conversation, so you assume it can hold a rulebook too. 

It can't.

Not reliably. 

So split the work. Interpretation goes to the model. The deciding goes to something that doesn't guess.

That lender's agent is the clean version of the split. Reading “I can't pay this month, can you pause it” and turning it into a structured request is the interpretation job, and the model nailed it every time. Deciding whether a pause is allowed on a 0% promo, where a pause silently voids the promo, is the judgement job. That's the one it should never have been holding.

Write the rules down where a machine can read them

Before anything can enforce what's allowed, “what's allowed” has to live somewhere a machine can read. In most companies it doesn't. It's spread across policy PDFs, a senior rep's head, the support team's memory, prose a human interprets case by case. Or just a pile of contradictory edge cases nobody has reconciled. A model inherits that fog and interpolates through it, which is a generous word for guessing.

So write the rules down as a structured model of your domain. The word for that model is an ontology (a fancy word, and a load-bearing one). It has 3 layers.

Entities and their states: the nouns in your domain, and the conditions each one can be in. A customer, an order, a subscription, a payment. Each sits in a state, and the state is half the rule: an order is draft, placed, shipped, or refunded; a subscription is trialing, active, past-due, or cancelled. Most bad decisions are really a right action taken in the wrong state.

Actions and their transitions: the verbs in your domain, each one a transition that moves an entity from one state to another. Refund an order, pause a subscription, upgrade a plan, grant a role. Every action has a precondition (the states it's allowed from) and an effect (the state it leaves behind): a refund moves an order from placed to refunded; a pause moves a subscription from active to paused. That effect is the other half of the rule, and it's the half a model never reliably holds onto, because some actions change more than they appear to. Extending a due date doesn't just move a date, it shifts the whole schedule. Grant a role and you've opened doors nobody mentioned. Pause a plan sitting on a 0% promo and the pause voids the promo. Most expensive mistakes are an action whose real effect was never written down.

Policy: which actions are allowed, in which states, under which conditions. A refund over $5k needs a manager. A plan can't be downgraded mid-term. A past-due account can't be handed new credit. A schedule-altering action can't run on a 0% promo. One place, stated flatly, instead of re-derived from scratch in every conversation by a model improvising under pressure.

Here's the part worth sitting with: a model is excellent at building the ontology. Point it at your policy documents and your messy data and it will draft the entities, propose the rules, and surface the contradictions faster than any human. That's the right job for a probability engine: read, draft, propose, with a person confirming. Guessing is fine when you're proposing a schema. It is not fine when you're the thing enforcing it live.

The ontology is the load-bearing artifact. Skip it and the gate in the next section has nothing to read: every rule downstream depends on the rules existing somewhere exact. This is the part to get right.

For the lender, this was 3 short tables. “Pause” is an action, and its effect is “alters the schedule.” A 0% promo is a state with a condition: the schedule can't change. Put those 2 facts in one model and the trap that cost them real money stops being a surprise. It's a rule you can point at: no schedule-altering action on a promotional plan.

Put a deterministic gate between the decision and the act

Now that the rules exist in a form a machine can read, put them in the path. Between the agent's intention and anything that actually happens, you place a deterministic check. The agent doesn't execute. It proposes. The check reads the real state, tests the proposed action against the policy, and either runs it or refuses, with the rule it failed.

That “reason” isn't the gate reasoning. It's the gate returning the specific policy the action tripped: “denied, no schedule-altering action on a promotional plan.” A static label attached to a rule, not an argument the model can negotiate with.

“Anything that actually happens” is the part worth slowing down on: money moving, a term changing, access being granted, a notice going out. The things you can't take back. That's where you want a gate.

The model proposes. The deterministic layer disposes.

This check can't be moved by a sympathetic story or a confident tone, because it isn't reasoning. It's looking something up. The one decision that has to be right is made by the part of the system that can't be wrong in the way models are wrong.

Back to the lender's pause.

The old agent got talked into it: a customer in a genuinely hard spot, a warm and reasonable request, and the model did what models do under that kind of pressure, it reached for the most agreeable next move. Put a gate in front and “pause this plan, keep the 0%” hits the check, fails one rule, and comes back refused, with the rule it failed. Same request. Opposite outcome. Nothing about the model changed. It just lost the final say.

Keep the model at the edges

Notice what just happened to the model's job. It shrank, on purpose. It reads the input at the front, drafts the response at the back, and in the middle, where the deciding lives, it isn't invited. Probability at the edges, where approximately right is exactly what you want. Determinism in the core, where approximately right is a liability.

There's nothing creative about enforcing a policy, which is exactly why you don't want your most creative component doing it. People are bad at boring rule-following for the same reason the model is: attention wanders, an exception feels reasonable, a rule quietly gets bent. The fix isn't a better mood. It's not handing a deterministic job to a probabilistic system.

It feels like a downgrade. Like you've crippled the agent, pulled its teeth. You have, in the one place the teeth were pointed at you. To the user, nothing changes: same speed, same warmth. What changes is that the agent loses the authority to do something the company can't stand behind. It keeps everything it was good at and gives up the one thing it was quietly bad at.

Take the record away from the model

There's a second handoff that matters as much as the gate. If the agent that makes the decision also writes the log, your failures read as wins for as long as it takes someone to check by hand. The thing that got the call wrong is the same thing reporting it went fine. And there's more wrong than you'd think, because a probabilistic system fails at a rate, not by accident.

0.95^10 ≈ 0.60. That isn’t a rare failure. It’s arithmetic.

So take the record away from it too. The deterministic layer logs what actually executed, against the rule that permitted it, in a form your finance or ops team can reconcile. “How much exposure is sitting in here” stops being a forensic read of thousands of transcripts and becomes a query.

The agent narrates. The gate records. Only one of them is allowed to be believed.

“Isn't this just a workflow with an LLM on top?”

If you're technical, you've had this objection building: strip the language layer off and this is a rules engine with a chatbot stapled to the front. Where's the agent.

Mostly, yes. That's the design, not an accident. The deterministic core is boring on purpose, because the work it does (enforce a known rule, exactly, every time) is work you want boring. The model earns its place precisely where a rules engine falls apart: the messy, ambiguous, endlessly varied job of reading open input and answering like a human would. Boring core, smart edges. That isn't a downgrade from a “real agent.” It's what a real agent has to be the moment it can do something you can't take back.

It's also the shape I keep landing on once an agent can touch something irreversible, and I don't think that's a coincidence. The alternative only survives until it meets real users.

You still have to prove the gate holds

One honest hole.

A gate is only worth as much as your confidence that it refuses everything it should, including the ugly state combinations nobody thought to test. In a real system those multiply fast: an account is past-due and on a promo and mid-dispute and inside a grace period, all at once. The easy combinations you'll catch. It's the rare overlaps, the ones no one sat down to imagine, that slip the gate.

“We wrote some rules and added a few tests” is not that confidence: tests check the cases you imagined, and the failures that hurt are the ones you didn't.

Proving a gate holds for every reachable state, not just the ones you tested, is a different discipline. It has a name, formal verification, but I'll speak about it in another piece.

For now, the move survives every model upgrade between today and whenever the next one lands: decide what your agent is allowed to do, write it down where a machine can read it, and put it somewhere the model can't talk its way past. The model keeps the part it's brilliant at. It just doesn't get the final say on anything you'd have to answer for.

Split the guessing from the deciding, and the agent becomes safe to trust.

(If you want the longer case for why a clean demo hides this failure in the first place, I made it in an earlier piece: Your Agent Works in the Demo Because the Demo Is a Lie.)

—---

An agent that demos and runs clean but can't be trusted with anything irreversible is exactly the rebuild I take on. If that's the one you're sitting on, that's a conversation worth having.