Put the Decision Model Inside the Agent Loop, Not Beside It
agentsaillmengineering
The cost nobody budgets for
Build an agent and count where the tokens go. A surprising share is not the work. It is the agent asking itself small questions.
Is this tool call safe to run. Does this document answer the question. Should this go to a human. Is this output in the right format. Did the last step actually succeed.
Each one is a yes, a pick, or a score. Each one currently costs a full generation: prompt in, reasoning out, parse the text, hope the format held. At a thousand of those a minute, the checking costs more than the task.
So teams stop checking. They drop the safety gate, skip the retrieval relevance test, let the agent run unguarded. Not because they decided the checks were unnecessary, but because the checks were too expensive to keep.
A different shape of model
A decision model answers without writing. Given the situation and a typed question, it returns a probability over the options in one forward pass.
Three shapes cover nearly everything an agent asks itself:
| Shape | Returns | In the loop |
|---|---|---|
| yes or no | Probability the statement holds | Safety gate, retry decision, escalation |
| pick one | A probability per option | Routing, tool selection, triage |
| score | A probability per level | Relevance, risk, confidence |
Nothing to parse. No format drift. No reasoning tokens to wait through.
Why calibration is the whole point
Accuracy alone does not let you build a gate. A gate needs a threshold, and a threshold needs probabilities that mean what they say.
If the model says 0.9 and is right nine times in ten, you can route everything above 0.9 automatically and send the rest to a human, and you can predict your error rate before shipping. If the model is accurate on average but its 0.9 means anything from 0.6 to 0.99, no threshold is safe and you are back to reading the output yourself.
This is why decision benchmarks score calibration, not just hit rate. Yes or no questions use Brier score. Choice questions score the probability mass on the correct option. Scales use expected score error. A model can top an accuracy table and still be useless as a gate.
What it changes in practice
Once checking is cheap you can check things you previously could not justify:
- Gate every tool call instead of the risky-looking ones
- Score every retrieved passage for relevance instead of trusting the top k
- Ask after every step whether the step succeeded, instead of discovering failure at the end
- Keep a confidence number on every decision so that the escalation rule is a threshold, not a guess
The architecture shift is small. The decision model sits inside the loop as a function call rather than beside it as another agent.
Reading the public boards
Decision models are now measured on public leaderboards such as S1MB, which currently covers 197 models across 137 benchmarks and ranks by aggregating ranks rather than averaging scores.
When comparing entries, read which column a claim quotes, the date it was read, and how many of the benchmarks the model was actually scored on. Most public boards mix self-reported and verified numbers.
Open-weight options exist in this category, so the approach is testable on your own traffic before it is a procurement decision. That is the right way to evaluate it: run your own decision log through a candidate and see whether the probabilities hold up at the threshold you intend to use.