The model
advises.
An agent keeps control of the workflow and asks hundreds of specific questions. We tested 369 identical requests per model across five common decision points.
Five questions an agent can delegate.
Which one of 15 tools should be called next?
Which completed candidate actually meets the task?
Does this tool result help with the current task?
Does the observed evidence meet the exact subgoal?
Should this search hit enter the agent's working context?
The cases are a frozen synthetic fixture set: 40 tool choices, 24 trajectory choices, 40 result checks, 40 subgoal checks, and 225 file judgments across 15 search tasks. Some cases vary order or add irrelevant text; those variants are not independent tasks. Labels and every response are inspectable below.
Performance by decision.
A useful role, with gates.
Jev got every tool, trajectory, and subgoal case right in this fixture set. It also found every useful file, but accepted 13 irrelevant ones. CLM was fast and conservative: it made no false-positive file selections, but missed 14 of 18 useful files. In this workload, Jev looks promising for bounded decisions with validation; CLM's missed positives make it a poor automatic context filter.
On 72 paired variants with changed order or irrelevant text, Jev kept the same semantic answer in 72 pairs; CLM did so in 52. These synthetic counts are not a reliability guarantee for arbitrary agents.
What if the agent abstains?
Move the probability threshold to see how many decisions would be accepted and how often those accepted answers were correct. This is an empirical filter on this small suite, not a calibrated guarantee.
For Choice, probability is the largest option probability. For Noul, it is the larger of yes and no. A high score can still be wrong; inspect false positives and misses below.
Inspect every case.
Read the result carefully.
Accuracy is not enough for retrieval
Only 18 of 225 file judgments are positive. A model that rejects everything would score 92% accuracy while retrieving nothing. We report positive recall and F1 beside accuracy.
Useful within a scaffold
These are isolated decisions, not a live end-to-end agent. A production agent would still need tool arguments, validation, retries, escalation, and outcome checks. The suite tests whether these particular judgments look safe to delegate.