EVALUATION 02 / BOUNDED COMPONENT DECISIONS

The model
advises.

An agent keeps control of the workflow and asks hundreds of specific questions. The suite tested 369 identical requests per model across five common decision points.

[ TEST / 02 ]

Five questions an agent can delegate.

01Tool routing

Which one of 15 tools should be called next?

02Trajectory selection

Which completed candidate actually meets the task?

03Result relevance

Does this tool result help with the current task?

04Subgoal satisfaction

Does the observed evidence meet the exact subgoal?

05File context

Should this search hit enter the agent's working context?

The cases are a frozen synthetic fixture set: 40 tool choices, 24 trajectory choices, 40 result checks, 40 subgoal checks, and 225 file judgments across 15 search tasks. Some cases vary order or add irrelevant text; those variants are not independent tasks. Labels and every response are inspectable below.

[ OUTCOME / 02 ]

Performance by decision.

IDENTICAL REQUESTS · NO CLIENT ANSWER CACHE
[ READOUT / 02 ]

A useful role, with gates.

Jev got every tool, trajectory, and subgoal case right in this fixture set. It also found every useful file, but accepted 13 irrelevant ones. CLM was fast and conservative: it made no false-positive file selections, but missed 14 of 18 useful files. In this workload, Jev looks promising for bounded decisions with validation; CLM's missed positives make it a poor automatic context filter.

On 72 paired variants with changed order or irrelevant text, Jev kept the same semantic answer in 72 pairs; CLM did so in 52. These synthetic counts are not a reliability guarantee for arbitrary agents.

[ PUBLISHED / 02 ]

Related results, different tests.

Tool calling

The team's zero-shot BFCL v4 chart reports CLM 95.2% and Jev 99.2%. This 40-case suite found 50% and 100% on hand-authored 15-tool choices; a separate evaluation covers 1,253 public BFCL cases. The team's exact case list and scorer remain unpublished, so neither test is an exact reproduction.

Trajectory verification

The team reports CLM 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1, ahead of Jev at 71.1% and 83.1%. They fine-tuned separate CLM heads to choose among complete coding solutions. The 14/24 result here used the public base head zero-shot on short synthetic trajectories, so it does not test those claims.

[ GATE / 02 ]

What if the agent abstains?

Move the probability threshold to see how many decisions would be accepted and how often those accepted answers were correct. This is an empirical filter on this small suite, not a calibrated guarantee.

For Choice, probability is the largest option probability. For Noul, it is the larger of yes and no. A high score can still be wrong; inspect false positives and misses below.

[ AUDIT / 02 ]

Inspect every case.

369 CASES
[ METHOD / 02 ]

Read the result carefully.

Accuracy is not enough for retrieval

Only 18 of 225 file judgments are positive. A model that rejects everything would score 92% accuracy while retrieving nothing. Positive recall and F1 are reported beside accuracy.

Useful within a scaffold

These are isolated decisions, not a live end-to-end agent. A production agent would still need tool arguments, validation, retries, escalation, and outcome checks. The suite tests whether these particular judgments look safe to delegate.