EVALUATION NOTES / 2026-09-24

The results,
in words.

A numbered account of what happened in each setup: the first performance checks, the full action-policy runs, the smaller agent decisions, and the public BFCL cases.

[ FIRST PASS / 01 ]

The early baseline.

These pilot runs used a European GPU pod. CLM served its public head locally, while Jev was called through TypeSafe's hosted API. Their response times include those different network paths.

  1. SUPPORT TICKETS / CPU HEAD

    Jev routed every ticket; CLM missed the technical ones.

    On 40 hand-written support tickets with four fixed team choices, CLM picked the intended team 27 times and Jev did so 40 times. CLM misrouted all ten technical tickets in this CPU-head run. This is a small, clearly labeled fixture set.

  2. SUPPORT TICKETS / NEW REQUESTS

    CLM answered fresh tickets sooner on this deployment.

    For previously unseen ticket text, median client response time was 35.39 ms for local CLM and 252.49 ms for hosted Jev. At the 95th percentile, it was 37.18 ms and 287.68 ms. These are end-to-end times, not model compute times.

  3. SUPPORT TICKETS / EXACT REPLAY

    Repeating the same request exposed CLM's cache advantage.

    When the exact 40 tickets were sent again, CLM's median dropped to 1.95 ms. Jev's median was 263.10 ms. The large replay difference mainly reflects CLM's local state-and-action cache and Jev's hosted round trip.

  4. SUPPORT TICKETS / GPU HEAD TRIAL

    Moving CLM's head to the GPU changed speed more than accuracy.

    In one exploratory variant, CLM's median for new tickets fell from 35.39 to 29.86 ms, while exact replay took 2.10 ms. It answered 26 of 40 correctly, one fewer than the CPU-head run; a close billing-versus-sales choice changed.

  5. T-REX PILOT / SAFETY SHIELD ON

    Both models survived, with very different reliance on the shield.

    On five shielded courses, CLM and Jev each survived five. The deterministic shield intervened 4,060 times for CLM and 63 times for Jev. Planner agreement was 66.5% and 95.5%; median answer arrival was 16.5 and 282.9 ms.

  6. T-REX PILOT / SAFETY SHIELD OFF

    Removing the shield changed the outcome.

    In a shorter, 30-second pilot on three seeds, CLM survived zero of three courses and Jev survived two of three. CLM died eight times in total; Jev died once. The later main policy test used five 60-second courses, so its totals belong to a separate condition.

OPEN THE POLICY DETAIL PAGE
[ POLICY / 02 ]

The full action-policy run.

The main test used the repository's original T-Rex course, seeds 0-4, 60 seconds per seed, a labeled prompt, and six requests in flight. Both models followed the same rules, although their actions led to different later game states.

  1. MAIN RUN / NO SHIELD

    Neither model reliably controlled the game alone.

    Without safety overrides, CLM survived zero of five courses and Jev survived one of five. CLM recorded 33 deaths across its runs; Jev recorded seven. The model's selected action was applied even when unsafe.

  2. MAIN RUN / NO SHIELD

    Very high action agreement did not guarantee survival.

    Jev matched the physics planner on 99.4% of its unshielded decisions yet failed four of five courses. CLM matched 88.8% and failed all five. A small share of bad actions can still decide a sequential outcome.

  3. MAIN RUN / SAFETY SHIELD ON

    The combined systems both survived.

    With the deterministic safety shield, both models survived all five courses with no deaths. The shield acted 5,259 times for CLM and 20 times for Jev; planner agreement was 63.9% and 98.8%. Survival here reflects the model plus the planner, rather than the model alone.

  4. MAIN RUN / RESPONSE ARRIVAL

    The local path remained faster, with a network caveat.

    CLM's median answer arrival was 16.4 ms with or without the shield. Jev's was 132.0 ms without it and 132.1 ms with it. CLM used loopback from the client pod; Jev used a hosted HTTPS connection, so this does not isolate inference speed.

  5. MAIN RUN / CLEAN EXECUTION

    Crashes came from chosen actions, not API failures.

    All four main policy runs reported zero API errors and zero dropped host seconds. The same five random seeds and game rules were used for both models, but the prompts after the first choices were not identical because each trajectory evolved differently.

EXPLORE THE FIVE COURSES
[ COMPONENTS / 03 ]

Small decisions inside an agent.

The same 369 typed requests were sent to each model across five categories. This suite uses hand-authored and synthetic fixtures, so the numbers describe these bounded questions rather than general agent reliability.

  1. ALL 369 REQUESTS

    The total correct count hides the rare positive cases.

    CLM answered 295 requests correctly and Jev answered 350. But 207 of the 225 file questions had a negative answer; a model that rejected every file would score 207 there. The positive-case measures below are more revealing.

  2. TOOL ROUTING / 15 OPTIONS

    Jev chose the intended tool on every fixture.

    Across 40 one-of-15 tool choices, CLM matched the expected tool 20 times and Jev matched it 40 times. This was a narrow routing task, not a complete tool call with arguments or execution.

  3. TRAJECTORY SELECTION / FOUR OPTIONS

    Jev also led on short trajectory comparisons.

    When choosing one of four short synthetic trajectories, CLM picked the labeled winner in 14 of 24 cases and Jev in 24 of 24. This test used CLM's public base head; it does not test the separately fine-tuned verifier heads used in the team's coding benchmarks.

  4. TOOL-RESULT RELEVANCE

    CLM missed most results that should have been kept.

    CLM found 10% of relevant tool results, while Jev found all of them. Including both misses and incorrect inclusions, their positive-class F1 scores were 0.18 and 0.87. A high overall accuracy could conceal this kind of missed evidence.

  5. SUBGOAL SATISFACTION

    Jev caught every satisfied subgoal in this fixture set.

    CLM identified 40% of the satisfied subgoals; Jev identified 100%. Their positive-class F1 scores were 0.57 and 1.00. This is a check of specific labeled subgoals, not a test of long-term planning.

  6. FILE CONTEXT / 225 JUDGMENTS

    The models made different context-selection errors.

    Only 18 of the 225 candidate files belonged in context. CLM found 22% of those relevant files and added no irrelevant ones; Jev found all of them but also added 13 irrelevant files. Their positive-class F1 scores were 0.36 and 0.73.

  7. PROMPT VARIANTS / 72 PAIRS

    Jev's answers stayed steadier under small prompt changes.

    When option order or unrelated text changed, Jev preserved the same semantic answer in 72 of 72 paired variants. CLM did so in 52 of 72. These variants are related copies of base tasks, not 72 independent new cases.

  8. CLIENT RESPONSE TIME

    CLM replied sooner on these bounded calls.

    CLM's median was about 35-38 ms in four categories and 90 ms for trajectory selection; Jev's category medians were 114-136 ms. The clients ran on one pod, but CLM was local and Jev was hosted, so the difference includes transport.

INSPECT THE 369 PAIRED ANSWERS
[ BFCL / 04 ]

Public function-selection cases.

This separate run used all 200 multiple and 1,053 live_multiple cases in a pinned BFCL v4 data release. Only the chosen function name was checked; the official BFCL score also judges arguments and other task categories.

  1. NAME + DESCRIPTION / ALL 1,253

    Jev routed more requests correctly with concise options.

    CLM chose the gold function in 963 cases, or 76.86%; Jev did so in 1,223, or 97.61%. Both saw the same requests, candidate order, and function descriptions through the same typed Choice interface.

  2. NAME + DESCRIPTION / TWO SPLITS

    The live-multiple split was harder for CLM.

    On the 200 multiple cases, CLM got 175 right and Jev 199. On the 1,053 live_multiple cases, CLM got 788 right and Jev 1,024. This split difference is descriptive; the sets also differ in their underlying requests and option counts.

  3. PARAMETERS IN PROSE / EXPLORATORY

    Adding parameter details in prose reduced CLM's score.

    CLM chose correctly in 732 of 1,253 cases, or 58.42%; Jev did so in 1,238, or 98.80%. This view was added after the raw-JSON result was inspected. It is useful as another comparison, but was not planned in advance and does not isolate one cause.

  4. FULL FUNCTION OBJECT / RAW JSON

    CLM struggled most with compact JSON options.

    CLM got 595 of 1,253 function names right, or 47.49%; Jev got 1,239, or 98.88%. The JSON text includes parameter names, descriptions, and required fields. These results do not measure whether either model could supply valid arguments.

  5. CANDIDATE PRESENTATION

    CLM's result moved much more across the three views.

    From description to raw JSON, CLM fell by 29.37 percentage points while Jev rose by 1.27. In the description view, Jev was right on 276 cases CLM missed; CLM was right on 16 cases Jev missed. Because the views change content as well as wording, they do not prove a single formatting cause.

  6. BFCL RESPONSE TIME

    Longer candidate text took longer for both clients.

    Median times for CLM and Jev were 35.43 and 111.44 ms with descriptions; 42.25 and 117.04 ms with prose parameters; and 66.34 and 119.26 ms with raw JSON. These are end-to-end deployment times across local and hosted paths.

EXPLORE ALL 1,253 CASES
[ CONTEXT / 05 ]

How published results fit.

The figures below are from the CLM team, not extra runs in this report. They help locate which comparisons are close and which use a different model head or test protocol.

  1. TEAM'S SHIELDED T-REX RUN

    The main shielded pattern reproduced.

    The team reported five of five courses survived for each model, 4,883 shield actions for CLM and 28 for Jev, and planner agreement of 65.8% and 98.7%. The independent shielded run also had five of five survived, with 5,259 and 20 shield actions and 63.9% and 98.8% agreement. The team did not publish a matching unshielded main result.

  2. TEAM'S BFCL V4 CHART

    Its 95.2% and 99.2% scores cannot be compared directly with this test.

    The team's chart labels CLM at 95.2% and Jev at 99.2% for "BFCL v4 tool calling." Its public release does not identify the exact case list, prompt adapter, or scorer. The 1,253-case function-name slice here is a related, auditable test rather than an exact reproduction.

  3. TEAM'S CODING VERIFIER CHART

    The published coding scores use fine-tuned CLM heads.

    The team reports CLM at 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1, versus Jev at 71.1% and 83.1%. It trained separate CLM verifier heads to pick among complete coding solutions. The short trajectory test here used the public base head and does not test that claim.

  4. OVERALL INTERPRETATION

    Speed and reliability depended on the role and the setup.

    CLM returned answers sooner from the local deployment, especially on cached repeats. Jev made more correct choices in the ticket, bounded-decision, and public tool-routing tests. A shield rescued the game controller but hid many CLM mistakes. None of these bounded tests establishes the reliability of an open-ended agent.