Where should a
decision model sit?
The evaluation tested CLM and TypeSafe Jev in two roles: choosing every next action in a small agent environment, and making bounded decisions inside an agent workflow.
The model
drives.
Every jump, duck, or run becomes an action. No safety shield in the primary test. Survival, crashes, and decision latency were measured on five matched courses.
The model
advises.
369 narrow calls per model across tool choice, trajectory selection, result relevance, subgoal checks, and file context. Inspect every prompt and answer.
The model
routes a tool.
1,253 public BFCL v4 requests per model. Compare which function they choose, see how candidate formatting changes the answer, and inspect every case.
The results
in words.
A numbered record of every setup, condition, outcome, and notable finding across the first run, the main tests, and the team's published figures.
A controlled comparison,
with visible limits.
Both models receive the same typed question format. The narrow-decision requests are byte-for-byte identical. The game uses the same seeds and protocol, but model actions cause trajectories to diverge.
The first two tests are small synthetic and simulated tasks. The third uses public BFCL v4 tool-calling cases, adapted to score function selection only. CLM runs locally on a GPU pod; Jev runs through its hosted API. Response times include those network paths.