EVALUATION 03 / PUBLIC TOOL-CALLING CASES

Which tool
gets called?

CLM and Jev received the same 1,253 public BFCL v4 requests and each chose one function. The answer depends sharply on how the candidate functions are written.

[ TEST / 03 ]

A public test of one agent decision.

The Berkeley Function Calling Leaderboard includes requests where a model must select among several functions. This test used every case in its multiple and live_multiple sets, retained the official request and candidate order, and checked the chosen function name against the official answer.

SCORING SCOPETOP-1 FUNCTION NAME

The official BFCL leaderboard also checks generated arguments and other categories. This adaptation measures routing only, so its percentages are not BFCL leaderboard scores.

[ RESULTS / 03 ]

The paired run.

1,253 CASES PER MODEL · SAME INPUTS WITHIN EACH VIEW

Loading results...

The description and raw JSON views were planned first. The prose view was added after CLM's raw JSON result was inspected; it is an exploratory check of formatting.

Accuracy and median client-observed response time for each candidate representation
VIEWCLM CORRECTCLM ACCURACYCLM P50JEV CORRECTJEV ACCURACYJEV P50
INTERACTIVE COMPARISON

How presentation changes selection

Bars show top-1 function-name accuracy. Each view uses the same cases and gold names; only candidate wording changes.

[ READOUT / 03 ]

Tool choice is sensitive to the interface.

Loading the paired results...

This result applies to the CLM base head used here. It does not measure a separately trained BFCL head or a full agent that validates and executes calls.

[ PUBLISHED / 03 ]

How the team's chart compares.

The CLM team's published "BFCL v4 tool calling" chart reports 95.2% for CLM and 99.2% for Jev. The public release does not specify the case list, Choice prompt adapter, or scorer used for that chart. This public-case, function-name-only test is therefore a related check, not an exact reproduction.

The team's reported BFCL v4 numbers and the closest view in this report are not directly equivalent
RESULTCLMJEVWHAT IT MEASURES
TEAM CHART95.2%99.2%"BFCL V4 TOOL CALLING"; DETAILS UNPUBLISHED
DESCRIPTION VIEW--FUNCTION NAME ONLY; NAME + DESCRIPTION OPTIONS

Do not read the difference between these rows as a model regression: the test protocol and possibly the cases differ. The team's latency numbers also use an unspecified measurement path.

[ CASES / 03 ]

Inspect every decision.

PUBLIC REQUESTS · OFFICIAL GOLD FUNCTION

[ METHOD / 03 ]

What changed between views.

Same question, same answer key

Each view uses the same 1,253 requests and official gold function names. Both models receive identical typed Choice requests within a view. The options stay in the dataset's order.

Three ways to present options

Descriptions use the function name and purpose. Raw schemas include the full function object as JSON. The post-hoc prose view keeps parameter details but writes them as readable text.

CLM ran on a local GPU pod with its public v0.1-8B head; Jev used TypeSafe's hosted API. Client response time includes these different network paths and is not a pure compute-speed comparison. The results and runner are pinned to the public BFCL data revision shown below.