Measured models · Fast 250 report cards
A model is more than one number.
Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.
Selected model · 250/250 current Fast 250 trials
lfm2.5-350m-gtap-v3
slmb_fast_lfm2_5-350m-gtap-v3.jsonl · reply token allowance 3,750 tokens
Pass@1
12.0%
30/250 · Wilson 9%–17%
Clustered
11.7%
242 answer shapes · Wilson 8%–16%
Grade weighted
0.096
valid outcomes, difficulty weighted
Mean score
0.145
12.0% valid-pass
Capability map
Pass@1 · each family's own denominator
Select
Tool selection
25.0%
4/16 first-answer passes · open trials
Args
Argument filling
40.0%
8/20 first-answer passes · open trials
Flows
Multi-step flows
15.0%
3/20 first-answer passes · open trials
Shell
Terminal skill
0.0%
0/16 first-answer passes · open trials
Files
File operations
0.0%
0/20 first-answer passes · open trials
Classify
Classifier judgments
30.0%
6/20 first-answer passes · open trials
Robust
Robustness
20.0%
4/20 first-answer passes · open trials
Omarchy
Omarchy control
0.0%
0/20 first-answer passes · open trials
Arch
Arch system care
0.0%
0/20 first-answer passes · open trials
Shell+
Pro shell tools
0.0%
0/18 first-answer passes · open trials
Keys
Hotkey knowledge
5.0%
1/20 first-answer passes · open trials
Code
Coding judgment
20.0%
4/20 first-answer passes · open trials
Auto
Computer automation
0.0%
0/20 first-answer passes · open trials
Grade cliff
Pass@1 across difficulty grades assigned by work, args, and shell composition.
Cost and outcomes
Model time
51ms
Tokens out
18
Peak heap
12.4 KB
Outcome kinds