SLMB 0.1.0 · Fast 250 · gate GREEN · fp 71f186ff · 250 trials

Measured models · Fast 250 report cards

A model is more than one number.

Inspect first-answer pass, answer-shape confidence, family strengths, grade cliffs, provider time, and outcome kinds from the same synced strike rows that power the suite browser.

Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.

Selected model · 250/250 current Fast 250 trials

lfm2.5-350m-gtap-v3

slmb_fast_lfm2_5-350m-gtap-v3.jsonl · reply token allowance 3,750 tokens

Pass@1

12.0%

30/250 · Wilson 9%–17%

Clustered

11.7%

242 answer shapes · Wilson 8%–16%

Grade weighted

0.096

valid outcomes, difficulty weighted

Mean score

0.145

12.0% valid-pass

Capability map

Pass@1 · each family's own denominator

Capability map

Grade cliff

Pass@1 across difficulty grades assigned by work, args, and shell composition.

easy · 6317.5%
medium · 6212.9%
hard · 6212.9%
extreme · 634.8%

Cost and outcomes

Model time

51ms

Tokens out

18

Peak heap

12.4 KB

Outcome kinds

oracle_mismatch165
sandbox55
pass30