SLMB 0.1.0 · Fast 250 · gate GREEN · fp 71f186ff · 250 trials

Agent bench · Fast 250 · engine 0.1.0

250 lean trials

216 hand seeds + 34 generated trials across 13 families. References pass, null cases fail, and shortcuts fail the gate. 1 measured Fast 250 model; open a trial for its result.

Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.

Suite

250 trials

216 seeds · 34 generated

Gate

GREEN

250 trials · fp 71f186ff

Witnessed

100%

107/107 byte-reading oracles demand the tool

Oracle cost

47.06 ms

mean eval · 250 bound rows

Measured

12.0%

lfm2.5-350m-gtap-v3 · 250 trials · pass@1

What these numbers can bear

gate: reference passes · null fails · the lazy route fails

Shortcut-resistant
107 / 107
byte-reading oracles that demand the tool, or prove the lazy route fails
Independent cells
242 / 250
8 rows repeat an answer shape — confidence intervals use cells, not rows
Grades
63 / 62 / 62 / 63
easy / medium / hard / extreme, scored from steps, checked args, and shell composition
Cheat ceiling
3.2%
no no-knowledge route wins; best blind guess is modal-label
Gate detail
GREEN
0 duplicate prompts · 0 shortcuts passing · 0 null-kind mismatches

Measured · lfm2.5-350m-gtap-v3 · 250 trials

pass@1 12.0% · Wilson [9%, 17%] · score 0.145 · weighted 0.096 · valid-pass 12.0% (30/250)

clustered 242 answer shapes at 11.7% · Wilson [8%, 16%] — repeated shapes are not independent evidence

mean model 51ms · mean 18 tokens out · mean heap 12.4 kb

jobs 1 · samples 1 · temp 0 · filter full · suite fp 71f186ff ✓ matches current suite

harness fp 433e94bf · rules fp d26209f7 · seed 7 · reply cap 3,750 tokens · source slmb_fast_lfm2_5-350m-gtap-v3.jsonl

By difficulty

easy63

25%

medium62

25%

hard62

25%

extreme63

25%

Design a strike

exact settings · Rust CLI · badges via sync

Cohort Fast 250 · 250 tests · fixed selection

reply token allowance 3,750 tokens — standard · valid 0 or 64–16,384

Auto negotiates JSON Schema, JSON object, then text when the provider rejects a format. Fixed modes stay fixed. Fast 250 always runs its fixed 250-trial cohort — sampling and family/grade slices are only available on Full 2500.

Enter the loaded model's exact API id to copy the strike.

slmb strike --mode fast --model 'MODEL_ID' --jobs 1 --samples 1 --temperature 0 --seed 7 --timeout-s 120 --response-format auto --max-tokens 3750
slmb sync

18 shown · 143 in-process (50 ms) · 107 sandboxed-shell (2 s) · measured 250