Agent bench · Fast 250 · engine 0.1.0
250 lean trials
Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.
Suite
250 trials
216 seeds · 34 generated
Gate
GREEN
250 trials · fp 71f186ff
Witnessed
100%
107/107 byte-reading oracles demand the tool
Oracle cost
47.06 ms
mean eval · 250 bound rows
Measured
12.0%
lfm2.5-350m-gtap-v3 · 250 trials · pass@1
What these numbers can bear
gate: reference passes · null fails · the lazy route fails
- Shortcut-resistant
- 107 / 107
- byte-reading oracles that demand the tool, or prove the lazy route fails
- Independent cells
- 242 / 250
- 8 rows repeat an answer shape — confidence intervals use cells, not rows
- Grades
- 63 / 62 / 62 / 63
- easy / medium / hard / extreme, scored from steps, checked args, and shell composition
- Cheat ceiling
- 3.2%
- no no-knowledge route wins; best blind guess is modal-label
- Gate detail
- GREEN
- 0 duplicate prompts · 0 shortcuts passing · 0 null-kind mismatches
Measured · lfm2.5-350m-gtap-v3 · 250 trials
pass@1 12.0% · Wilson [9%, 17%] · score 0.145 · weighted 0.096 · valid-pass 12.0% (30/250)
clustered 242 answer shapes at 11.7% · Wilson [8%, 16%] — repeated shapes are not independent evidence
mean model 51ms · mean 18 tokens out · mean heap 12.4 kb
jobs 1 · samples 1 · temp 0 · filter full · suite fp 71f186ff ✓ matches current suite
harness fp 433e94bf · rules fp d26209f7 · seed 7 · reply cap 3,750 tokens · source slmb_fast_lfm2_5-350m-gtap-v3.jsonl
By difficulty
25%
25%
25%
25%
Design a strike
exact settings · Rust CLI · badges via sync
Cohort Fast 250 · 250 tests · fixed selection
reply token allowance 3,750 tokens — standard · valid 0 or 64–16,384
Auto negotiates JSON Schema, JSON object, then text when the provider rejects a format. Fixed modes stay fixed. Fast 250 always runs its fixed 250-trial cohort — sampling and family/grade slices are only available on Full 2500.
Enter the loaded model's exact API id to copy the strike.
slmb strike --mode fast --model 'MODEL_ID' --jobs 1 --samples 1 --temperature 0 --seed 7 --timeout-s 120 --response-format auto --max-tokens 3750slmb sync18 shown · 143 in-process (50 ms) · 107 sandboxed-shell (2 s) · measured 250