Agent bench · Full 2500 · engine 0.1.0
2,500 lean trials
Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.
Suite
2,500 trials
2188 seeds · 312 generated
Gate
GREEN
2500 trials · fp b8c02463
Witnessed
100%
1206/1206 byte-reading oracles demand the tool
Oracle cost
50.30 ms
mean eval · 2500 bound rows
Measured
none yet
slmb strike --mode full --model MODEL_ID
What these numbers can bear
gate: reference passes · null fails · the lazy route fails
- Shortcut-resistant
- 1,206 / 1,206
- byte-reading oracles that demand the tool, or prove the lazy route fails
- Independent cells
- 2,193 / 2,500
- 307 rows repeat an answer shape — confidence intervals use cells, not rows
- Grades
- 381 / 872 / 878 / 369
- easy / medium / hard / extreme, scored from steps, checked args, and shell composition
- Cheat ceiling
- 1.5%
- no no-knowledge route wins; best blind guess is prompt-substring
- Gate detail
- GREEN
- 0 duplicate prompts · 0 shortcuts passing · 0 null-kind mismatches
By difficulty
15%
35%
35%
15%
Design a strike
exact settings · Rust CLI · badges via sync
Cohort Full 2500 · 2500 tests
reply token allowance published trial caps — Full 2500 standard · valid 0 or 64–16,384
Auto negotiates JSON Schema, JSON object, then text when the provider rejects a format. Fixed modes stay fixed. Sample size spreads trials across family × difficulty.
Enter the loaded model's exact API id to copy the strike.
slmb strike --mode full --model 'MODEL_ID' --jobs 1 --samples 1 --temperature 0 --seed 7 --timeout-s 120 --response-format autoslmb sync2,500 shown · 1294 in-process (50 ms) · 1206 sandboxed-shell (2 s)
2500 rows · page 1 / 63