SLMB 0.1.0 · Full 2500 · gate GREEN · fp b8c02463 · 2,500 trials

Agent bench · Full 2500 · engine 0.1.0

2,500 lean trials

2188 hand seeds + 312 generated trials across 13 families. References pass, null cases fail, and shortcuts fail the gate. No current measured model runs yet — strike LM Studio from the shell to badge trials.

Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.

Suite

2,500 trials

2188 seeds · 312 generated

Gate

GREEN

2500 trials · fp b8c02463

Witnessed

100%

1206/1206 byte-reading oracles demand the tool

Oracle cost

50.30 ms

mean eval · 2500 bound rows

Measured

none yet

slmb strike --mode full --model MODEL_ID

What these numbers can bear

gate: reference passes · null fails · the lazy route fails

Shortcut-resistant
1,206 / 1,206
byte-reading oracles that demand the tool, or prove the lazy route fails
Independent cells
2,193 / 2,500
307 rows repeat an answer shape — confidence intervals use cells, not rows
Grades
381 / 872 / 878 / 369
easy / medium / hard / extreme, scored from steps, checked args, and shell composition
Cheat ceiling
1.5%
no no-knowledge route wins; best blind guess is prompt-substring
Gate detail
GREEN
0 duplicate prompts · 0 shortcuts passing · 0 null-kind mismatches

By difficulty

easy381

15%

medium872

35%

hard878

35%

extreme369

15%

Design a strike

exact settings · Rust CLI · badges via sync

Cohort Full 2500 · 2500 tests

reply token allowance published trial caps — Full 2500 standard · valid 0 or 64–16,384

Auto negotiates JSON Schema, JSON object, then text when the provider rejects a format. Fixed modes stay fixed. Sample size spreads trials across family × difficulty.

Enter the loaded model's exact API id to copy the strike.

slmb strike --mode full --model 'MODEL_ID' --jobs 1 --samples 1 --temperature 0 --seed 7 --timeout-s 120 --response-format auto
slmb sync

158 shown · 1294 in-process (50 ms) · 1206 sandboxed-shell (2 s)

158 rows · page 1 / 4