small language model bench · every size under 27B
2,500 lean trials for small local-LLM agents.
Full 2500 · 2,500-test run
Pick the right tool.
SLMB ranks language models under 27B parameters, by total parameters, from the smallest up. It asks which small model — light enough for a low-VRAM laptop or a low-tier GPU, fast enough to stream ~200+ tokens a second — can most reliably drive an Omarchy Linux machine. 2188 hand seeds and 312 deterministic trials pass the suite gate; every measured row records pass, latency, tokens, and heap.
Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.
{
"prompt": "Apply the catppuccin theme with `omarchy theme set`. The mock records it in .oma-state/theme.",
"oracle": {
"expected": "catppuccin\n",
"kind": "file_exact",
"path": ".oma-state/theme",
"ran": [
"omarchy"
]
}
}Gate contract, not model output · no model execution on this page
The bar: small, fast, in control.
SLMB ranks models you can actually run — weights that fit a low-tier GPU or a low-VRAM laptop and still hold interactive speed. The board answers one question: which light model gets closest to reliably controlling an Omarchy Linux install — packages, services, shell, keys — not just chatting about it.
Fits
low VRAM
laptop-class weights and low-tier GPUs — no datacenter required
Streams
≥200 tok/s
interactive speed — the model keeps up with the machine
Controls
Omarchy
arch, pacman, systemctl, journal — a real desktop, end to end
The bench at a glance
Suite
2,500
2188 seeds · 312 generated
Gate
GREEN
2500 trials gated
Fingerprint
b8c02463
b8c024631971e34e
Measured
none
slmb strike --mode full --model MODEL_ID
Oracle cost
50.30 ms
mean eval · bounds, not model
Witnessed
1,206/1,206
byte-reading oracles that demand the tool actually ran
Independent cells
2,193
of 2,500 rows — repeated answer shapes are one question
Grades
878 hard · 369 extreme
scored from steps, checked args, and shell composition
Thirteen skills, one bench.
Each family probes one agent capability with its own oracle kinds and enforced time budget. Counts are the live suite — fingerprint b8c024631971.
01 / Select · 210 trials
Tool selection
Pick the right tool from 3–12 announced candidates, then call it.
calls_exact · 50 ms
02 / Args · 221 trials
Argument filling
Turn a request into one announced call: right keys, values, and enum codes.
calls_exact · 50 ms
03 / Flows · 214 trials
Multi-step flows
Chain 2–3 calls in order. Prefix partial credit.
calls_exact · 50 ms
04 / Shell · 303 trials
Terminal skill
Single sandboxed shell command: stdout matches, exit 0.
stdout_regex · exit_zero · 4000 ms
05 / Files · 138 trials
File operations
Read, write, and inspect files inside the scratch dir.
stdout_regex · file_exact · 4000 ms
06 / Classify · 129 trials
Classifier judgments
BERT-style labels: sentiment, intent, topic, safety.
label · 50 ms
07 / Robust · 158 trials
Robustness
Corrupted input keeps the label its clean twin has, by construction.
label · calls_exact · 50 ms
08 / Omarchy · 271 trials
Omarchy control
Drive the omarchy CLI on a mock machine: themes, updates, capture, brightness.
stdout_regex · file_exact · 4000 ms
09 / Arch · 230 trials
Arch system care
pacman queries, systemctl state, journal tails — read-only plus state flips.
stdout_regex · file_exact · 4000 ms
10 / Shell+ · 140 trials
Pro shell tools
eza, fd, rg, bat, tldr, zoxide over a fixture tree, the Omarchy way.
stdout_regex · 4000 ms
11 / Keys · 91 trials
Hotkey knowledge
Super-chords, tmux chords, and menus from the Omarchy manual.
label · 50 ms
12 / Code · 230 trials
Coding judgment
Read short code, predict its output or spot the bug, and pick the right tool call for an engineering task.
label · calls_exact · 50 ms
13 / Auto · 165 trials
Computer automation
Script small chores in the sandbox: batch renames, log digests, config edits, cron-style jobs.
stdout_regex · file_exact · 4000 ms
Coverage
- easy381
- extreme369
- hard878
- medium872
1294 in-process · 1206 shell
Measured, never simulated.
Everything on this page is suite, gate, and harness data. Model results appear here only after a real strike.
No measured runs yet
Everything on this page is suite, gate, and harness data. Model results appear here only after a real strike — start LM Studio, then run:
slmb strike --mode full --model MODEL_ID --samples 3slmb syncThe retired 1,500-trial editions measured 56 models. Browse the Legacy results.
Follow the evidence.
Move from the official ranking to individual capability profiles, paired model outcomes, suite-wide consensus, and the provider contracts behind every published score.