small language model bench · every size under 27B
250 lean trials for small local-LLM agents.
Fast 250 · 250-test run
Pick the right tool.
SLMB ranks language models under 27B parameters, by total parameters, from the smallest up. It asks which small model — light enough for a low-VRAM laptop or a low-tier GPU, fast enough to stream ~200+ tokens a second — can most reliably drive an Omarchy Linux machine. 216 hand seeds and 34 deterministic trials pass the suite gate; every measured row records pass, latency, tokens, and heap.
Fast 250 measured lfm2.5-350m-gtap-v3 · 12.0% pass@1
Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.
{
"prompt": "The user says: \"save Cy's number 555-0100.\" Announce exactly one call from the announced tools.",
"oracle": {
"calls": [
{
"match": "subset",
"name": "add_contact"
}
],
"kind": "calls_exact"
}
}Gate contract, not model output · no model execution on this page
The bar: small, fast, in control.
SLMB ranks models you can actually run — weights that fit a low-tier GPU or a low-VRAM laptop and still hold interactive speed. The board answers one question: which light model gets closest to reliably controlling an Omarchy Linux install — packages, services, shell, keys — not just chatting about it.
Fits
low VRAM
laptop-class weights and low-tier GPUs — no datacenter required
Streams
≥200 tok/s
interactive speed — the model keeps up with the machine
Controls
Omarchy
arch, pacman, systemctl, journal — a real desktop, end to end
The bench at a glance
Suite
250
216 seeds · 34 generated
Gate
GREEN
250 trials gated
Fingerprint
71f186ff
71f186ff72776cc0
Measured
12.0%
lfm2.5-350m-gtap-v3 · 250 trials · pass@1
Oracle cost
47.06 ms
mean eval · bounds, not model
Witnessed
107/107
byte-reading oracles that demand the tool actually ran
Independent cells
242
of 250 rows — repeated answer shapes are one question
Grades
62 hard · 63 extreme
scored from steps, checked args, and shell composition
Thirteen skills, one bench.
Each family probes one agent capability with its own oracle kinds and enforced time budget. Counts are the live suite — fingerprint 71f186ff7277.
01 / Select · 16 trials
Tool selection
Pick the right tool from 3–12 announced candidates, then call it.
calls_exact · 50 ms
02 / Args · 20 trials
Argument filling
Turn a request into one announced call: right keys, values, and enum codes.
calls_exact · 50 ms
03 / Flows · 20 trials
Multi-step flows
Chain 2–3 calls in order. Prefix partial credit.
calls_exact · 50 ms
04 / Shell · 16 trials
Terminal skill
Single sandboxed shell command: stdout matches, exit 0.
stdout_regex · exit_zero · 4000 ms
05 / Files · 20 trials
File operations
Read, write, and inspect files inside the scratch dir.
stdout_regex · file_exact · 4000 ms
06 / Classify · 20 trials
Classifier judgments
BERT-style labels: sentiment, intent, topic, safety.
label · 50 ms
07 / Robust · 20 trials
Robustness
Corrupted input keeps the label its clean twin has, by construction.
label · calls_exact · 50 ms
08 / Omarchy · 20 trials
Omarchy control
Drive the omarchy CLI on a mock machine: themes, updates, capture, brightness.
stdout_regex · file_exact · 4000 ms
09 / Arch · 20 trials
Arch system care
pacman queries, systemctl state, journal tails — read-only plus state flips.
stdout_regex · file_exact · 4000 ms
10 / Shell+ · 18 trials
Pro shell tools
eza, fd, rg, bat, tldr, zoxide over a fixture tree, the Omarchy way.
stdout_regex · 4000 ms
11 / Keys · 20 trials
Hotkey knowledge
Super-chords, tmux chords, and menus from the Omarchy manual.
label · 50 ms
12 / Code · 20 trials
Coding judgment
Read short code, predict its output or spot the bug, and pick the right tool call for an engineering task.
label · calls_exact · 50 ms
13 / Auto · 20 trials
Computer automation
Script small chores in the sandbox: batch renames, log digests, config edits, cron-style jobs.
stdout_regex · file_exact · 4000 ms
Coverage
- easy63
- extreme63
- hard62
- medium62
143 in-process · 107 shell
Measured, never simulated.
Selected Fast 250 provider strike across 250 trials. Badges on /slmb come only from provider rows.
lfm2.5-350m-gtap-v3 · Fast 250 · 250-test run · pass@1
12.0%
Wilson [9%, 17%] · score 0.145 · weighted 0.096 · valid-pass 12.0% (30/250)
242 independent answer shapes at 11.7% · Wilson [8%, 16%]
filter full · samples 1 · temp 0
Follow the evidence.
Move from the official ranking to individual capability profiles, paired model outcomes, suite-wide consensus, and the provider contracts behind every published score.