SLMB 0.1.0 · Full 2500 · gate GREEN · fp b8c02463 · 2,500 trials

small language model bench · every size under 27B

2,500 lean trials for small local-LLM agents.

Full 2500 · 2,500-test run

Pick the right tool.

SLMB ranks language models under 27B parameters, by total parameters, from the smallest up. It asks which small model — light enough for a low-VRAM laptop or a low-tier GPU, fast enough to stream ~200+ tokens a second — can most reliably drive an Omarchy Linux machine. 2188 hand seeds and 312 deterministic trials pass the suite gate; every measured row records pass, latency, tokens, and heap.

Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.

Trial KB-T0102 · Omarchy controlbudget 4000 ms
{
  "prompt": "Apply the catppuccin theme with `omarchy theme set`. The mock records it in .oma-state/theme.",
  "oracle": {
    "expected": "catppuccin\n",
    "kind": "file_exact",
    "path": ".oma-state/theme",
    "ran": [
      "omarchy"
    ]
  }
}

Gate contract, not model output · no model execution on this page

The bar: small, fast, in control.

SLMB ranks models you can actually run — weights that fit a low-tier GPU or a low-VRAM laptop and still hold interactive speed. The board answers one question: which light model gets closest to reliably controlling an Omarchy Linux install — packages, services, shell, keys — not just chatting about it.

Fits

low VRAM

laptop-class weights and low-tier GPUs — no datacenter required

Streams

≥200 tok/s

interactive speed — the model keeps up with the machine

Controls

Omarchy

arch, pacman, systemctl, journal — a real desktop, end to end

The bench at a glance

Suite

2,500

2188 seeds · 312 generated

Gate

GREEN

2500 trials gated

Fingerprint

b8c02463

b8c024631971e34e

Measured

none

slmb strike --mode full --model MODEL_ID

Oracle cost

50.30 ms

mean eval · bounds, not model

Witnessed

1,206/1,206

byte-reading oracles that demand the tool actually ran

Independent cells

2,193

of 2,500 rows — repeated answer shapes are one question

Grades

878 hard · 369 extreme

scored from steps, checked args, and shell composition

Thirteen skills, one bench.

Each family probes one agent capability with its own oracle kinds and enforced time budget. Counts are the live suite — fingerprint b8c024631971.

  1. 01 / Select · 210 trials

    Tool selection

    Pick the right tool from 3–12 announced candidates, then call it.

    calls_exact · 50 ms

  2. 02 / Args · 221 trials

    Argument filling

    Turn a request into one announced call: right keys, values, and enum codes.

    calls_exact · 50 ms

  3. 03 / Flows · 214 trials

    Multi-step flows

    Chain 2–3 calls in order. Prefix partial credit.

    calls_exact · 50 ms

  4. 04 / Shell · 303 trials

    Terminal skill

    Single sandboxed shell command: stdout matches, exit 0.

    stdout_regex · exit_zero · 4000 ms

  5. 05 / Files · 138 trials

    File operations

    Read, write, and inspect files inside the scratch dir.

    stdout_regex · file_exact · 4000 ms

  6. 06 / Classify · 129 trials

    Classifier judgments

    BERT-style labels: sentiment, intent, topic, safety.

    label · 50 ms

  7. 07 / Robust · 158 trials

    Robustness

    Corrupted input keeps the label its clean twin has, by construction.

    label · calls_exact · 50 ms

  8. 08 / Omarchy · 271 trials

    Omarchy control

    Drive the omarchy CLI on a mock machine: themes, updates, capture, brightness.

    stdout_regex · file_exact · 4000 ms

  9. 09 / Arch · 230 trials

    Arch system care

    pacman queries, systemctl state, journal tails — read-only plus state flips.

    stdout_regex · file_exact · 4000 ms

  10. 10 / Shell+ · 140 trials

    Pro shell tools

    eza, fd, rg, bat, tldr, zoxide over a fixture tree, the Omarchy way.

    stdout_regex · 4000 ms

  11. 11 / Keys · 91 trials

    Hotkey knowledge

    Super-chords, tmux chords, and menus from the Omarchy manual.

    label · 50 ms

  12. 12 / Code · 230 trials

    Coding judgment

    Read short code, predict its output or spot the bug, and pick the right tool call for an engineering task.

    label · calls_exact · 50 ms

  13. 13 / Auto · 165 trials

    Computer automation

    Script small chores in the sandbox: batch renames, log digests, config edits, cron-style jobs.

    stdout_regex · file_exact · 4000 ms

  14. Coverage

    • easy381
    • extreme369
    • hard878
    • medium872

    1294 in-process · 1206 shell

Measured, never simulated.

Everything on this page is suite, gate, and harness data. Model results appear here only after a real strike.

No measured runs yet

Everything on this page is suite, gate, and harness data. Model results appear here only after a real strike — start LM Studio, then run:

strike
slmb strike --mode full --model MODEL_ID --samples 3
slmb sync

The retired 1,500-trial editions measured 56 models. Browse the Legacy results.