SLMB 0.1.0 · Fast 250 · gate GREEN · fp 71f186ff · 250 trials

small language model bench · every size under 27B

250 lean trials for small local-LLM agents.

Fast 250 · 250-test run

Pick the right tool.

SLMB ranks language models under 27B parameters, by total parameters, from the smallest up. It asks which small model — light enough for a low-VRAM laptop or a low-tier GPU, fast enough to stream ~200+ tokens a second — can most reliably drive an Omarchy Linux machine. 216 hand seeds and 34 deterministic trials pass the suite gate; every measured row records pass, latency, tokens, and heap.

Fast 250 measured lfm2.5-350m-gtap-v3 · 12.0% pass@1

Fast 250 · a fixed 250-test run drawn unchanged from Full 2500 — standard reply allowance 3,750 tokens. Full 2500 · the complete 2,500-test run — standard reply allowance published trial caps. Separate leaderboards and evidence; a Fast score is never a Full score or estimate. Provider-time figures reflect each run's recorded conditions — shared-GPU or concurrent timing is not standalone tok/s.

Trial KB-T0012 · Tool selectionbudget 50 ms
{
  "prompt": "The user says: \"save Cy's number 555-0100.\" Announce exactly one call from the announced tools.",
  "oracle": {
    "calls": [
      {
        "match": "subset",
        "name": "add_contact"
      }
    ],
    "kind": "calls_exact"
  }
}

Gate contract, not model output · no model execution on this page

The bar: small, fast, in control.

SLMB ranks models you can actually run — weights that fit a low-tier GPU or a low-VRAM laptop and still hold interactive speed. The board answers one question: which light model gets closest to reliably controlling an Omarchy Linux install — packages, services, shell, keys — not just chatting about it.

Fits

low VRAM

laptop-class weights and low-tier GPUs — no datacenter required

Streams

≥200 tok/s

interactive speed — the model keeps up with the machine

Controls

Omarchy

arch, pacman, systemctl, journal — a real desktop, end to end

The bench at a glance

Suite

250

216 seeds · 34 generated

Gate

GREEN

250 trials gated

Fingerprint

71f186ff

71f186ff72776cc0

Measured

12.0%

lfm2.5-350m-gtap-v3 · 250 trials · pass@1

Oracle cost

47.06 ms

mean eval · bounds, not model

Witnessed

107/107

byte-reading oracles that demand the tool actually ran

Independent cells

242

of 250 rows — repeated answer shapes are one question

Grades

62 hard · 63 extreme

scored from steps, checked args, and shell composition

Thirteen skills, one bench.

Each family probes one agent capability with its own oracle kinds and enforced time budget. Counts are the live suite — fingerprint 71f186ff7277.

  1. 01 / Select · 16 trials

    Tool selection

    Pick the right tool from 3–12 announced candidates, then call it.

    calls_exact · 50 ms

  2. 02 / Args · 20 trials

    Argument filling

    Turn a request into one announced call: right keys, values, and enum codes.

    calls_exact · 50 ms

  3. 03 / Flows · 20 trials

    Multi-step flows

    Chain 2–3 calls in order. Prefix partial credit.

    calls_exact · 50 ms

  4. 04 / Shell · 16 trials

    Terminal skill

    Single sandboxed shell command: stdout matches, exit 0.

    stdout_regex · exit_zero · 4000 ms

  5. 05 / Files · 20 trials

    File operations

    Read, write, and inspect files inside the scratch dir.

    stdout_regex · file_exact · 4000 ms

  6. 06 / Classify · 20 trials

    Classifier judgments

    BERT-style labels: sentiment, intent, topic, safety.

    label · 50 ms

  7. 07 / Robust · 20 trials

    Robustness

    Corrupted input keeps the label its clean twin has, by construction.

    label · calls_exact · 50 ms

  8. 08 / Omarchy · 20 trials

    Omarchy control

    Drive the omarchy CLI on a mock machine: themes, updates, capture, brightness.

    stdout_regex · file_exact · 4000 ms

  9. 09 / Arch · 20 trials

    Arch system care

    pacman queries, systemctl state, journal tails — read-only plus state flips.

    stdout_regex · file_exact · 4000 ms

  10. 10 / Shell+ · 18 trials

    Pro shell tools

    eza, fd, rg, bat, tldr, zoxide over a fixture tree, the Omarchy way.

    stdout_regex · 4000 ms

  11. 11 / Keys · 20 trials

    Hotkey knowledge

    Super-chords, tmux chords, and menus from the Omarchy manual.

    label · 50 ms

  12. 12 / Code · 20 trials

    Coding judgment

    Read short code, predict its output or spot the bug, and pick the right tool call for an engineering task.

    label · calls_exact · 50 ms

  13. 13 / Auto · 20 trials

    Computer automation

    Script small chores in the sandbox: batch renames, log digests, config edits, cron-style jobs.

    stdout_regex · file_exact · 4000 ms

  14. Coverage

    • easy63
    • extreme63
    • hard62
    • medium62

    143 in-process · 107 shell

Measured, never simulated.

Selected Fast 250 provider strike across 250 trials. Badges on /slmb come only from provider rows.

lfm2.5-350m-gtap-v3 · Fast 250 · 250-test run · pass@1

12.0%

Wilson [9%, 17%] · score 0.145 · weighted 0.096 · valid-pass 12.0% (30/250)

242 independent answer shapes at 11.7% · Wilson [8%, 16%]

filter full · samples 1 · temp 0