Measure what your machine does with local AI.

Install & run

One command. It walks you through the setup interactively: it detects Ollama, lists your installed models, and asks you to pick one.

npx @opensourcesai/bench

You need

What it measures

Real output from our own machines

The first five runs of the published tool, hours after release, fetched from npm with the command above on the two machines we develop on. They are not mockups and not cherry-picked: the slow run and the refusals are the point. Our records have since grown to a full sweep of the 22 models installed on the Linux machine, including runs that ended with no valid figure at all, kept on file as such.

Generation throughput          143.46 tok/s  (CV 0.06%)
Prefill throughput             9353.02 tok/s  (CV 0.18%)
Time to first token            199.11 ms  (CV 1.13%)
Cold load time                 2.06 s
Pass failure rate              0.00% (0/16 scheduled measured passes)

Measured 2026-08-01 with bench 0.10.0 on Ollama 0.32.5, protocol osai-bench/1.3, scoring osai-bench-derive/1.3. Each result file records all three, so you know which rules produced a number. These numbers describe each machine on its own terms: the two setups run different models under different runtime settings, and they cannot be lined up against each other.

We broke the same model twice, and it caught both

The diagnostics card above is a claim, so here is its test. During a full-catalog sweep on our Linux rig (RTX 3080 10 GB, bench 0.10.0, 2026-08-03) we ran the identical llama3.1:8b Q4_K_M three ways: untouched, pinned to 8 GPU layers, and forced onto the CPU. Same weights, same session, same machine; the only variable is the misconfiguration.

Each figure is the median of 5 measured passes after one warmup, generating 512 tokens from a 45-token prompt in a 4,096-token context at temperature 0 (seed 42). These were measured mid-sweep, which reads about 3.6% below an isolated run of the same model on this rig, so compare the three against each other, not against other figures.

Each diagnostic fired on its own failure mode and stayed silent on the other two runs: no false alarms, no misses. That is also the argument for running the benchmark at all. All three configurations work, Ollama answers prompts in each of them, and the screen gives no sign that anything is wrong. The 12.9× gap is invisible until something measures it. All three numbers come from the same machine in the same session; they say nothing about how this card compares to yours, and they are not meant to.

What this page leaves out

No leaderboards, no percentiles, no “your PC ranks better than X%”. A fair machine-to-machine comparison needs many independent submissions measured under identical rules. The protocol itself requires that before any comparison may be shown, and that corpus does not exist yet. Until it does, your results are a diagnostic of yourmachine, for you.

Scope limits: protocol v1 covers one discrete GPU or CPU-only execution on Windows and Linux. A second GPU is caught by the preconditions and refused. Apple Silicon is the sharper edge, so here is what happens: the benchmark will run on a Mac and complete (we have done it on our own M1), but v1’s detection cannot see an Apple GPU. The run is mislabelled CPU-only, each placement diagnostic returns not-applicable, no bandwidth ceiling gets computed, and nothing in the output warns you. Treat any number produced on a Mac as outside the protocol. Measuring unified memory properly needs protocol work that has not been done yet, and until it is, a clean-looking Mac result is the one output of this tool you should not take at face value.

Questions you will have

Why did the benchmark refuse to run?

That is the run-quality guard doing its job. It refuses when anything could contaminate the measurement: the GPU already above 10% utilization, non-Ollama processes holding more than a token amount of GPU memory, a different model still resident, or more than one GPU (v1 measures one). The usual culprits are a leftover model and jumping back in too fast. ollama stop <name> clears a leftover model, and note that a long OLLAMA_KEEP_ALIVE keeps models resident for hours after you last touched them; after unloading, give the GPU a few seconds to settle. If you must run anyway, --quality-override will, and it marks the result JSON as compromised so the shortcut cannot be laundered out later.

How long does a run take?

Under a minute is typical when the model fits in VRAM. Misconfiguration stretches it far more than model size does. Our worst real case, a 31B model split across GPU and CPU on a 10 GB card, took about half an hour. The tool prints an estimate up front and revises it once, live, when the first real measurements show what your hardware delivers.

Where do my results go?

A single JSON file, osai-bench-result-<timestamp>.json, in the directory you ran it from, and nowhere else. Nothing is uploaded. Useful flags: --model selects a model non-interactively, --output picks the path (it refuses to overwrite an existing file), and --memory-bandwidth supplies the figure by hand if your GPU is missing from the built-in bandwidth table.

Why is time to first token “unavailable” for my thinking model?

Reasoning models can spend the entire measured budget thinking, and Ollama discards that reasoning server-side, so there is no first token to time. The tool reports the metric as unavailable instead of inventing one; since 0.11.0 it also records how many passes withheld output and reports a separate time-to-first-visible-token figure from the long-form workload, which answers the question you were probably asking.

Does it run on my Mac?

It will run and complete, and that is the problem. See the scope section above: v1 cannot see Apple GPUs, so a Mac result carries no diagnostics, no bandwidth ceiling, and no warning that anything is missing. Treat Mac numbers as outside the protocol.

Ran it? Bring the result somewhere useful.

A result file is worth more with someone to read it with, especially a strange one. Our Discord has a #bench-results channel for that: post what your machine measured, attach the JSON (it carries no file paths and nothing personal), and ask what a diagnostic is telling you. Nothing posted there feeds this site’s data. It is somewhere to think out loud, not a collection point.

There is a monthly Cleanest Run challenge too. It turns on the lowest run-to-run variance, which rewards the most carefully measured run rather than the fastest hardware. A modest GPU on a quiet machine can beat a much faster one with a browser left open.

Next steps