Skip to content

Measurements

Every number, and how it was taken.

On an RTX 5080 with 16 GB unless a row says otherwise. Dates, settings, repeat counts and the rest of the method are in the measurement notes. Back to the runtime

The full tables

Everything measured.

On an RTX 5080 with 16 GB. Dates, settings, repeat counts and the rest of the method are in the measurement notes.

RTX 5080
RTX 5080measured
Load a 7B chat model and get the first answerabout 4.3 s
gpt-oss 20B reading an 11,052-token documentabout 14 s
Answer starts after a short questionunder 0.1 s
Reading a 2,000-token document, 7B model0.53 s
Chat speed, 7B model150 tokens/s
30B expert model on a 16 GB card, short chat143 tokens/s
Eight users at once, 7B model, shared setting (total)about 350 tokens/s
First picture after starting the app (SD 1.5)about 12 s
Each picture after that (SD 1.5 / SDXL at 512 px)8.3 s / 9.1 s
A minute of audio to textabout 3 s
Reading text aloud2× real time

A token is a piece of a word, about ¾ of a word on average. A 30B expert model uses about 3B of itself for each token, which is why it keeps up on a 16 GB card.

Chat, tokens generated per second
Chat, tokens generated per secondmeasured
Qwen2.5 7B / 14B153.9 / 77.5
Llama 3.1 8B129.8
Mistral 7B v0.3148.0
Qwen3 8B / 4B130.6 / 205.1
Phi-3.5 mini198.7
Gemma 3 4B / Gemma 2 2B180.5 / 258.4
Qwen2.5 7B after 8,192 tokens of conversation122.9

For comparison, llama.cpp reaches 159 tokens/s on the same card and the same 7B file. Measured a second way the 7B here gives 150, which is the figure quoted above.

Reading a prompt · Qwen2.5 7B
Reading a prompt · Qwen2.5 7Bmeasured
55 tokens82 ms
728 tokens195 ms
1,000 tokens263 ms
2,000 tokens530 ms

From 96 tokens up, prompts are read on the card’s tensor cores in int8, on NVIDIA cards whose driver offers it. llama.cpp reads the same 728-token prompt in 114 ms, so long prompts are about 1.7× slower here. Shorter prompts are read the previous way, which is as fast at that size.

Speech-to-text · 73 recorded clips, 1,150 words
Speech-to-text · 73 recorded clips, 1,150 wordsmeasured
Whisper base.en8.4% errors · 28× real time
Whisper small.en5.1% errors · 14× real time
Whisper small (multilingual)6.1% errors · 12× real time
Whisper large-v3-turbo3.3% errors · 18× real time

Word error rate after Whisper’s English text normaliser. 18× real time means a minute of audio in about 3.3 seconds. Reading aloud is measured the same way: Kokoro runs at 2× real time.

Pictures · 20 steps, model already loaded
Pictures · 20 steps, model already loadedmeasured
SDXL, 512 × 5129.1 s
SDXL, 768 × 76815.4 s
SD 1.5, 512 × 5128.3 s
Expert models · Qwen3 30B-A3B Q3_K_S
Expert models · Qwen3 30B-A3B Q3_K_Smeasured
All experts on the card, about 256 tokens of conversation142.7 tokens/s
Chosen automatically on 16 GB, about 10,500 tokens, the rest in PC memory24.3 tokens/s
Granite 3.1 3B-A800M, all experts on the card333 tokens/s
Qwen3 30B-A3B, reading a 4,234-token prompt / a 720-token prompt8.0 s / 1.2 s
gpt-oss-20b MXFP4, all experts on the cardabout 130 tokens/s
gpt-oss-20b, reading a 5,746-token / 11,052-token document5.1 s / 13.7 s

An expert model keeps only the parts it needs on the card. When there is not room for all of them, the most-used stay there and the rest are read from PC memory as needed. Measured on the 5080 only.

Accuracy

Perplexity measures how well a model predicts real text; lower is better. The same file and the same text in VulRunWG and in llama.cpp should give nearly the same number. It checks the model’s arithmetic, not the quality of its answers.

Perplexity · VulRunWG / llama.cpp
Perplexity · VulRunWG / llama.cppmeasured
Llama 3.1 8B26.27 / 26.09
Mistral Nemo 12B18.62 / 18.63
Ministral 8B19.59 / 19.56
Phi-4 14B / Phi-4 mini19.88 / 19.76 · 30.74 / 30.72
Phi-3.5 mini / Phi-3 mini17.21 / 17.20 · 19.24 / 19.15
Gemma 3 1B57.69 / 57.61
DeepSeek-R1-Distill-Qwen 7B69.33 / 69.51
DeepSeek-R1-0528-Qwen3 8B29.22 / 29.43
SmolLM3 3B32.80 / 32.99
Granite 3.3 8B / Granite 3.1 3B-A800M21.59 / 21.58 · 27.06 / 27.11
Qwen3 30B-A3B (Q3_K_S)21.90 / 21.97

Every supported file type was checked the same way. The Granite rows use llama.cpp’s way of splitting text, because VulRunWG follows Granite’s own, which keeps numbers whole.

Support

What it runs, and where.

Operating systems

Windows 11
Every figure on this page except those marked Linux.
Linux
Runs. Chat, pictures, speech and search all work, and on the same laptop it is quicker than Windows. Tried on one machine so far, Linux Mint with an NVIDIA card and an AMD chip.
Windows 10
Not tested.
macOS
Not supported: the engine needs Vulkan.

What you need

PC
Windows 11, with an NVIDIA or AMD graphics card and a current driver. Linux also runs.
Graphics memory
4 GB is enough for smaller models, speech and SD 1.5 pictures. 8 GB runs 7B chat models with about 4,500 tokens of conversation. 16 GB runs 14B and the 30B expert model.
PC memory
Tested on PCs with 32 GB. The 30B expert model needs about 12 GB free.
Disk
About 38 MB for VulRunWG, plus the models you choose (from under 1 GB to about 18 GB each).

What it runs

Chat
Qwen, Llama, Mistral, Phi, Gemma, Granite, SmolLM and DeepSeek models, 0.5B to 30B.
Speech to text
Whisper, base to large-v3-turbo. Long recordings and live dictation.
Read aloud
Kokoro voices (English) and SpeechT5.
Pictures
SD 1.5 and SDXL.
Also
Image understanding, document search, background removal.
Model files
GGUF (Q3_K to Q8_0, IQ4_NL, IQ4_XS), safetensors and checkpoints.