Measurements
Every number, and how it was taken.
On an RTX 5080 with 16 GB unless a row says otherwise. Dates, settings, repeat counts and the rest of the method are in the measurement notes. Back to the runtime
The full tables
Everything measured.
On an RTX 5080 with 16 GB. Dates, settings, repeat counts and the rest of the method are in the measurement notes.
| RTX 5080 | measured |
|---|---|
| Load a 7B chat model and get the first answer | about 4.3 s |
| gpt-oss 20B reading an 11,052-token document | about 14 s |
| Answer starts after a short question | under 0.1 s |
| Reading a 2,000-token document, 7B model | 0.53 s |
| Chat speed, 7B model | 150 tokens/s |
| 30B expert model on a 16 GB card, short chat | 143 tokens/s |
| Eight users at once, 7B model, shared setting (total) | about 350 tokens/s |
| First picture after starting the app (SD 1.5) | about 12 s |
| Each picture after that (SD 1.5 / SDXL at 512 px) | 8.3 s / 9.1 s |
| A minute of audio to text | about 3 s |
| Reading text aloud | 2× real time |
A token is a piece of a word, about ¾ of a word on average. A 30B expert model uses about 3B of itself for each token, which is why it keeps up on a 16 GB card.
| Chat, tokens generated per second | measured |
|---|---|
| Qwen2.5 7B / 14B | 153.9 / 77.5 |
| Llama 3.1 8B | 129.8 |
| Mistral 7B v0.3 | 148.0 |
| Qwen3 8B / 4B | 130.6 / 205.1 |
| Phi-3.5 mini | 198.7 |
| Gemma 3 4B / Gemma 2 2B | 180.5 / 258.4 |
| Qwen2.5 7B after 8,192 tokens of conversation | 122.9 |
For comparison, llama.cpp reaches 159 tokens/s on the same card and the same 7B file. Measured a second way the 7B here gives 150, which is the figure quoted above.
| Reading a prompt · Qwen2.5 7B | measured |
|---|---|
| 55 tokens | 82 ms |
| 728 tokens | 195 ms |
| 1,000 tokens | 263 ms |
| 2,000 tokens | 530 ms |
From 96 tokens up, prompts are read on the card’s tensor cores in int8, on NVIDIA cards whose driver offers it. llama.cpp reads the same 728-token prompt in 114 ms, so long prompts are about 1.7× slower here. Shorter prompts are read the previous way, which is as fast at that size.
| Speech-to-text · 73 recorded clips, 1,150 words | measured |
|---|---|
| Whisper base.en | 8.4% errors · 28× real time |
| Whisper small.en | 5.1% errors · 14× real time |
| Whisper small (multilingual) | 6.1% errors · 12× real time |
| Whisper large-v3-turbo | 3.3% errors · 18× real time |
Word error rate after Whisper’s English text normaliser. 18× real time means a minute of audio in about 3.3 seconds. Reading aloud is measured the same way: Kokoro runs at 2× real time.
| Pictures · 20 steps, model already loaded | measured |
|---|---|
| SDXL, 512 × 512 | 9.1 s |
| SDXL, 768 × 768 | 15.4 s |
| SD 1.5, 512 × 512 | 8.3 s |
| Expert models · Qwen3 30B-A3B Q3_K_S | measured |
|---|---|
| All experts on the card, about 256 tokens of conversation | 142.7 tokens/s |
| Chosen automatically on 16 GB, about 10,500 tokens, the rest in PC memory | 24.3 tokens/s |
| Granite 3.1 3B-A800M, all experts on the card | 333 tokens/s |
| Qwen3 30B-A3B, reading a 4,234-token prompt / a 720-token prompt | 8.0 s / 1.2 s |
| gpt-oss-20b MXFP4, all experts on the card | about 130 tokens/s |
| gpt-oss-20b, reading a 5,746-token / 11,052-token document | 5.1 s / 13.7 s |
An expert model keeps only the parts it needs on the card. When there is not room for all of them, the most-used stay there and the rest are read from PC memory as needed. Measured on the 5080 only.
Accuracy
Perplexity measures how well a model predicts real text; lower is better. The same file and the same text in VulRunWG and in llama.cpp should give nearly the same number. It checks the model’s arithmetic, not the quality of its answers.
| Perplexity · VulRunWG / llama.cpp | measured |
|---|---|
| Llama 3.1 8B | 26.27 / 26.09 |
| Mistral Nemo 12B | 18.62 / 18.63 |
| Ministral 8B | 19.59 / 19.56 |
| Phi-4 14B / Phi-4 mini | 19.88 / 19.76 · 30.74 / 30.72 |
| Phi-3.5 mini / Phi-3 mini | 17.21 / 17.20 · 19.24 / 19.15 |
| Gemma 3 1B | 57.69 / 57.61 |
| DeepSeek-R1-Distill-Qwen 7B | 69.33 / 69.51 |
| DeepSeek-R1-0528-Qwen3 8B | 29.22 / 29.43 |
| SmolLM3 3B | 32.80 / 32.99 |
| Granite 3.3 8B / Granite 3.1 3B-A800M | 21.59 / 21.58 · 27.06 / 27.11 |
| Qwen3 30B-A3B (Q3_K_S) | 21.90 / 21.97 |
Every supported file type was checked the same way. The Granite rows use llama.cpp’s way of splitting text, because VulRunWG follows Granite’s own, which keeps numbers whole.
Support
What it runs, and where.
Operating systems
- Windows 11
- Every figure on this page except those marked Linux.
- Linux
- Runs. Chat, pictures, speech and search all work, and on the same laptop it is quicker than Windows. Tried on one machine so far, Linux Mint with an NVIDIA card and an AMD chip.
- Windows 10
- Not tested.
- macOS
- Not supported: the engine needs Vulkan.
What you need
- PC
- Windows 11, with an NVIDIA or AMD graphics card and a current driver. Linux also runs.
- Graphics memory
- 4 GB is enough for smaller models, speech and SD 1.5 pictures. 8 GB runs 7B chat models with about 4,500 tokens of conversation. 16 GB runs 14B and the 30B expert model.
- PC memory
- Tested on PCs with 32 GB. The 30B expert model needs about 12 GB free.
- Disk
- About 38 MB for VulRunWG, plus the models you choose (from under 1 GB to about 18 GB each).
What it runs
- Chat
- Qwen, Llama, Mistral, Phi, Gemma, Granite, SmolLM and DeepSeek models, 0.5B to 30B.
- Speech to text
- Whisper, base to large-v3-turbo. Long recordings and live dictation.
- Read aloud
- Kokoro voices (English) and SpeechT5.
- Pictures
- SD 1.5 and SDXL.
- Also
- Image understanding, document search, background removal.
- Model files
- GGUF (Q3_K to Q8_0, IQ4_NL, IQ4_XS), safetensors and checkpoints.