Notes · VulRunWG
Big models, no CUDA: what fits, and why
· written by the developer of VulRunWG
People keep asking whether you can run large AI models without NVIDIA’s CUDA, and on AMD graphics at all. VulRunWG does it every day, and has run on AMD graphics since July 2026. This note shows what fits where, how a model bigger than the graphics card still runs, and where the real limit is. Every number names the machine and the date.
Not a wrapper
Most local AI tools are built on llama.cpp or on PyTorch with CUDA. VulRunWG is built on neither. It has its own engine: about 200 GPU programs written for it in WGSL, run through Vulkan, which a current NVIDIA or AMD driver already includes. There is no CUDA, ROCm or PyTorch to install, which is why the whole download is about 38 MB.
That is checked, not just claimed. One test rigs llama.cpp to crash the moment anything touches it, then loads a model and generates. The answers come out right and the trap never fires. A second check walks everything the running engine loads and must find no PyTorch, ONNX Runtime or llama.cpp.
A 30B model in an 8 GB budget
Mixture-of-experts models, such as Qwen3-30B-A3B, have 30 billion parameters, but only a few “experts” are used for each word. VulRunWG keeps the experts in use on the graphics card and leaves the rest in the PC’s memory until they are needed. The card holds a fraction of the model, and the room it saves goes to a longer conversation.
| Qwen3-30B-A3B (Q3_K_S) | on the card | conversation | speed |
|---|---|---|---|
| Automatic, 16 GB card | 11.1 GB | 10,511 tokens | 20.4 tok/s |
| Held to an 8 GB card’s budget | 5.9 GB | 43,951 tokens | 18.5 tok/s |
RTX 5080, 26 September 2026, with the card budget forced for the second row. Answers were correct in every row. The whole model on the card would need 13.0 GB and leave room for only 256 tokens of conversation.
On a laptop with no graphics card
The Radeon 780M is the graphics built into AMD laptop processors, with 4 GB taken from system memory. No ROCm, no extra drivers: the same VulRunWG download that runs on an RTX 5080.
| Radeon 780M | result |
|---|---|
| Chat, Qwen2.5 1.5B (Windows 11) | 29.6 tok/s |
| Read a 728-token prompt, 1.5B (Windows 11) | 1.6 s |
| Chat, Qwen2.5 3B (Linux) | 13.8 tok/s |
| A minute of speech to text, Whisper (Linux) | 3.0 s |
| Qwen2.5 7B | refused: too big for 4 GB |
Windows 11, 30 September 2026. Linux Mint with Mesa 25.2.8, 28–30 September 2026.
Against llama.cpp, same card, same file
| Qwen2.5 7B Q4_K_M, RTX 5080 | VulRunWG | llama.cpp |
|---|---|---|
| Writing the answer | 150.4 tok/s | 159.0 tok/s |
28 September 2026. llama.cpp build b11146 with its Vulkan backend, run with its own benchmark tool, which carries less overhead than VulRunWG’s figure. Reading a long prompt is still about 1.7× slower than llama.cpp; that is the biggest remaining gap.
Where the limit is
The limit is memory, not a number written into the software. VulRunWG works out what a model needs before loading it, and refuses with a plain message rather than crashing the graphics driver. In practice:
- A mixture-of-experts model can be bigger than the graphics card, as above. It runs slower when experts have to be fetched, and the PC needs enough memory for the rest of the model.
- An ordinary (dense) model has to fit on the card. On the 4 GB 780M that means up to 3B, on an 8 GB card up to 7B, and on a 16 GB card up to 14B.
- One model runs on one graphics card. Splitting a single model across two cards is not built yet.
- A model family runs only after it passes a conformance check against exact arithmetic and against llama.cpp. 15 model files from seven families have passed: Qwen 2.5 and 3, Llama 3, Mistral, Phi-3, Gemma 2 and 3, and SmolLM2. A family that hasn’t passed is refused at load, not run badly.
What hasn’t been tested
- AMD Radeon RX graphics cards. Only the 780M processor graphics has been measured so far.
- Intel graphics.
- Linux beyond one laptop.
- Handhelds such as the ROG Ally or Steam Deck. We don’t own one.
Try it
VulRunWG is in testing and free to use at home, at school and at work. Schools, clinics, nonprofits and developers who want early access can email for early access. The full tables are on the measurements page, with every date, setting and method in the measurement notes.