Running AI Locally on a Small Machine: What Is Realistic (and What Is Not)

Local AI’s real appeal

Running a language model on your own machine means privacy, no per-message cost, and tools that work offline. Tools like Ollama (see the Tools Directory) made installation genuinely simple. The question is not “does it run?” — it is “what runs at a usable speed on your hardware?”

The hardware truth

Model size is the whole story. On a modest machine — a couple of CPU cores, no dedicated GPU, single-digit gigabytes of RAM:

• Small models (around half a billion to a few billion parameters) respond in seconds and handle drafting, summarising, simple Q&A and code completion at a basic level.
• Mid-size models may load but answer slowly enough to break a workflow.
• Large models are effectively unusable without serious RAM and preferably a GPU. Downloading one anyway just fills your disk.

Good jobs for a small local model

• First drafts and headline variations while offline.
• Summarising your own notes and transcripts.
• A private scratchpad for ideas you do not want in a cloud service.
• Learning how prompts and context actually behave, at zero cost.

Bad jobs

Anything where a wrong answer is expensive, anything needing current information, and long creative writing where quality differences are obvious. Use a cloud assistant — within its current free limits — for those, and verify important facts either way.

A sane setup checklist

1. Install from the official site only.
2. Start with the smallest recommended model and test real speed.
3. Keep models on a disk with room to spare; they are large files.
4. Talk to the local API rather than interactive terminals for repeatable tests.

The bottom line

Local AI on small hardware is a useful apprentice, not a supercomputer. Match the model to the machine and the job to the model, and it earns its place in a free creator stack.