In One Sentence

Running an LLM locally means downloading an open-weight language model to your own computer or server and loading it with inference software, so you can chat, write or code without relying on a cloud service.

Why Run Locally

Your data stays on your machine. For contracts, medical records or internal code, keeping data on your own hardware is reassuring.

It works offline. No network needed, and no exposure to provider rate limits or outages.

No per-use fees. Once the hardware is in place, calls don’t cost extra, which suits large or repetitive batch jobs.

Full control. Choose model versions, tune parameters, load fine-tuned weights and experiment freely.

The trade-offs are real too. Model size is capped by your hardware, so local models usually trail the strongest cloud models, and installation, updates and troubleshooting are on you.

Hardware First: Memory Is the Bottleneck

The first question is whether the model weights fit in memory. A rough estimate:

Weight memory ≈ parameter count × bytes per parameter

A 7-billion-parameter (7B) model stored at 16 bits (2 bytes) per parameter needs about 14 GB for the weights alone. Quantized to 4 bits (0.5 bytes), it needs about 3.5 GB. At runtime you also need room for the context cache (KV cache) and the inference software itself, and longer contexts use more.

Common hardware setups:

  • Discrete GPU: the fastest option. VRAM size is the number that matters. Things run best when the whole model fits in VRAM.
  • Unified-memory machines: such as Macs with Apple Silicon, where CPU and GPU share memory. These can load larger models than a similarly priced discrete GPU.
  • CPU only: works, especially for small quantized models, but generation is noticeably slower.

If VRAM falls short, some inference tools can put part of the model on the GPU and the rest in system RAM. It runs, just more slowly.

Quantization: Slimming Models Down

Quantization is the workhorse of local inference. It converts weights from high-precision numbers (such as 16-bit floats) to lower-precision forms (such as 8-bit or 4-bit integers), cutting memory use and speeding up inference.

It costs some accuracy, and the loss grows as bit width shrinks. Around 4 bits is commonly seen as a reasonable balance between size and quality, but results vary by model and task, so test for yourself.

Common formats in the local ecosystem include GGUF, used by llama.cpp, and GPU-oriented methods such as GPTQ and AWQ. Model file names usually indicate the quantization used.

Ollama. A command-line tool that downloads and runs a model with one command and exposes a local API for other programs. Great for getting started fast.

LM Studio. A desktop app for browsing, downloading and chatting with models, with an optional local server. Good if you’d rather skip the terminal.

llama.cpp. A C/C++ inference engine supporting CPUs and several GPU backends. Many higher-level tools are built on it. Good when you want fine-grained control.

vLLM and other server frameworks. Built for serving many users from GPU servers, with a focus on throughput and concurrency.

Many local tools offer APIs compatible with popular cloud API formats, so a program written for a cloud model can often switch to a local one by changing the endpoint URL.

Getting Started

Using Ollama as an example:

  1. Download and install Ollama from the official site.
  2. Pick a model from its library with a parameter count that suits your hardware.
  3. Run ollama run <model-name> in a terminal. The first run downloads the weights.
  4. Once downloaded, chat right in the terminal.
  5. To use it from other apps, call the HTTP API Ollama serves on your machine.

Start with a smaller model, confirm it runs smoothly, then try larger ones.

Common Misconceptions

“Local models match top cloud models.” Hardware limits mean you’ll usually run small to mid-sized models, which can fall short on complex reasoning or long documents. Give them the tasks they handle well, and keep cloud models for the hard ones.

“A fast GPU is all you need.” VRAM capacity often matters more than raw compute. If the model doesn’t fit, speed drops sharply no matter how powerful the card is.

“Lower-bit quantization is always a better deal.” Aggressive quantization can visibly hurt output, causing contradictions or garbled language. Find the balance between size and quality.

“Local means fully secure.” Models and tools still come from the internet, so download from official or trusted sources. If you enable a local API server, don’t expose it to the public internet without access control.

“Open weights means free for any commercial use.” Licenses vary widely, and some restrict commercial use. Read the license that ships with the model.

Further Reading