How to Run an LLM Locally: Ollama and LM Studio Tutorial

Running a large language model on your own computer used to mean compiling C++ and hunting for model weights. In 2026 it takes a few minutes: install Ollama or LM Studio, download a model, and you have a private, offline chatbot plus an OpenAI-compatible API on localhost. This tutorial shows how to run an LLM locally step by step, how much RAM and VRAM you need, which models to start with, and how to call your local model from code.

Why run an LLM locally?

Privacy: prompts and documents never leave your machine, which matters for client code, contracts or medical notes.

No per-token bills: after the hardware, inference is free, so you can experiment, batch-process files or run agents all day.

Offline use: once a model is downloaded, it works on a plane or in an air-gapped network.

Control: you pick the exact model and version, so behaviour does not change underneath you.

The trade-off is quality and speed. Open-weight models that fit on a laptop are very capable for summarising, drafting, coding help and retrieval-augmented generation, but the largest hosted frontier models are still stronger on hard reasoning and long agentic tasks. If you are new to the terminology, read what an LLM is first.

How much RAM and VRAM do you need?

Model size is measured in parameters, and local tools run quantized models, usually 4-bit, to save memory. A useful rule of thumb for 4-bit (Q4) models is about 0.6 GB of memory per billion parameters, plus 1 to 4 GB for the context window (the KV cache), which grows with longer prompts.

3B to 4B models: about 3 to 4 GB. Runs on almost any modern laptop, even CPU-only.

7B to 9B models: about 6 to 8 GB. A good default on an 8 GB GPU or a 16 GB Mac.

12B to 14B models: about 10 to 12 GB. Needs a 12 to 16 GB GPU or a Mac with 24 GB or more.

27B to 32B models: about 18 to 24 GB. Fits a 24 GB GPU or a 32 GB+ Mac.

70B models: about 40 to 48 GB. Needs two GPUs, a 48 GB workstation card, or a Mac with 64 GB or more.

On Apple Silicon, the GPU shares unified memory with the system, so a 32 GB Mac can dedicate roughly two thirds of it to a model. On PCs, NVIDIA GPUs with CUDA are the best supported, and AMD GPUs work through ROCm or Vulkan in most tools. Without a GPU, models still run on the CPU from system RAM, just more slowly. As a rough guide, 10 or more tokens per second feels interactive for chat.

Quantization labels you will see:

Q4_K_M: the common default, good quality at about a quarter of the original size.

Q5_K_M and Q6_K: slightly better quality, more memory.

Q8_0: near-original quality, about half the original size.

GGUF is the file format used by llama.cpp, Ollama and LM Studio; MLX is Apple's format, which LM Studio can use on Macs.

Option 1: run an LLM locally with Ollama

Ollama is a free, open source tool that downloads, runs and serves models with one command. It has desktop apps for macOS and Windows and a one-line installer for Linux.

Install Ollama

On macOS or Windows, download the app from ollama.com. On Linux, run:

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

Run your first model

ollama run llama3.2:3b

The first run downloads the model (about 2 GB), then drops you into a chat prompt. Type a question, and use /bye to exit. Model names follow the pattern name:tag, where the tag usually encodes the size. Tags change as new models are released, so browse ollama.com/library and pick a model that fits your memory. Good starting points are a 3B to 4B model for low-end machines, a 7B to 9B general model such as a Qwen, Llama, Gemma or Mistral variant for 8 GB GPUs, and a dedicated coding model if you mainly want programming help.

The everyday commands:

ollama pull qwen3:8b
ollama list
ollama ps
ollama show qwen3:8b
ollama rm llama3.2:3b

ollama ps shows which models are loaded and whether they run on the GPU or the CPU. If you see a CPU percentage when you expected GPU, the model is too large for your VRAM; choose a smaller model or a lower quantization.

Increase the context window

Small default context windows are the most common cause of a local model "forgetting" the start of a long document. Set a larger default for the server with an environment variable, keeping in mind that more context uses more memory:

OLLAMA_CONTEXT_LENGTH=16384 ollama serve

Or bake settings into your own model variant with a Modelfile:

FROM qwen3:8b
PARAMETER num_ctx 16384
PARAMETER temperature 0.3
SYSTEM You are a concise senior software engineer. Answer with code first.

ollama create code-helper -f Modelfile
ollama run code-helper

Call Ollama from code

Ollama serves an HTTP API on port 11434. Its native chat endpoint looks like this:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Explain a race condition in two sentences."}],
  "stream": false
}'

It also exposes an OpenAI-compatible endpoint under /v1, so existing OpenAI SDK code works by changing the base URL. Install the SDK with pip install openai, then:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
completion = client.chat.completions.create(
    model="qwen3:8b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a Python function that reverses a string."},
    ],
)
print(completion.choices[0].message.content)

The API key is ignored by Ollama but required by the SDK. This makes it easy to develop against a local model and switch to a hosted one later. See the OpenAI API in Python tutorial for the hosted side.

Option 2: run an LLM locally with LM Studio

LM Studio is a free desktop app for macOS, Windows and Linux with a graphical interface, which makes it the easiest way to try local models if you prefer not to use a terminal.

Download LM Studio from lmstudio.ai and install it.

Open the Discover tab and search for a model. LM Studio shows each download's size and marks whether it is likely to fit in your memory.

Pick a quantization (Q4_K_M is a sensible default; on Apple Silicon you can also choose MLX builds) and download it.

Open the Chat tab, load the model and start chatting. You can attach documents for simple retrieval and adjust the context length, GPU offload and temperature in the side panel.

To use the model from code, open the Developer tab and start the local server. It listens on port 1234 with OpenAI-compatible endpoints, so the Python example above works with base_url set to http://localhost:1234/v1. LM Studio also ships a CLI called lms:

lms get qwen3-8b
lms load qwen3-8b
lms server start
lms ps

Ollama vs LM Studio: which should you use?

Choose Ollama if you live in the terminal, want a background service other apps can call, run on a headless Linux server, or want to script model management.

Choose LM Studio if you want a polished GUI, like browsing Hugging Face models visually, or want MLX models on a Mac.

Choose vLLM or a similar server when several users or an application with real traffic need the model at once. Ollama and LM Studio are built for one person; vLLM adds continuous batching and much higher throughput on GPUs.

Our Ollama vs vLLM vs LM Studio comparison goes deeper into performance and deployment trade-offs.

Performance and security tips

Keep the whole model on the GPU. Partial CPU offload works but can be several times slower. If a model barely fits, drop to a smaller size before dropping below Q4.

Close other GPU-heavy apps. Browsers with hardware acceleration and games compete for VRAM.

Mind the context length. Doubling the context roughly doubles KV cache memory. Use what you need, not the maximum.

Do not expose the API to the internet. By default Ollama listens only on localhost. Setting OLLAMA_HOST=0.0.0.0 makes it reachable from your network without authentication, so put it behind a reverse proxy with auth or a VPN such as Tailscale if you need remote access.

Check model licences. Open-weight models come with different licences; some restrict commercial use or require attribution.

Next steps

Once a local model is running, the natural next projects are a private document chat using retrieval-augmented generation, a local coding assistant in your editor, or an agent built with the LangChain tutorial, which works with local models through the langchain-ollama integration. Because both Ollama and LM Studio speak the OpenAI API format, almost any tool that supports a custom OpenAI base URL can use your local LLM.