Using Local LLM Models

Why Use Local Models?

Privacy: Your code and data never leave your machine
Zero cost: No API bills, no rate limits
Offline capability: Works without internet access
Customization: Fine-tune models on your own data
Compliance: Easier to meet data residency requirements

📝

Hardware Requirements

Local models require sufficient RAM/VRAM. Minimum 8 GB RAM for 7B parameter models (quantized). 16 GB recommended for 13B models. GPU acceleration (NVIDIA CUDA, Apple Metal, AMD ROCm) greatly speeds up inference.

Ollama Setup

Ollama is the recommended way to run local models. It provides a simple API compatible with OpenClaw.

Step 1: Install Ollama

# Linux / macOS
curl -fsSL https://ollama.ai/install.sh | sh

# macOS via Homebrew
brew install ollama

# Windows: Download installer from https://ollama.ai/download

Step 2: Pull a Model

# General purpose — best balance
ollama pull llama3.2

# Lightweight and fast
ollama pull phi3

# Good for coding tasks
ollama pull codellama

# High quality reasoning
ollama pull mistral

# Advanced coding
ollama pull qwen2.5-coder

Step 3: Configure OpenClaw

openclaw config set provider ollama
openclaw config set ollama.base_url http://localhost:11434
openclaw config set model llama3.2

Step 4: Start Ollama and Test

# Start Ollama server (runs in background automatically after install)
ollama serve &

# Test OpenClaw with local model
openclaw run "Hello, what model are you?"
# Response: I am Llama 3.2, a local AI model running through Ollama...

Supported Ollama Models

Model	Size	Best For	RAM Required
`llama3.2`	3B	General tasks, fast responses	4 GB
`llama3.2:8b`	8B	Complex reasoning	8 GB
`mistral`	7B	Instruction following	8 GB
`phi3`	3.8B	Efficient, fast	4 GB
`codellama`	7B	Code generation	8 GB
`qwen2.5-coder`	7B	Advanced coding	8 GB

LM Studio Setup

LM Studio provides a GUI for managing and running local models and exposes an OpenAI-compatible API:

Download LM Studio from lmstudio.ai
Download a model through the LM Studio interface
Click "Start Server" in LM Studio (defaults to port 1234)
Configure OpenClaw to use LM Studio:

openclaw config set provider ollama
openclaw config set ollama.base_url http://localhost:1234/v1
openclaw config set model "local-model"

Performance Tips

Use quantized models: Q4 quantization reduces size by 4x with minimal quality loss
Enable GPU acceleration: Set OLLAMA_NUM_GPU=1 to use GPU layers
Limit context window: Use --max-steps 10 for simpler tasks to reduce token usage
Choose the right model size: 3B models are often sufficient for simple file/code tasks

💡

GPU Acceleration

On NVIDIA GPUs, Ollama automatically uses CUDA. On Apple Silicon, Metal acceleration is used automatically. On AMD GPUs, set OLLAMA_ROCM=1. GPU inference can be 10-50x faster than CPU.

💡

See Also

API Keys Setup — cloud provider key configuration
Ollama Integration Guide — detailed Ollama docs
Configuration Examples — privacy-first config example

Routing: Local vs Cloud Per Task

Run lightweight tasks locally (free, private) and complex tasks in the cloud (more capable). Define routing rules in config:

# config.yaml — intelligent routing
provider: auto
routing:
  rules:
    - if: "tokens_estimate < 4096 and complexity == low"
      use: ollama/llama3.2:8b
    - if: "task_type == code_review"
      use: anthropic/claude-3-5-sonnet
    - default:
      use: openai/gpt-4o-mini

# Manually force local/cloud per command
openclaw run "Quick rename task" --provider ollama --model phi3
openclaw run "Full codebase security audit" --provider anthropic

LM Studio Integration

LM Studio provides an OpenAI-compatible API on your local machine. Configure OpenClaw to use it:

# 1. In LM Studio: load a model, then click "Start Local Server"
#    Default: http://localhost:1234/v1

# 2. Configure OpenClaw:
openclaw config set provider openai_compatible
openclaw config set openai_compatible.base_url http://localhost:1234/v1
openclaw config set openai_compatible.api_key lm-studio  # any non-empty string
openclaw config set model "your-loaded-model-name"

# 3. Test:
openclaw run "Say hello" --debug

Hardware Requirements by Model Size

Model Size	RAM (CPU only)	VRAM (GPU)	Speed (M2 Mac)	Examples
3B	4 GB	3 GB	Fast (30+ t/s)	phi3:mini, gemma:2b
7–8B	8 GB	5 GB	Good (15 t/s)	llama3.2:8b, mistral:7b
13B	16 GB	10 GB	Moderate (8 t/s)	llama2:13b, codellama:13b
34B	32 GB	20 GB	Slow (3 t/s)	codellama:34b
70B	64 GB	40 GB	Very slow	llama3.1:70b

GPU Acceleration Setup

# NVIDIA CUDA (check GPU is detected)
ollama run llama3.2 --verbose  # Look for "GPU layers: N" in output

# Apple Silicon (Metal) — automatic, no setup needed
# AMD ROCm on Linux
ROCR_VISIBLE_DEVICES=0 ollama serve

# Verify GPU is being used:
ollama ps        # shows running models + memory location

← API Keys Environment Variables →