Kyle Rockman
Lead Infrastructure Engineer OpsLevel.com
Local LLM
Infinite Tokens For Everyone
Why would I
want Local LLM?
- Data stays local
- No Per-Token Costs
- No 5-hr Limit
- Works Offline
- Tuning
Why Macs are surprisingly good at Local LLM
- Unified Memory CPU & GPU
- CPU is like a GPU
- High CPU-MEM Bandwidth
- 64GB can easily fit 35B
- Low power Draw
Ollama
> brew install ollama
> ollama run qwen3.6:27b-mlx
> ollama launch claude --model qwen3.8:27b-mlx
Easiest To Get Started With
Demo
Next Steps
- Other Servers (omlx, llama.cpp, vllm)
- Fine Tuning (top-p, kv cache, temperature)
- What is MOE? (35B-A3B)
- What is Quantization? (16FP, 8bit, 4bit)
- Model Comparisons (HuggingFace)
- Wisper Flow
- MCP
- Agent Harnesses
Don't ask - "what is the best local model?"
Ask - "what is the smallest model that does the job well?"
Copy of Custom Actions Templates
By Kyle Rockman
