Lab = Models + Plumbing
The models are the soul. The plumbing keeps them reachable.
Three thin layers, each replaceable. Nothing here is clever; it is simply kept running.
01 · llama.cpp
Local inference
Gemma 4 E4B and Qwen3 30B-A3B served from CPU RAM on a Proxmox VM. One model resident at a time; swaps are logged.
02 · LiteLLM
One API for all
Local and hosted models (GPT, Claude, Gemini, OpenRouter free tier) behind a single /v1. Keys, budgets and routing live in config.
03 · Cloudflare Access
Private by default
No open ports. Every hostname sits behind a tunnel and an identity check; the API takes a bearer key on top.
Design approach
Every request is visible. Every watt is counted.
Chat through Open WebUI
One interface for every model in the lab. Switch between a local Gemma and a hosted Claude mid-conversation; history, files and prompts stay on the box.
chat.chalard.dev →Observe in Grafana
Tokens per second, which model is currently in RAM, host CPU temperature and power draw. When a 30B model feels slow, the graph says why.
grafana.chalard.dev →Models
Six names, one model field.
gemma4-e4b
local · CPU
Gemma 4 E4B. Fast enough for daily chat on the Xeon; default model.
qwen3-30b-a3b
local · CPU
Qwen3 30B MoE, 3B active. Stronger reasoning, slower load.
gpt-5-mini
hosted · OpenAI
Cheap frontier fallback when a local model is not enough.
claude-sonnet
hosted · Anthropic
Long-context writing and code review.
gemini-flash
hosted · Google
Quick multimodal tasks and large inputs.
or-free
hosted · OpenRouter
Rotating free-tier models for experiments.