May 2026
The most practical AI setup I've built isn't a single powerful box — it's three ordinary machines that each do one thing well.
Every self-hosted AI guide I've read assumes you'll run everything on one machine. A Linux server with a big GPU, Ollama, a vector database, maybe a web interface. Clean, simple, one box to rule them all.
I tried that. It fights you.
The GPU wants Windows drivers and DirectML. The web server wants to live behind a tunnel in a sandboxed VM. The development tools want a daily driver that isn't either of those things. Squeezing it all into one host means constant compromise — driver conflicts, resource contention, and a nagging feeling that you're one `apt upgrade` away from breaking the whole stack.
So I split it. Three machines, three roles, one coherent system. Here's why that works better than anything you can do with a single server.
The architecture breaks into three functional layers. Each machine handles exactly one role, and they communicate over plain HTTP on a local network — no complex orchestration, no Kubernetes, no service mesh.
This machine has exactly one job: touch the internet so nothing else has to.
It runs a minimal Linux install with nginx, a Cloudflare tunnel, and the agent framework. No GPU, no heavy databases, no development tools. Just a web server and the AI agent that coordinates workflows.
The agent living here is the closest thing to a "brain" in the stack — it receives tasks, decides which model to call, retrieves context from the knowledge base, and formats responses. But it doesn't do any heavy computation itself. It's a dispatcher, not a worker.
The isolation is deliberate. If something goes wrong here — a compromised dependency, a malformed request, a tunnel breach — the blast radius is limited to this one VM. The GPU host and the development environment are on separate machines, reachable only from the local network, with no public exposure.
This is the machine with the expensive GPU. In my case, it's an AMD Radeon RX 6900 XT running on Windows with Ollama, handling all model inference for the entire stack.
This machine has no public services. No web server, no SSH (Windows doesn't enable it by default), no exposed ports beyond what Ollama needs on the local network. It sits behind the router, unreachable from outside, and does exactly one thing: load models and run inference.
The design decision here was pragmatic, not architectural. The GPU was already installed in a gaming PC running Windows. Rather than repurpose that machine, dual-boot it, or attempt GPU passthrough to a Linux VM — all of which introduce their own complexity and failure modes — I left it as a Windows desktop and pointed Ollama at the local network. Linux machines call its API. Windows continues to handle its original job. Everyone wins.
The models loaded here range from ~7B to ~12B parameters — Gemma 4, Qwen 3.5, DeepSeek variants. Enough for agent reasoning, code generation, and document analysis. Not enough for bleeding-edge 70B models, but those rarely fit in a homelab budget anyway.
The development workstation. This is where code gets written, containers get orchestrated, databases get queried, and the knowledge base gets maintained.
This machine runs multiple Ollama instances (one for embeddings, one for lighter chat models), the MCP servers that connect the AI agent to tools and data, the vector database client, and the development environment itself. It's the most active host in the stack but also the most replaceable — if it goes down, the other two machines continue serving and inferring without it. I just can't develop or query the knowledge base until it's back.
The communication pattern is deliberately simple: HTTP across a flat local network.
When the agent on Host A needs to answer a question, it sends a POST to Host B's Ollama API with the prompt and model name. Host B does the compute, streams tokens back, and the agent formats the response. If the agent needs to check the knowledge base, it queries the vector database running on Host A itself (it's co-located with the agent for latency reasons). If I'm developing a new tool or workflow from Host C, I can call either of the other machines directly.
No message queues. No service discovery. No container orchestration. Just three machines that know each other's IPs and speak plain HTTP. It's not elegant. It's not cloud-native. It works without fail every single time, and when something breaks, the failure is obvious — a timeout, a connection refused, a model not found — instead of being buried in a distributed tracing log.
The obvious question: why not consolidate everything into a single Linux box with a passed-through GPU?
I've tried that path. GPU passthrough to a KVM VM is fragile across kernel updates. NVIDIA drivers on Linux are a known pain point — I've spent more time debugging `nvidia-smi` not finding the card than I have actually running models. Windows Subsystem for Linux works for development but has quirks with USB device access, networking, and GPU sharing.
Beyond the technical friction, the blast radius argument convinced me. If everything runs on one host, a bad update breaks everything. A power supply failure takes down the website, the AI agent, and the development environment simultaneously. With three machines, I have natural fault isolation. Host A keeps serving pages even if Host C is rebooting. Host B keeps running inference even if Host A is re-deploying.
The cost argument works too. Three used machines with reasonable specs cost less than one high-end workstation with a datacenter GPU. And when I upgrade, I upgrade one component at a time — a better GPU for Host B, more RAM for Host C, a faster SSD for Host A — instead of replacing the entire stack.
Network latency is not a problem. A 1 Gb local network adds maybe 2-3 milliseconds to inference time. Token generation is dominated by GPU compute, not network round trips. I can't tell the difference between running a model locally and calling it across the wire.
Windows as a server is harder than it should be. The GPU host runs Windows, which means no SSH, no headless management, and no automation without RDP. Every update requires sitting at the console or VNCing into the desktop. It works, but it's the least "infrastructure-as-code" part of the stack.
The agent's location matters. I originally ran the agent on the control plane alongside development tools. But the agent needs to be on the same host as the web server and the tunnel to minimize latency on user-facing requests. Moving it to Host A improved response times noticeably — the agent now lives where the traffic comes in.
The vector database is the most critical component. People think the GPU model is the center of the stack. It's not. The knowledge base — what the model retrieves context from — is what makes the system useful. Without good retrieval, the model is just a generic chatbot. The vector database outranks the GPU in practical importance.
It's not simpler. Managing three machines is more work than managing one. You need a network that works, a backup strategy that covers multiple hosts, and the discipline to not let the stack sprawl into something unmanageable.
But the ceiling is higher. When a single machine can't do everything you need — and if you're running AI agents, it won't be long before it can't — splitting the stack across purpose-built hosts is the natural next step. Three machines working together will outperform one machine trying to be everything.
And honestly? It's more fun. Each host has a distinct personality and a clear job. Debugging a distributed system across three machines is satisfying in a way that fixing a single overloaded server never is.
Built on a home lab, powered by local models, and owned by Andrew Katana.