← all posts

Self-Hosted AI Stack: Architecture Reference

May 2026

A technical companion to "AI Lost My Files" — the actual infrastructure behind the story.

Safety first. No AI model or agent runs outside a sandbox. The framework for containment changes with the use case (filesystem, container, or VM access rights have been granted), but AI has never been granted root-level access to bare-metal resources.

Architecture Overview

The guiding principle is a hard separation between what touches the internet and what doesn't.


                         INTERNET
                     visitors → sites
                            │
                     CloudFlare tunnel
                     (only ingress point)
                            │
              ═════════════╧══════════════  DMZ
              ║  exposed via tunnel only  ║
              ║                           ║
              ║  .176 (Ubuntu VM)         ║
              ║  ├─ nginx                 ║
              ║  │  atkatana.com          ║
              ║  ├─ Hermes agent          ║
              ║  └─ cloudflared gateway   ║
              ║                           ║
              ║  .120 (Garuda VM)         ║
              ║  └─ ask.atkatana.com      ║
              ╚════════════╤═══════════════╝
                           │ 10Gb data plane
              ╔════════════╧═══════════════╗
              ║  behind home firewall      ║
              ║                           ║
              ║  .16 Dev Hub              ║
              ║  ├─ Ollama :11434         ║
              ║  ├─ OpenCode / OB1        ║
              ║  └─ Homelable             ║
              ║                           ║
              ║  .154 Windows (GPU)       ║
              ║  └─ Ollama (6900 XT)      ║
              ╚═══════════════════════════╝

         .113 hpz840 — KVM host for both DMZ VMs
         VM disks on .112 NFS (10Gb attached)
        

The CloudFlare tunnel is the only open door to the internet. Everything in the top half is reachable from the outside — and only through that tunnel. Everything in the bottom half (AI tools, LLM, development agent) has no public route. They communicate across a 10 Gb data plane that ties the KVM host to the internal lab network.


How It Evolved (The Journey)

Phase 1: Single Arch VM

I started with an Arch Linux VM that was doing nothing in particular — I had tried to make it a temporary desktop but didn't have enough GPU bandwidth for a good experience. It became the first Hermes host, paired with Ollama on a remote Windows machine for GPU inference.

The goal was simple: AI-assisted coding and workflow without a subscription.

Early limitations became obvious quickly:

  • No persistence — Hermes had a working model but couldn't track anything beyond a single session
  • No shared context — no filesystem, no long-term memory
  • No knowledge base — markdown files and a simple wiki were placeholders
  • Phase 2: Adding Memory

    The persistence problem was solved by deploying [Open Brain (OB1)](https://github.com/NateBJones-Projects/OB1) — a vector database with an MCP server. This gave the Hermes agent a single source of truth that anything on the network could query.

    With OB1 in place, the job search stack came together:

  • A job scraper (LinkedIn → OB1)
  • Email automation
  • Application tracking
  • RAG-backed Q&A at ask.atkatana.com
  • Phase 3: Physical Consolidation

    The model choice landed on Qwen 3.5:9B running on an AMD 6900 XT (16 GB VRAM) on a Windows host. Two VM hosts became the final deployment target — spreading the load and running without intervention.

    But the migration uncovered a set of operational surprises that were harder to catch than the software configuration.


    Operational Lessons

    Dual Tunnel Instances (The Power-Cycle Problem)

    When planning the migration, I moved the CloudFlare tunnel setup to its own VM, then later to a container when power cycles during thunderstorms exposed startup-ordering issues.

    A week and a half later, both the temporary VM and the container were running simultaneously. Both had autostart enabled. Neither had been turned off during the transition. When the power blinked, both came back up.

    Not a hallucination. Not a planning failure. Just a missed detail — a loop that no one closed.

    Boot Ordering

    The VMs need shared storage up before they can start. This required:

  • Staggered VM boot (TrueNAS → 60s → Hermes → 30s → Worker4)
  • Systemd oneshot service on the KVM host
  • NFS mount dependency before VM start
  • Docker restart delay for Jellyfin after NFS is ready
  • Every layer adds a timing dependency. Each one was discovered because something broke on reboot.


    Host Inventory

    .16 — Dev Hub (HP Z640 Workstation)

    Layer Component Details
    Hardware HP Z640 2x Xeon E5-2687W v4 (24C/48T), 64 GB RAM, RTX 3070 Ti
    OS Garuda Linux (Arch) Rolling release
    Inference Ollama :11434 Local LLM runtime, nomic-embed-text for OB1 embeddings
    Database open_brain.db SQLite knowledge base, 324+ thoughts, 768-dim vectors
    Agent opencode AI coding assistant with remote MCP to OB1 + Homelable
    Container Jellyfin :8096 Media server (music, movies, photos)
    Container Homelable frontend :3000 Network topology UI
    Container Homelable MCP :8001 Network discovery + management API
    Container Homelable backend Graph data store, SQLite
    Container IPAM server IP address management
    Container Nexterm Terminal/SSH web client
    GPU NVIDIA RTX 3070 Ti Local inference, note: VA space exhaustion issue with driver 595+
    Storage /mnt/jellyfin-media NFS mount from .25 for media
    Storage OB1 SSE :8002 LAN-accessible MCP transport (API key)
    Storage OB1 FastMCP :8003 Streamable HTTP MCP transport (stateless)

    .25 — OptiPlex 7060 (Jump Box)

    Layer Component Details
    Hardware Dell OptiPlex 7060 6C/23 GB
    OS CachyOS (Arch) Jump box, CLI access
    Service xrdp :3389 Remote desktop
    Service lynx CLI web browser
    Container Metube :8081 YouTube/URL media downloader
    Inactive KVM/libvirt Former VM host — VMs migrated to .113

    .39 — NETGEAR XS708Ev2 (Network Backbone)

    Layer Component Details
    Hardware 10GbE managed switch Connects .113, .16, .112, .25

    .111 — TrueNAS VM

    Layer Component Details
    OS TrueNAS SCALE Storage appliance
    vCPU 12 vCPU Allocated from .113
    RAM 32 GB Allocated from .113
    Storage HBA passthrough 5 PCI devices for direct disk access
    Service SMB shares LAN file sharing
    Service NFS exports VM file sharing
    Service DNS (secondary) Fallback DNS for .120

    .112 — Newsata File Server

    Layer Component Details
    OS Debian 12 Base OS
    Service NFS Exports /mnt/onetbssd/vmz (VM disk images)
    Service Samba SMB file sharing
    Service nginx Web server
    Service Pi-hole (secondary) DNS ad-blocking
    Storage /mnt/onetbssd/vmz/ VM images: hermes-112.qcow2 (60G), worker4-112.raw (50G)

    .113 — HP Z840 Hypervisor

    Layer Component Details
    Hardware HP Z840 2x Xeon E5-2680 v4 (28C/56T), 128 GB RAM, GTX 1660 Ti
    OS Garuda Linux (Arch) Kernel 7.x-zen
    Hypervisor libvirtd KVM/libvirt — manages 3 VMs
    Boot order vm-boot.service Systemd oneshot: TrueNAS → 60s → Hermes → 30s → Worker4
    VM TrueNAS (.111) 12 vCPU, 32G RAM
    VM Hermes (.176) 16 vCPU, 31G RAM, Ubuntu 24.04
    VM Worker4 (.120) 2 vCPU, 31G RAM, Garuda Linux
    Network br-public bridge VM network (bridged to vnet interfaces)
    Network br-private bridge Private non-routed storage network
    Mount /mnt/vmz NFS mount to .112 for VM disks
    GPU GTX 1660 Ti Host GPU, not passed through

    .120 — Worker4 VM (Ask)

    Layer Component Details
    OS Garuda Linux (Arch) Rolling release
    vCPU 2 vCPU Allocated from .113
    RAM 31 GB Allocated from .113
    Service AnythingLLM :8080 RAG chat interface (ask.atkatana.com)
    Service Resume QA server Career Q&A backed by OB1
    DNS .171 (Pi-hole primary) + .111 (secondary)

    .154 — Windows Ollama Host (Primary GPU)

    Layer Component Details
    OS Windows 11 Base OS
    GPU AMD RX 6900 XT 16 GB VRAM
    Service Ollama :11434 LAN-accessible LLM inference
    Model qwen3.5:9b Primary Hermes chat model (~5 GB)
    Model gemma4:e4b Secondary chat model (~3 GB)
    Model qwen3:8b Tertiary chat (~4.5 GB)
    Model deepseek-r1:8b Reasoning / analysis (~4.5 GB)
    Model nomic-embed-text OB1 embeddings (~0.5 GB)
    Note DirectML GPU acceleration via DirectX 12 ML, not ROCm

    .171 — Raspberry Pi (Pi-hole)

    Layer Component Details
    Hardware Raspberry Pi Physical device
    OS Raspberry Pi OS
    Service Pi-hole :53 DNS sinkhole (tracking/ad blocking)
    Service unbound Recursive DNS resolver upstream of Pi-hole

    .176 — Hermes VM (Worker5)

    Layer Component Details
    OS Ubuntu 24.04 Base OS
    vCPU 16 vCPU Allocated from .113
    RAM 31 GB Allocated from .113
    Web nginx :8080 Serves atkatana.com + blog, proxies ask → .120
    Tunnel cloudflared CloudFlare tunnel to atkatana.com
    Agent Hermes (gateway :9090) Python agent, MCP-capable, routes models to .154
    Browser headless Chrome Hermes browser automation (LinkedIn scraping)
    Storage /mnt/vmz/hermes-112.qcow2 60G VM disk on .112 NFS
    DNS systemd-resolved stub Name resolution

    External Dependencies

    Component Purpose Cost
    CloudFlare (CDN + DNS) Terminates SSL, routes atkatana.com → tunnel Free tier
    atkatana.com domain Primary website ~$10/yr
    ask.atkatana.com domain RAG Q&A subdomain CNAME to above
    Router (.1) DHCP, upstream DNS, NAT ISP-provided
    Internet connection Broadband (~300 Mbps cable) Monthly

    Network Summary

    
    10Gb Backbone:  .39 (XS708Ev2) ── .113 (Z840) ── .16 (Z640) ── .112 (Newsata)
                         │
    1Gb LAN:         .111 (TrueNAS) ── .171 (Pi-hole) ── .25 (OptiPlex)
                         │
                      .154 (Windows) ── Router ── Internet
            

    MCP Server Endpoints (for AI Agents)

    Server Type Endpoint Auth
    open-brain (local) stdio ~/OB1/server/... None (local)
    open-brain (SSE) SSE .16:8002 X-API-Key
    open-brain (FastMCP) Streamable HTTP .16:8003/mcp None (LAN)
    homelable HTTP .16:8001/mcp/ X-API-Key
    Ollama HTTP .154:11434 (or .16:11434) None (LAN)

    *Companion to "AI Lost My Files" — see that article for the human side of this story.*

    Built on a home lab, powered by local models, and owned by Andrew Katana.

    Connect on LinkedIn →