May 2026
I run a 322-thought knowledge base called Open Brain (OB1) that stores my entire career — 12+ years of cloud architecture, P&L data, interview prep, and technical writing. When I query it, I want answers grounded in my own evidence, not hallucinations or irrelevant job listings.
I've been running a naive RAG pipeline (ChromaDB + Ollama embeddings + LLM synthesis) for months. It worked well enough for broad questions. But I kept hitting edge cases where it refused to answer questions I knew the data could handle. So I ran a systematic benchmark: 20 queries across two retrieval architectures, scored against ground truth.
The results explain why "just throw it in a vector DB" is not a strategy.
20 queries across five categories — simple factual, multi-hop relational, cross-document synthesis, temporal/staleness, and negative/edge cases. Each query was run against:
System A (Naive RAG): Vector search (Qdrant + `nomic-embed-text`) → top-6 results → LLM synthesis (gemma4:12b). The same pipeline most RAG tutorials recommend.
System B (Graph + Direct File Read): Knowledge graph traversal (NetworkX, 197 nodes, 2,385 edges spanning 13 OKF concept directories) → targeted file reads → extracted answer. No LLM synthesis pass — just structured retrieval.
| Score | System A (RAG) | System B (Graph) |
|---|---|---|
| Correct | 4 (20%) | 8 (40%) |
| Partial | 3 (15%) | 12 (60%) |
| Wrong | 13 (65%) | 0 (0%) |
The headline number: System A failed 65% of queries. System B failed zero.
Even the "partial" results for System B would likely become correct with a synthesis pass — the data was retrieved; it just wasn't formatted as a natural answer.
1. 55% refusal rate (11/20 queries). The most common response from System A was: "The provided context does not contain information regarding..." — followed by a list of retrieved sources that *did* contain the answer. The LLM was too conservative. It was trained to refuse rather than synthesize from partial matches, and that safety behavior destroyed recall.
Example query: *"What certifications does Andrew hold?"*
System A returned: "The provided context does not contain information regarding what certifications Andrew holds."
System B found the answer immediately: AWS Certified Solutions Architect Professional, Azure Solutions Architect Expert, Terraform IaC Certification — all listed in the Master CV file.
2. Job posting noise pollutes the vector space. My Qdrant instance stores both career evidence and scraped LinkedIn job postings. Semantic search doesn't distinguish between "a job posting that mentions AWS" and "my own AWS experience." The query *"What companies has Andrew worked at?"* returned four job listings as top sources. The actual career data was in the database but ranked below noise.
3. Limited context window. The pipeline feeds only top-6 sources to the LLM. If even one of those is a job posting, the signal-to-noise ratio degrades. For multi-hop queries like *"Compare my fit at Sentinel Security vs Apex Cyber,"* the relevant context lives across three separate documents. The RAG pipeline couldn't pull all three.
The OKF (Open Knowledge Format) bundle organizes career data as a typed, labeled knowledge graph — 197 concept nodes with 2,385 edges. Company profiles link to required skills link to interview questions link to blog posts. The graph doesn't just store text; it stores *relationships*.
For the query *"Which companies match my GCP skills?":*
The graph didn't need to find the exact text "GCP." It used the relationship structure.
Personal knowledge bases are different from enterprise document corpora in three ways that naive RAG handles poorly:
| Property | Enterprise Docs | Personal KB |
|---|---|---|
| Entity density | Low (one topic per doc) | High (many topics per entry) |
| Noise tolerance | Low (curated corpus) | High (scrapes + notes mixed) |
| Query specificity | Broad ("summarize Q3 report") | Narrow ("has my LinkedIn been updated recently?") |
Enterprise RAG systems assume clean, dedicated corpora. Personal knowledge bases are heterogeneous by nature — journal entries, scraped jobs, blog drafts, company research, all in one vector space. Semantic similarity can't distinguish "a job listing about cloud security" from "my own cloud security experience."
The most practical fix: use the knowledge graph as a routing layer before vector search. The question is classified by entity type (company, skill, blog, interview), the graph identifies the relevant subgraph, and vector search runs only within that subgraph. This eliminates the "pulling job listings for a career question" problem.
Question → Entity Classification → Graph Subgraph → Targeted Vector Search → LLM Synthesis
In my testing, this would have prevented 7 of the 11 refusal failures by ensuring the vector search space was already constrained to the right domain.
Separate career evidence from ephemeral data (scraped jobs, daily notes) into distinct collections with different retrieval priorities. Queries about "my experience" should never search the same vector space as "jobs I looked at last week."
Implementation: tag-based routing. A thought with tag `#career-evidence` gets priority over `#job-scrape` when the intent classifier detects a personal-experience question.
The LLM refusal problem needs a prompt fix: instead of "answer only from the provided context" (which triggers refusal on partial matches), use "synthesize from the provided context, and cite which sources support each statement." This shifts the behavior from rejection to attribution.
Current: "Answer from the provided context. If unsure, say so."
Better: "Synthesize from the provided context. Cite sources for each claim."
Questions like "What was my most recent full-time role?" and "Has my LinkedIn been updated recently?" require temporal reasoning that neither RAG nor graph handles well. Add a timestamp-indexed retrieval step that prioritizes the most recent facts for temporal queries.
In my benchmark, System A failed all four temporal queries (0/4 correct). System B found partial answers from direct file reads and coordination logs.
The underlying issue is that raw text + embeddings lacks structure. The Open Knowledge Format (OKF) addresses this by encoding every concept with typed frontmatter (`type`, `tags`, `timestamp`, `description`) and explicit relationships between concepts. The format is portable, diffable, and agent-friendly — designed to be consumed by both humans and automated retrieval systems.
Naive RAG works when your corpus is clean and your questions are broad. For personal knowledge bases — where the data is noisy, the questions are specific, and the cost of hallucination is high — structured retrieval isn't optional.
The 55% LLM refusal rate I measured is not a model problem. It's a retrieval problem. When your retrieval layer can't reliably surface the right context, the LLM defaults to "I don't know." Fix retrieval first, then tune synthesis.
The graph approach (System B) costs almost nothing — a NetworkX graph built from existing markdown files, zero API calls, zero embedding overhead. It changed retrieval from a probability to a traversal. That's the difference between hoping the answer is nearby and knowing where it lives.
*Full benchmark data (20 queries, per-query scoring, source timestamps) published as companion data alongside this post. Graph implementation: `scripts/okf_graph.py` in the ai-resume repository. OKF v0.1 spec available at `outputs/okf/career/`. All pipelines ran on local hardware (Ollama, Qdrant, internal-qa) — no cloud services involved.*
Built on a home lab, powered by local models, and owned by Andrew Katana.