Building a Persistent Knowledge Base RAG System with FastAPI, llama.cpp, Chroma, and Open WebUI

Have you ever wanted to chat with your own PDF collection – textbooks, research papers, internal documentation – using a local LLM, while keeping your data completely private? This is exactly what I built. In this article, I’ll walk you through a complete, production‑ready setup that: Ingests a folder of PDFs into a vector database (Chroma) Serves an OpenAI‑compatible RAG API using FastAPI Uses llama.cpp as the local LLM backend (any GGUF model works) Connects seamlessly to Open WebUI for a beau