Built a local-first retrieval-augmented generation (RAG) service that indexes Markdown and PDF documents into Supabase pgvector and streams answers using local Ollama models (llama3, mistral).
Cloud LLM APIs (OpenAI, Anthropic) incur ongoing per-token costs, add network latency, and cannot be used for sensitive or air-gapped document collections where data privacy is a strict requirement.
Engineered a hybrid search pipeline that combines dense vector similarity (pgvector HNSW index) with sparse keyword matching (PostgreSQL tsvector). Streamed inference chunks over HTTP using Hono and Ollama's local API.
A reproducible local Q&A pipeline that runs without external API dependencies, delivers sub-second first-token latency on Apple Silicon, and keeps all document embeddings and queries strictly on-premises.