Grounded — document Q&A with citations

01

Context

LLMs are fluent but ungrounded — they hallucinate, and they can't answer questions about your private documents. Retrieval-augmented generation is the standard fix, but most demos stop at “embed some text, stuff it in a prompt.” Grounded is the version built to survive contact with real use: correct citations, honest refusals, multi-turn conversations, deduplicated ingestion, and a clean provider abstraction — the parts that separate a toy from something you'd actually trust.

The goal was a production-shaped RAG system I could reason about end to end, from the pgvector index up to the streaming chat UI.

02

What I built

Upload your own documents, then chat with an AI that answers only from them — and cites the exact passage behind every claim.

• Ingestion pipeline

  Parse → chunk → embed → store. Per-format parsers (PDF via unpdf, DOCX via mammoth, TXT/MD), ~800-token chunks with 150-token overlap snapped to whitespace boundaries, and SHA-256 content-hashing for idempotent, deduplicated re-ingestion. Documents move through a visible pending → processing → ready-or-failed state machine, and chunk writes are atomic.

• Semantic retrieval

  Top-k nearest-neighbour search over vector(768) embeddings using pgvector's cosine operator, accelerated by an HNSW index. Scoped to a collection or a single document.

• Grounded generation with citations

  The model answers strictly from retrieved context and returns structured citations, each carrying a verbatim quote so the UI can highlight the exact supporting passage. If the context doesn't cover the question, it says so rather than guessing.

• Multi-turn chat

  A query-rewrite step turns follow-ups like “what about the second one?” into standalone questions for retrieval, while generation still sees the user's original wording. Answers stream to the UI over NDJSON.

• Collections & persisted conversations

  Documents are organised into corpora; chat threads are scoped to a collection and saved, with auto-generated titles.

• CLI tooling

  Ingest, search, and ask scripts (with --dry-run, --force, --k, and --doc flags) to exercise each stage of the pipeline in isolation.

03

Technical decisions

• Postgres + pgvector over a dedicated vector DB

  Keeps documents, chunks, embeddings, and app data in one transactional store. No second system to sync, and real joins and cascade deletes come for free.

• Cosine distance + HNSW

  Cosine compares embedding direction — the right measure for semantic similarity — and HNSW gives high-recall approximate search with no training step, unlike IVFFlat.

• Drizzle ORM, raw SQL only for KNN

  Type-safe schema and generated migrations everywhere, dropping to a sql template literal solely for the distance-ordered nearest-neighbour query, where raw SQL is genuinely clearer.

• Provider abstraction

  Embeddings and generation sit behind small EmbeddingProvider and LLMProvider interfaces; swapping Gemini for OpenAI is an additive two-step change — new file plus one case — with zero feature-code churn.

• Citations as structured data, not string-parsing

  The model returns markers plus verbatim quotes, so highlighting is exact and every citation provably points at a chunk that was actually retrieved.

• Honesty as an explicit contract

  The model commits to an answerable boolean, which structurally discourages the slide into plausible-but-ungrounded answers.

04

Outcome

A working, end-to-end RAG application that takes a raw document to a cited, grounded answer — covering the parts most demos skip: honest refusals, exact-passage citations, conversational follow-ups, safe re-ingestion, and streaming responses.

Just as valuable was what it forced me to understand: how embedding spaces and cosine similarity actually drive retrieval quality, why chunking strategy is a genuine engineering trade-off, and how to design an AI integration that isn't welded to a single vendor. The provider interface means the entire app is one adapter away from running on a different model.