What Building RAG Systems Taught Me About Retrieval and Memory

I used to think RAG was one pattern: put documents in a vector database, retrieve the closest chunks, and let the LLM answer.

That is a useful starting point, but it is not the whole design. The retrieval layer has to match the kind of question people ask, how often the data changes, and what the team can actually operate.

The useful distinction for me is between semantic retrieval, relationship aware retrieval, and time aware memory for agents. These solve different problems. They are options to evaluate, rather than stages every application must pass through.

Why I started with the simple version

The first RAG system I needed was a product assistant. It had to answer questions from documents that were already available: help articles, product pages, and internal notes.

The first version was deliberately boring. I chunked the content, stored embeddings, retrieved a small set of candidates, and sent those candidates to the model with source references. That made it possible to ship, observe real questions, and learn where the answer quality actually failed.

That is still the advice I would give myself now: start with the smallest retrieval system that makes a useful answer possible.

Stage 1: Vector RAG for semantic discovery

Vector RAG represents text as embeddings, then finds chunks that are close to the user’s question in semantic space. pgvector can keep those vectors next to product data in PostgreSQL; a dedicated vector store can be a sensible option when its operating model fits better.

document → chunk + metadata → embedding → vector index
question → embedding → candidate chunks → rerank / cite → LLM answer

This is a strong fit when people ask questions in their own words: support search, internal documentation, product discovery, and FAQs where the language in the question rarely matches the heading in a document.

What it does not automatically solve is relationship assembly. Retrieving five good chunks does not guarantee that the model sees the right order, the relevant entities, or the missing link between two documents.

Stage 2: Hybrid search before a graph

“Vectorless RAG” is a broad label for retrieval without a vector search stage. It can mean lexical search, document tree navigation, or model guided selection. Here I focus on the lexical option: full text search, BM25, or another term based ranking method.

Lexical retrieval is useful when terminology matters: policy references, product SKUs, error codes, or model identifiers. Use a structured lookup for an exact identifier when possible. PostgreSQL's built in full text search can be combined with pgvector for hybrid retrieval; its standard ranking functions are distinct from BM25.

question → keyword candidates + semantic candidates → fuse / rerank → grounded answer

Hybrid search gives exact terminology and semantic recall a chance to correct one another. It also makes evaluation easier: I can see whether a failure came from indexing, retrieval, reranking, or the answer prompt.

Stage 3: LightRAG when the question spans concepts

LightRAG builds entity and relationship structure during indexing, then uses dual level retrieval for specific facts and broader concepts. Its incremental ingestion model is useful when the corpus changes regularly.

The important point is not that LightRAG is universally faster or cheaper. Those outcomes depend on the corpus, model, retrieval settings, graph store, and the evaluation set. The useful change is architectural: the system has a way to retrieve connected context rather than only neighbouring chunks.

documents → entities + relationships + chunks
question → local detail + global concept retrieval → connected context → answer

I reach for this kind of architecture when users ask comparison questions, cross document questions, or “how does X relate to Y?” questions often enough that a hybrid vector pipeline keeps missing context.

Stage 4: Neo4j GraphRAG for explicit domain relationships

Some products already have a domain graph: customers own orders, orders contain components, components have incidents, and people belong to teams. For that kind of data, modelling the relationships directly can be clearer than extracting them repeatedly from prose.

Neo4j’s GraphRAG package supports knowledge graph construction, vector retrieval, and graph oriented retrieval patterns such as Text2Cypher. It is useful when the relationship model is part of the product, not merely an implementation detail.

Stage 5: Graphiti for temporal agent memory

Graphiti is different again. It is an open source framework for temporal knowledge graphs: it can represent episodes, entities, facts, and the time windows in which those facts were valid. It supports pluggable graph backends including Neo4j, FalkorDB, and Amazon Neptune.

That makes it interesting for agent memory, audits, changing account context, and systems where “what was true then?” is a real part of the question. It should not be presented as a better default RAG stack. It is a more specialised tool with a larger data model and operational commitment.

The decision table I use now

What a production RAG system actually needs

The retrieval algorithm is only one layer. A useful production system also needs:

• A source of truth: object storage or a document system for originals; PostgreSQL or Supabase for metadata, users, permissions, and document versions.
• An ingestion path: durable jobs for parsing, chunking, embedding, and graph extraction. Do not do a large re index inside a user request.
• A retrieval contract: permission filters before context reaches the model, a bounded context budget, source links, and evaluation cases based on real user questions.
• Operations: metrics for latency, retrieval quality, queue depth, model cost, and failure rate; logs that make an answer traceable.

For a small self hosted deployment, that can mean containers on a Hetzner server managed by Coolify, PostgreSQL/Supabase, and a durable worker queue. For a team that needs managed infrastructure or AWS native controls, the compute and queue can move to AWS. The important design decision is not the vendor: it is whether the slow work is durable, observable, and recoverable.

What I would tell my earlier self

Do not begin with a graph because it sounds advanced. Do not dismiss keyword search because embeddings are popular. And do not call a system “production ready” until it can explain the source of an answer, handle a failed ingestion job, and tell you when its data is stale.

Start with the questions. Measure the failure mode. Add complexity only where it earns its place.

For a side by side account of these options, see the retrieval architecture comparison. Supabase's hybrid search guide gives a concrete PostgreSQL implementation of lexical and vector retrieval together.