vectorize-io/hindsight

Most agent memory systems are glorified conversation logs with a vector index bolted on. Hindsight takes a different position: the problem isn't retrieval speed, it's that agents never actually learn anything — they just look things up. The architecture reflects that bet in concrete ways.

The core abstraction is a memory bank, an isolated store for one user, agent, or project. When you call retain, an LLM extracts entities, relationships, and temporal data, then normalizes them into canonical forms before indexing. That's more expensive than a straight embed-and-store, but it's what makes the next step possible: background consolidation into observations, which are deduplicated, evidence-backed beliefs that get refined rather than overwritten as new facts arrive. A fact that contradicts an existing observation doesn't silently replace it — it updates the evidence count and potentially shifts the belief. That's a meaningful distinction from append-only memory stores.

Recall runs four strategies in parallel — semantic vector similarity, BM25 keyword matching, entity/temporal graph traversal, and time-range filtering — then merges results via reciprocal rank fusion and a cross-encoder reranker. The reflect operation goes further: it synthesizes across memories to form new connections rather than just retrieving existing ones, which is where use cases like "why has this outreach been failing?" become tractable.

Mental models are the most interesting piece. You define a standing question once — "what are this user's preferences?" — and Hindsight writes and continuously rewrites the answer as the bank accumulates evidence. Reading a mental model at agent startup is a plain database read, no LLM call required. That's a real architectural win for agents that need settled context fast.

The benchmark claim is bold: top performance on LongMemEval as of January 2026, with independent reproduction credited to Virginia Tech's Sanghani Center and The Washington Post. The live benchmark page publishes per-model accuracy, latency, and cost. Competitor scores are self-reported, which is a meaningful caveat the README does acknowledge.

Deployment options are thorough — Docker, pip, Helm, embedded Python with no server, and a managed cloud tier. The LLM wrapper integration is genuinely low-friction: two lines to wrap an existing OpenAI or Anthropic client, with automatic retain and recall on every call. The MCP endpoint ships by default on every server, one per bank, which means any MCP-aware agent gets retain/recall/reflect as tools without extra configuration.

The Memory Defense feature — scanning every retain for PII and secrets against 45 patterns before storage — is opt-in per bank, which is the right default for most deployments but worth knowing about for anything handling sensitive user data.

Where this gets complicated is cost. Every retain fires an LLM call for extraction and normalization. Background consolidation fires more. For high-volume pipelines, that adds up fast, and the README is honest that Hindsight may be overkill for simple automation workflows. The tradeoff is real: richer memory structure costs more to build.