The problem

Why existing benchmarks are measuring the wrong thing

LoCoMo, LongMemEval, and every benchmark built on top of them measure answer quality through a generative judge. This design cannot distinguish a memory system that retrieves one correct belief from one that retrieves its entire store.

✗

Answer-quality benchmarks

Retrieve everything. Route through a model. Score on F1. A system with 0.05 retrieval precision looks "competitive" because the downstream model compensates for the noise.

-- retrieve everything

SELECT * FROM beliefs

-- hope the model sorts it out

-- 18 results injected

-- precision: 0.05

✓

PrecisionMemBench

Assert against retrieved beliefs directly. No generative model in the evaluation loop. mustExclude and shouldOnlyInclude make noise a hard failure.

-- query at the source

SELECT * FROM beliefs

WHERE scope = 'project:api'

AND alias_match('redis')

-- 1 result. precision: 1.0

Strip the generative model out and route retrieved context to a classifier, rules engine, or structured pipeline: current retrieval implementations fail immediately. The generative model was never a neutral consumer, it was load-bearing infrastructure compensating for retrieval noise.

Results

Leaderboard

89 cases across 13 configurations. Active retrieval passes measure whether query-dependent retrieval returned the complete allowable belief set without prohibited or unrelated beliefs.

System	Active passes	Total passes	Mean precision	Mean recall	Retrieval p50	Ingestion total
tenure this	43/43	77/77	1.00	1.00	9.77ms	1.00s
open-knowledge-format	18/43	36/77	0.47	0.91	3534.53ms	0.00s
supermemory	4/43	21/77	0.22	0.71	69.41ms	5.20s
agentmemory	0/43	7/77	0.17	0.97	82.28ms	1.10s
yourmemory	0/43	21/77	0.17	0.88	313.39ms	16.40s
atomicmemory	0/43	9/77	0.15	0.95	71.01ms	658.90s
gbrain	5/43	34/77	0.14	0.17	543.84ms	28.60s
zep	0/43	9/77	0.09	0.95	124.36ms	897.00s
vectormxbai-embed-large	0/43	11/77	0.09	1.00	71.87ms	—
hindsight	0/43	9/77	0.06	1.00	589.86ms	173.30s
mem0	0/43	9/77	0.06	0.99	64.94ms	111.30s
cognee	0/43	11/77	0.05	0.92	2891.04ms	—
a-mem	0/43	9/77	0.06	0.99	13.80ms	178.80s

Active retrieval passes

Cases requiring query-dependent retrieval in which every required belief was returned, every prohibited belief was excluded, and the returned set stayed within the complete allowable set.

Structural passes

Cases asserting scope isolation, supersession exclusion, or type routing without a precision assertion, and the structural property holds.

Trivially empty passes

Cases where the expected result set is empty by design (empty query, maxBeliefs: 0). Any system returning an empty set passes by construction. retrievalPrecision is null for these cases.

Provider	Active retrieval	Structural	Trivially empty
tenure	43	25	9
open-knowledge-format	18	13	5
gbrain	5	20	9
supermemory	4	14	3
a-mem	0	6	3
agentmemory	0	5	2
atomicmemory	0	6	3
cognee	0	7	4
hindsight	0	6	3
mem0	0	6	3
vector	0	8	3
yourmemory	0	15	6
zep	0	6	3

Comparison wrappers adapt each provider to the common add, search, and reset interface. When a provider does not expose stable belief IDs, the harness maps returned content back to seeded IDs for scoring. Wrappers do not alter provider ranking, thresholds, candidate selection, or retrieval behavior. Results reflect pinned, reproducible artifacts. Reproduce locally →

Results by category

Where the failures happen

89 cases across 14 categories. Alias resolution and session-level noise isolation are where the precision gap is most stark.

Category	Cases	Tenure	Vector	Mem0	Zep	Hindsight	Cognee
Alias resolution	23	23/23	0/23	0/23	0/23	0/23	0/23
Session-level noise isolation	12	12/12	0/12	0/12	0/12	0/12	1/12
Scope disambiguation	12	12/12	1/12	0/12	0/12	0/12	0/12
Fuzzy matching & prefix guards	7	7/7	0/7	0/7	0/7	0/7	0/7
Type routing / open questions	6	6/6	0/6	0/6	0/6	0/6	1/6
Design boundary cases	6	6/6	2/6	2/6	2/6	2/6	3/6
Budget eviction & capacity	5	5/5	3/5	3/5	3/5	3/5	3/5
Persona prelude content	4	4/4	2/4	1/4	1/4	1/4	1/4
Relation expansion	4	4/4	0/4	0/4	0/4	0/4	0/4
Supersession chain exclusion	3	3/3	0/3	0/3	0/3	0/3	0/3
Ranking stability	3	3/3	1/3	1/3	1/3	1/3	1/3
Counter-signal retrieval	2	2/2	0/2	0/2	0/2	0/2	0/2
Cross-user isolation	1	1/1	1/1	1/1	1/1	1/1	1/1
Cold start behavior	1	1/1	1/1	1/1	1/1	1/1	1/1
Total	89	89/89	11/89	9/89	9/89	9/89	12/89

Open Knowledge Format run

A format is not the same thing as a retrieval layer

The Open Knowledge Format run is not a test of markdown as a storage format. It evaluates the runtime access pattern implied when an OKF-style bundle is placed into an AI tool today: files are available, the model may inspect them, and PrecisionMemBench scores the belief IDs corresponding to the files the model actually retrieves.

OKF is the strongest non-Tenure run on active retrieval with 18/43 active passes, but it pays for that access pattern in latency: 3,534.53ms retrieval p50 and 3,349.45ms session p50. The result supports the distinction this benchmark is meant to expose: storage format, runtime access, and precision-controlled retrieval are separate layers.

Cognee run

Less context is not the same thing as correct context

Cognee's graph-memory framing directly tests the claim that a memory layer can retrieve only what is needed for a given prompt. On PrecisionMemBench, Cognee reached high recall but low precision: 0.0548 mean precision, 0.9186 mean recall, 11/77 non-session passes, and 0 active retrieval passes.

This is the pattern PrecisionMemBench is designed to expose. The right belief is often somewhere in the retrieved set, but the model is also handed unrelated, stale, adjacent, or cross-scope state. Context stuffing may be cooked, but reducing the token pile does not prove the remaining context is the right context.

Embedding model invariance

A bigger model doesn't fix this

The natural objection: use a more capable embedding model. We tested across a 20× range in scale. Precision is identical. The fix is not a better ruler, it is a different measurement instrument.

Model	Dimensions	Precision	Passes	Mean latency	p95 latency
nomic-embed-text	768	0.09	11/77	43ms	85ms
mxbai-embed-large	1024	0.09	11/77	96ms	257ms
qwen3-8b	4096	0.09	11/77	1131ms	2605ms

At roughly 20× the mean latency of nomic-embed-text and over 1,100ms mean per query, the 8-billion parameter qwen3 model produces identical precision. All 11 passes in every configuration are trivially empty or budget-forced cases. Active retrieval passes across all three: 0.

Multi-turn noise isolation

After 8 off-topic turns, can the system get the right context back?

The session eval tests whether off-topic drift turns contaminate later retrieval, whether latency degrades under session load, and whether beliefs updated mid-session surface immediately. The drift score is the fraction of retrieved non-pinned beliefs that came from drift-turn topics. A drift score of 0 is perfect isolation.

Provider	Turns passed	Pass rate	Mean drift	Noise isolation	Mean precision	Session p50
tenure	12/12	1.00	0.0000	1.00	1.0000	47.79ms
open-knowledge-format	2/12	0.17	0.2153	0.17	0.5694	3349.45ms
yourmemory	1/12	0.08	0.7365	0.08	0.1965	430.49ms
supermemory	1/12	0.08	0.7493	0.08	0.1825	172.32ms
cognee	1/12	0.08	0.8459	0.08	0.0772	4222.62ms
gbrain	1/12	0.08	0.0000empty result	0.08	—	535.61ms
agentmemory	0/12	0.00	0.8087	0.00	0.1913	98.49ms
atomicmemory	0/12	0.00	0.8449	0.00	0.1551	355.08ms
zep	0/12	0.00	0.8888	0.00	0.1112	418.13ms
vector	0/12	0.00	0.9142	0.00	0.0858	256.75ms
a-mem	0/12	0.00	0.9259	0.00	0.0741	25.66ms
hindsight	0/12	0.00	0.9285	0.00	0.0715	1880.60ms
mem0	0/12	0.00	0.9398	0.00	0.0602	377.93ms

gbrain returned no results for these session cases. Its drift score of 0.0 is recorded by construction because no returned beliefs could originate from drift topics. The correct belief also failed to surface, so this is an empty-result failure, not a genuine isolation pass. Cognee passed 1/12 session turns with 0.8459 mean drift and 4,222.62ms session p50.

Methodology

How it works

Fixed seed corpus

35 beliefs spanning two domain scopes, a three-hop supersession chain, and a secondary-user fixture for isolation validation. All comparison systems are pre-ingested and confirmed available before any retrieval case runs.

Hard assertions, not judge scores

Each case carries mustExclude and shouldOnlyInclude constraints evaluated directly against retrieved belief IDs. No generative model scores the output. A pass requires all asserted tiers to be simultaneously satisfied.

Output normalization without reranking

When a provider does not expose stable belief IDs, the harness maps returned content to the corresponding seeded IDs. This normalization supports common scoring without changing provider ranking, thresholds, candidate selection, or retrieval behavior.

Pinned Docker digests

Reference systems are run from pinned, reproducible artifacts. Wrappers adapt each provider to the benchmark API contract without changing the seed corpus, precomputing answers, or injecting hidden ranking thresholds.

Case taxonomy

Alias resolution (23): Whether shorthand, abbreviations, and natural variations resolve to the intended belief without unrelated results.
Scope disambiguation (12): Whether identical names remain isolated across projects and domains.
Session-level noise isolation (12): Whether returning to an earlier topic excludes memories introduced during intervening topic drift.
Fuzzy matching and prefix guards (7): Whether misspellings and transpositions resolve correctly without admitting similarly spelled but unrelated terms.
Type routing and open questions (6): Whether open questions remain distinct from established state and leave the active route when resolved.
Design boundary cases (6): Whether empty, ambiguous, or unsupported queries produce predictable results without low-confidence retrieval.
Budget eviction and capacity (5): Whether limited context capacity is allocated to required state under competing retrieval demands.
Relation expansion (4): Whether relationship queries return eligible associated state without unrelated or out-of-scope participants.
Persona prelude content (4): Whether durable preferences remain available across topics without consuming query-dependent retrieval capacity.
Supersession chain exclusion (3): Whether replaced state stops influencing current behavior and outdated terminology resolves to active rather than obsolete guidance.
Ranking stability (3): Whether specific queries rank the intended belief ahead of weaker matches.
Counter-signal retrieval (2): Whether rejected or replaced terminology surfaces the current governing decision.
Cross-user isolation (1): Whether one user's stored information remains inaccessible to another.
Cold-start behavior (1): Whether a user with no stored state receives a clean empty result.

Research

Published on arXiv

arXiv:2605.11325 [cs.IR] · May 2026 Full paper →

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

Every major benchmark for LLM memory systems measures whether a model answered correctly, not whether the memory system retrieved correctly. We demonstrate that common memory baselines can achieve high recall while still returning substantial retrieval noise on cases referencing their own extractions. We present PrecisionMemBench, an 89-case benchmark measuring retrieval precision independently of generative models, and Tenure, a local-first structured belief store achieving 77/77 non-session passes, 12/12 session passes, mean precision 1.0, and sub-15ms single-turn retrieval latency.

BibTeX

@article{flynt2026tenure,
  title={Structured Belief State and the First
         Precision-Aware Benchmark for LLM
         Memory Retrieval},
  author={Flynt, Jeffrey},
  journal={arXiv preprint arXiv:2605.11325},
  year={2026}
}

The benchmark that measures
what memory tools actually do.

Why existing benchmarks are measuring the wrong thing

Answer-quality benchmarks

PrecisionMemBench

Leaderboard

Where the failures happen

A format is not the same thing as a retrieval layer

Less context is not the same thing as correct context

A bigger model doesn't fix this

After 8 off-topic turns, can the system get the right context back?

How it works

Fixed seed corpus

Hard assertions, not judge scores

Output normalization without reranking

Pinned Docker digests

Case taxonomy

Published on arXiv

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

Run it yourself

The benchmark that measureswhat memory tools actually do.

Why existing benchmarks are measuring the wrong thing

Answer-quality benchmarks

PrecisionMemBench

Leaderboard

Where the failures happen

A format is not the same thing as a retrieval layer

Less context is not the same thing as correct context

A bigger model doesn't fix this

After 8 off-topic turns, can the system get the right context back?

How it works

Fixed seed corpus

Hard assertions, not judge scores

Output normalization without reranking

Pinned Docker digests

Case taxonomy

Published on arXiv

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

Run it yourself

The benchmark that measures
what memory tools actually do.