Every existing benchmark measures whether the model answered correctly.
LoCoMo, LongMemEval, and every benchmark built on top of them measure answer quality through a generative judge. This design cannot distinguish a memory system that retrieves one correct belief from one that retrieves its entire store.
Retrieve everything. Route through a model. Score on F1. A system with 0.05 retrieval precision looks "competitive" because the downstream model compensates for the noise.
Assert against retrieved beliefs directly. No generative model in the evaluation loop. mustExclude and shouldOnlyInclude make noise a hard failure.
Strip the generative model out and route retrieved context to a classifier, rules engine, or structured pipeline: current retrieval implementations fail immediately. The generative model was never a neutral consumer, it was load-bearing infrastructure compensating for retrieval noise.
89 cases. 13 systems. One metric that matters: active retrieval passes, which are cases where a retrievalPrecision assertion was satisfied.
| System | Active passes | Total passes | Mean precision | Mean recall | Retrieval p50 | Ingestion total |
|---|---|---|---|---|---|---|
| tenure this | 43/43 | 77/77 | 1.00 | 1.00 | 9.77ms | 1.00s |
| open-knowledge-format | 18/18 | 36/77 | 0.47 | 0.91 | 3534.53ms | 0.00s |
| supermemory | 4/4 | 21/77 | 0.22 | 0.71 | 69.41ms | 5.20s |
| agentmemory | 0/0 | 7/77 | 0.17 | 0.97 | 82.28ms | 1.10s |
| yourmemory | 0/0 | 21/77 | 0.17 | 0.88 | 313.39ms | 16.40s |
| atomicmemory | 0/0 | 9/77 | 0.15 | 0.95 | 71.01ms | 658.90s |
| gbrain | 5/5 | 34/77 | 0.14 | 0.17 | 543.84ms | 28.60s |
| zep | 0/0 | 9/77 | 0.09 | 0.95 | 124.36ms | 897.00s |
| vectormxbai-embed-large | 0/0 | 11/77 | 0.09 | 1.00 | 71.87ms | — |
| hindsight | 0/0 | 9/77 | 0.06 | 1.00 | 589.86ms | 173.30s |
| mem0 | 0/0 | 9/77 | 0.06 | 0.99 | 64.94ms | 111.30s |
| cognee | 0/0 | 11/77 | 0.05 | 0.92 | 2891.04ms | — |
| a-mem | 0/0 | 9/77 | 0.06 | 0.99 | 13.80ms | 178.80s |
Cases with a retrievalPrecision assertion that was satisfied. The only pass type that demonstrates verified retrieval capability. A system cannot accumulate these by returning everything or nothing.
Cases asserting scope isolation, supersession exclusion, or type routing without a precision assertion, and the structural property holds.
Cases where the expected result set is empty by design (empty query, maxBeliefs: 0). Any system returning an empty set passes by construction. retrievalPrecision is null for these cases.
| Provider | Active retrieval | Structural | Trivially empty |
|---|---|---|---|
| tenure | 43 | 25 | 9 |
| open-knowledge-format | 18 | 13 | 5 |
| gbrain | 5 | 20 | 9 |
| supermemory | 4 | 14 | 3 |
| a-mem | 0 | 6 | 3 |
| agentmemory | 0 | 5 | 2 |
| atomicmemory | 0 | 6 | 3 |
| cognee | 0 | 7 | 4 |
| hindsight | 0 | 6 | 3 |
| mem0 | 0 | 6 | 3 |
| vector | 0 | 8 | 3 |
| yourmemory | 0 | 15 | 6 |
| zep | 0 | 6 | 3 |
89 cases across 14 categories. Alias resolution and session-level noise isolation are where the precision gap is most stark.
| Category | Cases | Tenure | Vector | Mem0 | Zep | Hindsight | Cognee |
|---|---|---|---|---|---|---|---|
| Alias resolution | 23 | 23/23 | 0/23 | 0/23 | 0/23 | 0/23 | 0/23 |
| Session-level noise isolation | 12 | 12/12 | 0/12 | 0/12 | 0/12 | 0/12 | 1/12 |
| Scope disambiguation | 12 | 10/12 | 1/12 | 0/12 | 0/12 | 0/12 | 0/12 |
| Fuzzy matching & prefix guards | 7 | 7/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 |
| Type routing / open questions | 6 | 6/6 | 0/6 | 0/6 | 0/6 | 0/6 | 1/6 |
| Design boundary cases | 6 | 6/6 | 2/6 | 2/6 | 2/6 | 2/6 | 3/6 |
| Budget eviction & capacity | 5 | 5/5 | 3/5 | 3/5 | 3/5 | 3/5 | 3/5 |
| Persona prelude content | 4 | 4/4 | 2/4 | 1/4 | 1/4 | 1/4 | 1/4 |
| Relation expansion | 4 | 4/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| Supersession chain exclusion | 3 | 3/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 |
| Ranking stability | 3 | 3/3 | 1/3 | 1/3 | 1/3 | 1/3 | 1/3 |
| Counter-signal retrieval | 2 | 2/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 |
| Cross-user isolation | 1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 |
| Cold start behavior | 1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 |
| Total | 89 | 89/89 | 11/89 | 9/89 | 9/89 | 8/89 | 12/89 |
The Open Knowledge Format run is not a test of markdown as a storage format. It evaluates the runtime access pattern implied when an OKF-style bundle is placed into an AI tool today: files are available, the model may inspect them, and PrecisionMemBench scores the belief IDs corresponding to the files the model actually retrieves.
OKF is the strongest non-Tenure run on active retrieval with 18/18 active passes, but it pays for that access pattern in latency: 3,534.53ms retrieval p50 and 3,349.45ms session p50. The result supports the distinction this benchmark is meant to expose: storage format, runtime access, and precision-controlled retrieval are separate layers.
Cognee's graph-memory framing directly tests the claim that a memory layer can retrieve only what is needed for a given prompt. On PrecisionMemBench, Cognee reached high recall but low precision: 0.0548 mean precision, 0.9186 mean recall, 11/77 non-session passes, and 0 active retrieval passes.
This is the pattern PrecisionMemBench is designed to expose. The right belief is often somewhere in the retrieved set, but the model is also handed unrelated, stale, adjacent, or cross-scope state. Context stuffing may be cooked, but reducing the token pile does not prove the remaining context is the right context.
The natural objection: use a more capable embedding model. We tested across a 20× range in scale. Precision is identical. The fix is not a better ruler, it is a different measurement instrument.
| Model | Dimensions | Precision | Passes | Mean latency | p95 latency |
|---|---|---|---|---|---|
| nomic-embed-text | 768 | 0.09 | 11/77 | 43ms | 85ms |
| mxbai-embed-large | 1024 | 0.09 | 11/77 | 96ms | 257ms |
| qwen3-8b | 4096 | 0.09 | 11/77 | 1131ms | 2605ms |
At roughly 20× the mean latency of nomic-embed-text and over 1,100ms mean per query, the 8-billion parameter qwen3 model produces identical precision. All 11 passes in every configuration are trivially empty or budget-forced cases. Active retrieval passes across all three: 0.
The session eval tests whether off-topic drift turns contaminate later retrieval, whether latency degrades under session load, and whether beliefs updated mid-session surface immediately. The drift score is the fraction of retrieved non-pinned beliefs that came from drift-turn topics. A drift score of 0 is perfect isolation.
| Provider | Turns passed | Pass rate | Mean drift | Noise isolation | Mean precision | Session p50 |
|---|---|---|---|---|---|---|
| tenure | 12/12 | 1.00 | 0.0000 | 1.00 | 1.0000 | 47.79ms |
| open-knowledge-format | 2/12 | 0.17 | 0.2153 | 0.17 | 0.5694 | 3349.45ms |
| yourmemory | 1/12 | 0.08 | 0.7365 | 0.08 | 0.1965 | 430.49ms |
| supermemory | 1/12 | 0.08 | 0.7493 | 0.08 | 0.1825 | 172.32ms |
| cognee | 1/12 | 0.08 | 0.8459 | 0.08 | 0.0772 | 4222.62ms |
| gbrain | 1/12 | 0.08 | 0.0000empty result | 0.08 | — | 535.61ms |
| agentmemory | 0/12 | 0.00 | 0.8087 | 0.00 | 0.1913 | 98.49ms |
| atomicmemory | 0/12 | 0.00 | 0.8449 | 0.00 | 0.1551 | 355.08ms |
| zep | 0/12 | 0.00 | 0.8888 | 0.00 | 0.1112 | 418.13ms |
| vector | 0/12 | 0.00 | 0.9142 | 0.00 | 0.0858 | 256.75ms |
| a-mem | 0/12 | 0.00 | 0.9259 | 0.00 | 0.0741 | 25.66ms |
| hindsight | 0/12 | 0.00 | 0.9285 | 0.00 | 0.0715 | 1880.60ms |
| mem0 | 0/12 | 0.00 | 0.9398 | 0.00 | 0.0602 | 377.93ms |
gbrain returned no results for these session cases. Its drift score of 0.0 is recorded by construction because no returned beliefs could originate from drift topics. The correct belief also failed to surface, so this is an empty-result failure, not a genuine isolation pass. Cognee passed 1/12 session turns with 0.8459 mean drift and 4,222.62ms session p50.
35 beliefs spanning two domain scopes, a three-hop supersession chain, and a secondary-user fixture for isolation validation. All comparison systems are pre-ingested and confirmed available before any retrieval case runs.
Each case carries mustExclude and shouldOnlyInclude constraints evaluated directly against retrieved belief IDs. No generative model scores the output. A pass requires all asserted tiers to be simultaneously satisfied.
Comparison systems receive pin-status filtering, open-question routing, and scope isolation applied from Tenure's metadata. They do not fail due to structural technicalities. The score reflects whether the provider's runtime retrieval actually returns the asserted belief IDs without extra noise.
Reference systems are run from pinned, reproducible artifacts. Wrappers adapt each provider to the benchmark API contract without changing the seed corpus, precomputing answers, or injecting hidden ranking thresholds.
Every major benchmark for LLM memory systems measures whether a model answered correctly, not whether the memory system retrieved correctly. We demonstrate that common memory baselines can achieve high recall while still returning substantial retrieval noise on cases referencing their own extractions. We present PrecisionMemBench, an 89-case benchmark measuring retrieval precision independently of generative models, and Tenure, a local-first structured belief store achieving 77/77 non-session passes, 12/12 session passes, mean precision 1.0, and sub-15ms single-turn retrieval latency.
@article{flynt2026tenure,
title={Structured Belief State and the First
Precision-Aware Benchmark for LLM
Memory Retrieval},
author={Flynt, Jeffrey},
journal={arXiv preprint arXiv:2605.11325},
year={2026}
}The benchmark is published as a reusable artifact. Run it against any memory implementation — including your own.