Supermemory has a stronger architecture than raw vector memory systems, but its default PrecisionMemBench run still shows the core failure mode: broad recall with low precision. Mean retrieval precision is 0.2183, with 4/43 active retrieval passes. Tenure reaches 1.00 precision with deterministic scoped state.
Supermemory is architecturally more sophisticated than simple vector-memory systems. Its memory graph, local deployment path, and long-horizon positioning are real attempts to move beyond raw cosine recall.
But PrecisionMemBench measures what matters at retrieval time: whether the memory layer returns the right belief without flooding the model with nearby but wrong state. In the default no-threshold run, Supermemory reaches 0.2183 mean precision, 0.7065 mean recall, 21/77 total passes, and 4 active retrieval passes.
Supermemory's default profile is high recall, low precision. It often finds related material, but it returns too many irrelevant or stale beliefs alongside it. That means the model still has to sort signal from noise inside the prompt. Tenure's structural approach uses scoped state and deterministic retrieval so irrelevant beliefs are excluded before they reach the model.
The default comparison removes wrapper-level retrieval threshold tuning. A previous Supermemory wrapper used threshold: 0.69, which materially changed the benchmark result: 0.5321 mean precision, 0.7492 mean recall, 44/77 total passes, and 18 active retrieval passes.
That thresholded result is useful as an optimization data point, but it should not be treated as the out-of-box provider comparison unless the threshold is Supermemory's documented default behavior. PrecisionMemBench should measure what a normal user gets from the provider path, not what a benchmark wrapper can tune after observing failures.
In session mode, Supermemory passes 1/12 turns, records 0 active retrieval passes, and reaches only 0.1825 mean precision. Recall remains high at 0.9091, which shows the issue is not forgetting. The issue is over-retrieval.
Across drift turns, Supermemory repeatedly pulls in broad topical neighbors: Kubernetes, hosting, Vitest, Redis, auth questions, dead-letter queues, superseded linting beliefs, SQLAlchemy superseded beliefs, and other unrelated state. On re-entry, where the session should return to the earlier Redis topic, the system misses the expected Redis belief and surfaces unrelated/noise beliefs instead.
For agents, remembering too broadly is a production risk. A memory layer can have high recall and still degrade answers if it cannot isolate the exact state needed for the current turn. The model receives more context, but less certainty.
In the latest default run, Supermemory's p50 retrieval latency is 69.41ms and p95 latency is 116.45ms. Tenure's p50 retrieval latency is 9.77ms. In session mode, Supermemory's p50 turn latency is 172.32ms, compared with Tenure's 47.79ms.
Supermemory has both cloud-hosted and local/self-hosted deployment paths. The important distinction is not whether a process runs locally; it is whether the full memory pipeline stays local. In observed local traces, memory creation is performed through an LLM agent loop: source chunks are sent to the configured inference backend, and the model decides which memories to search, create, or update.
If that backend is local, the extraction path can remain local. If it points to Azure, OpenAI, Anthropic, or another hosted model provider, source chunks leave the machine even though the memory service itself is self-hosted. Observed traces also show outbound PostHog telemetry events for memory operations in a self-hosted run. The observed telemetry payloads were operational metadata rather than raw memory contents, but they still create an outbound network dependency operators should know about.
For production teams, "local" has to cover more than the database, graph engine, or embeddings. Memory systems have at least three separate paths: storage, retrieval, and extraction. If extraction is model-authored, raw chunks may be exposed to whichever LLM endpoint is configured. If analytics are enabled, usage metadata may leave the deployment even when memory contents do not.
Tenure's position is simpler: memory is governed state in the request path. It does not require an account, cloud memory API, hosted memory service, telemetry stream, or retrieval-time model loop to retrieve the relevant state for a turn.
| Property | Tenure | Supermemory |
|---|---|---|
| Mean retrieval precision | 1.00 | 0.2183 |
| Mean retrieval recall | 1.00 | 0.7065 |
| Active retrieval passes | 43/43 | 4/43 |
| Total passes (77 non-session) | 77/77 | 21/77 |
| Non-session pass rate | 1.00 | 0.2727 |
| Retrieval latency (p50) | 9.77ms | 69.41ms |
| Retrieval latency (p95) | Low double-digit ms | 116.45ms |
| Session turns passed | 12/12 | 1/12 |
| Session active retrieval passes | 12/12 | 0/12 |
| Session mean precision | 1.00 | 0.1825 |
| Session mean recall | 1.00 | 0.9091 |
| Session latency (p50) | 47.79ms | 172.32ms |
| Observed max drift | 0.00 | 1.00 |
| Local/self-hosted option | Yes | Yes |
| Account required | No | Depends on deployment |
| Memory extraction path | Deterministic state layer | LLM-authored memory creation |
| Source chunks sent to LLM | Not required for retrieval | Yes, to configured provider |
| Observed outbound telemetry | No product telemetry required | PostHog memory-operation metadata observed |
| Scope isolation | Hard filter | Partial / graph-mediated |
| Per-turn injection audit | Yes | No |
| Pricing | Free (MIT) | $19-$399+/mo for hosted plans |
| License | MIT | Proprietary / source-available components vary by deployment |
Supermemory numbers use the default no-threshold provider run. A separate wrapper-tuned run with threshold: 0.69 reached 0.5321 mean precision and 18 active retrieval passes, but is not used as the primary out-of-box comparison. Full methodology: arXiv:2605.11325. Dataset: HuggingFace. Live leaderboard: HuggingFace Spaces.
No memory SaaS account. No hosted memory API. No retrieval-time model loop. Thirty seconds to install. Runs on your machine forever.