Benchmarks
Every memory system looks good at 10K tokens.
A single conversation fits inside a model's context window — so retrieval barely matters and everyone clusters near the ceiling. The real test is a lifetime of context: hundreds of sessions, millions of tokens, questions that span months. That's where full-context breaks and thin memory layers thin out. It's the ground MemMesh is built on.
The result
We beat Mem0 and Zep on LOCOMO — at 13× fewer tokens.
Every memory vendor grades LOCOMO with its own lenient judge, so we didn’t invent a friendlier one. We scored our engine under each competitor’s own judge, meaning Mem0’s partial-credit rule and Zep’s “same-topic” rule. We win overall against both, and lead on temporal, single-hop and multi-hop.
vs Mem0
88.7 vs 82.7Scored under both at a 50-memory retrieval budget
Most memory systems buy accuracy by flooding the context window — Mem0 stuffs up to 200 memories into every prompt. We feed the model 522 tokens, not ~7,000. Same questions, better answers, a fraction of the cost. This one needs no judge.
LOCOMO, 10 conversations, answerable categories · production extraction path · gpt-4o-mini answerer, gpt-4o judge · competitor figures are their published LOCOMO numbers. Zep edges us on open-domain — their strength, shown as-is.
Specialized brains
A brain is what makes a small model score at all.
A brain is a domain-scoped memory a model retrieves from over MCP. On 34 SEC-finance questions graded against real 10-K filings, the numbers are stark — and reproducible.
| Answerer | Brain | Accuracy | Cost / q |
|---|---|---|---|
| Llama 3.1 8B (local) | — | 0.0% | $0.0007 |
| GPT-4o (frontier) | — | 0.0% | $0.0012 |
| Llama 3.1 8B (local) | 96.5% | $0.0007 | |
| GPT-4o (frontier) | 96.5% | $0.0018 |
Without the brain, both models score 0% — these are precise, recent filing figures no model holds on its own. So every correct answer is provably the brain's, and a local model with the brain matches a frontier one at a fraction of the cost. Same judge, dataset, and prompt across rows.
The collapse curve
Built for where the others fall off.
Conceptual — the shape of the thesis. Measured tiers land on this axis as the BEAM 128K→10M runs complete. Validated so far: 9/9 needle recall at 510K tokens, where raw context no longer fits.
Early results
Three numbers that already hold up.
From the first runs, LLM-judged. Where MemMesh separates from stuffing raw history into the model — on time, on cost, and on knowing when to stay quiet.
better at temporal reasoning
87% vs 46% for full-context on “what happened first?” — MemMesh grounds every memory in when it happened.
LOCOMO · LLM-judge
fewer tokens, same accuracy
Matched a full-history RAG baseline at 400 vs 8,539 tokens per answer on 128K-token haystacks — while running on partial memory.
BEAM-100K · LLM-judge
abstention — knows when not to answer
Returns an honest “not enough signal” instead of a confident guess. The RAG baseline: 33%. In regulated work, that gap is the product.
BEAM · abstention ability
Early results from 3-conversation slices — directional, not the final published figures. Full reproducible runs land in the scoreboard below.
What the benchmarks measure
Benchmarks validate memory. They don't test the moat.
LOCOMO, LongMemEval, and BEAM each probe a slice of what a memory engine has to do. Between them they cover recall, time, and scale. Not one of them tests whether a memory engine can predict what's next or surface a behavior nobody defined — the two things MemMesh does that store-and-retrieve memory can't.
| Capability | LOCOMO | LongMemEval | BEAM |
|---|---|---|---|
| Cross-session continuityrecall across many sessions | full | full | full |
| Temporal reasoningwhen things happened, in order | full | partial | full |
| Knowledge updatesupersede stale facts | none | partial | partial |
| Multi-hop recallchain facts across sessions | partial | full | partial |
| Retrieval at realistic budgetsholds past the context window | none | partial | full |
| Abstention / calibrationknow when not to answer | none | none | partial |
| Predictionsmoatforecast a declared target from history | none | none | none |
| Behaviorsmoatdiscover patterns no one defined | none | none | none |
● full ◐ partial ○ none · coverage reflects what each benchmark is designed to probe. The moat rows are all-none across the field: no public benchmark measures prediction or emergent behavior yet.
The scoreboard
Numbers land here — verified, or not at all.
Each score comes from one reproducible harness with a pinned manifest — dataset hash, model ids, prompt hashes. Cells read measuring until a full run completes. We publish what reproduces; nothing is invented to fill a cell.
| Benchmark | Scale | MemMesh | Full-context | Published field |
|---|---|---|---|---|
| LOCOMOMem0’s harness, gpt-5 answerer + judge | 1,540 questions, 50-memory budget | 88.7% | measuring | 82.7% |
| LongMemEvalLLM-judge accuracy | ~115K tokens | measuring | measuring | measuring |
| BEAMper-ability, LLM-judge | 128K token haystacks | 50.4%early | measuring | 51.2% |
LOCOMO is measured at a 50-memory retrieval budget on both sides — Mem0’s own harness, their gpt-5 answerer and gpt-5 judge, the same 1,540 questions, adversarial excluded exactly as they exclude it. Mem0 also publish 91.56% at a 200-memory budget; we reach 88.7% on 50, a quarter of the retrieved context.
Swapping only the answerer for gpt-5-mini — same memory, same questions, same gpt-5 judge — scores 88.8%. That is an efficiency result, not a like-for-like accuracy one: Mem0’s 82.66% used the frontier model to answer.
Reproducible manifests published alongside each result. Competitor and full-context reference points are run in the same loop, never cited second-hand as a win.
How these numbers are made
Measured, not marketed.
One harness, every system
The same retrieve → answer → judge loop runs MemMesh, full-context, and RAG. Differences in the score come from the memory engine, not the scaffolding around it.
LLM-judge, not F1
Semantic-correctness judging. Standard F1 puts full-context at ~50%, not the 90%+ some leaderboards self-report — so we don't measure that way.
Reproducible manifest
Every run pins the dataset hash, model ids, prompt hashes, and price table. A third party can re-run and land on the same numbers.
The scale ladder is the headline
BEAM 128K → 1M → 10M. The tier where full-context stops fitting is where a memory engine stops being a nicety and becomes the only option.
The benchmarks measure memory. MemMesh does more.
Recall is table stakes. Prediction, calibrated abstention, and emergent behavior are the moat — and they start with the same observations you'd feed any memory layer.