Benchmarks
Every memory system looks good at 10K tokens.
A single conversation fits inside a model's context window — so retrieval barely matters and everyone clusters near the ceiling. The real test is a lifetime of context: hundreds of sessions, millions of tokens, questions that span months. That's where full-context breaks and thin memory layers thin out. It's the ground MemMesh is built on.
Specialized brains
A brain is what makes a small model score at all.
A brain is a domain-scoped memory a model retrieves from over MCP. On 34 SEC-finance questions graded against real 10-K filings, the numbers are stark — and reproducible.
| Answerer | Brain | Accuracy | Cost / q |
|---|---|---|---|
| Llama 3.1 8B (local) | — | 0.0% | $0.0007 |
| GPT-4o (frontier) | — | 0.0% | $0.0012 |
| Llama 3.1 8B (local) | 85.3% | $0.0007 | |
| GPT-4o (frontier) | 85.3% | $0.0018 |
Without the brain, both models score 0% — these are precise, recent filing figures no model holds on its own. So every correct answer is provably the brain's, and a local model with the brain matches a frontier one at a fraction of the cost. Same judge, dataset, and prompt across rows.
The collapse curve
Built for where the others fall off.
Conceptual — the shape of the thesis. Measured tiers land on this axis as the BEAM 128K→10M runs complete. Validated so far: 9/9 needle recall at 510K tokens, where raw context no longer fits.
Early results
Three numbers that already hold up.
From the first runs, LLM-judged. Where MemMesh separates from stuffing raw history into the model — on time, on cost, and on knowing when to stay quiet.
better at temporal reasoning
87% vs 46% for full-context on “what happened first?” — MemMesh grounds every memory in when it happened.
LOCOMO · LLM-judge
fewer tokens, same accuracy
Matched a full-history RAG baseline at 400 vs 8,539 tokens per answer on 128K-token haystacks — while running on partial memory.
BEAM-100K · LLM-judge
abstention — knows when not to answer
Returns an honest “not enough signal” instead of a confident guess. The RAG baseline: 33%. In regulated work, that gap is the product.
BEAM · abstention ability
Early results from 3-conversation slices — directional, not the final published figures. Full reproducible runs land in the scoreboard below.
What the benchmarks measure
Benchmarks validate memory. They don't test the moat.
LOCOMO, LongMemEval, and BEAM each probe a slice of what a memory engine has to do. Between them they cover recall, time, and scale. Not one of them tests whether a memory engine can predict what's next or surface a behavior nobody defined — the two things MemMesh does that store-and-retrieve memory can't.
| Capability | LOCOMO | LongMemEval | BEAM |
|---|---|---|---|
| Cross-session continuityrecall across many sessions | full | full | full |
| Temporal reasoningwhen things happened, in order | full | partial | full |
| Knowledge updatesupersede stale facts | none | partial | partial |
| Multi-hop recallchain facts across sessions | partial | full | partial |
| Retrieval at realistic budgetsholds past the context window | none | partial | full |
| Abstention / calibrationknow when not to answer | none | none | partial |
| Predictionsmoatforecast a declared target from history | none | none | none |
| Behaviorsmoatdiscover patterns no one defined | none | none | none |
● full ◐ partial ○ none · coverage reflects what each benchmark is designed to probe. The moat rows are all-none across the field: no public benchmark measures prediction or emergent behavior yet.
The scoreboard
Numbers land here — verified, or not at all.
Each score comes from one reproducible harness with a pinned manifest — dataset hash, model ids, prompt hashes. Cells read measuring until a full run completes. We publish what reproduces; nothing is invented to fill a cell.
| Benchmark | Scale | MemMesh | Full-context | Published field |
|---|---|---|---|---|
| LOCOMOLLM-judge accuracy | ~1 conversation | 75.6%early | 66.2% | 68.0% |
| LongMemEvalLLM-judge accuracy | ~115K tokens | measuring | measuring | measuring |
| BEAMper-ability, LLM-judge | 128K token haystacks | 50.4%early | measuring | 51.2% |
Reproducible manifests published alongside each result. Competitor and full-context reference points are run in the same loop, never cited second-hand as a win.
How these numbers are made
Measured, not marketed.
One harness, every system
The same retrieve → answer → judge loop runs MemMesh, full-context, and RAG. Differences in the score come from the memory engine, not the scaffolding around it.
LLM-judge, not F1
Semantic-correctness judging. Standard F1 puts full-context at ~50%, not the 90%+ some leaderboards self-report — so we don't measure that way.
Reproducible manifest
Every run pins the dataset hash, model ids, prompt hashes, and price table. A third party can re-run and land on the same numbers.
The scale ladder is the headline
BEAM 128K → 1M → 10M. The tier where full-context stops fitting is where a memory engine stops being a nicety and becomes the only option.
The benchmarks measure memory. MemMesh does more.
Recall is table stakes. Prediction, calibrated abstention, and emergent behavior are the moat — and they start with the same observations you'd feed any memory layer.