We Benchmarked Our Own Memory Honestly: Calibration Beats Accuracy

We ran MemMesh against a naive retrieval-augmented-generation baseline on BEAM-100K — a long-term-memory benchmark scored by an LLM rubric across ten distinct abilities. We're publishing the results as-is, including where we lose, because the honest story is more useful than a cherry-picked one. And the honest story is this: on raw answer quality it's a tie — but a tie hides the two things that actually matter for memory.

The two numbers that matter

Overall rubric score was a statistical wash: MemMesh 50.4% vs naive RAG 51.2%. If "which one answers more questions correctly" were the whole story, you'd call it even. It isn't the whole story.

                    MemMesh    naive RAG
Overall (rubric)     50.4%       51.2%      ~tie
Tokens / question      400       8,539      21× fewer
Abstention            83.3        33.3      2.5× better

MemMesh reached the same answer quality using one twenty-first of the tokens — ~95% fewer. Naive RAG buys its points by stuffing 8,539 tokens of retrieved context into every prompt; MemMesh uses 400. At scale, that's the difference between a viable product and a bill.

Calibration beats accuracy

The single biggest gap in the whole benchmark is abstention: 83.3 vs 33.3. Abstention measures whether the system knows when it doesn't know — whether it declines to answer instead of confidently making something up. Naive RAG fails this two times out of three. MemMesh passes it five times out of six.

A system that answers confidently when it shouldn't isn't more accurate — it's more dangerous. The number to trust is the one that tells you when not to trust it.

This is why we lead with calibration, not accuracy. For anything that touches a real decision — a customer record, a portfolio, a diagnosis — a memory that hallucinates with confidence is a liability no rubric score redeems. Abstention is the difference between a demo and something you'd deploy.

Where memory beats retrieval

The wins cluster exactly where a memory layer should beat flat retrieval — on knowledge that changes over time:

• Knowledge update: 66.7 vs 50.0 — revising a fact when it changes, instead of retrieving the stale one.
• Multi-session reasoning: 54.2 vs 45.8 — connecting facts across separate conversations.
• Contradiction resolution: 20.8 vs 16.7 — reconciling sources that disagree.

These are the abilities that separate a memory from a search index. Retrieval finds the most similar chunk; memory tracks what's true now.

Where naive RAG still wins — and what it costs

We lose on several abilities, and we're not going to hide them:

• Summarization: 17.2 vs 45.0
• Instruction following: 75.0 vs 100
• Information extraction: 38.9 vs 58.3
• Preference following: 91.7 vs 100

There's a pattern: these are the tasks where having the entire raw context in the prompt is an advantage — and naive RAG has it, at 8,539 tokens a shot. Ask it to summarize a transcript it's holding in full, and of course it does well. MemMesh distils rather than hoards, so it trades some ground on total-recall tasks for the 21× token saving everywhere else. That's the tradeoff stated plainly — and summarization and extraction are exactly the two we're working to close.

The full board

Ability                    MemMesh   naive RAG
abstention                   83.3      33.3   ↑
preference_following         91.7     100.0
instruction_following        75.0     100.0
knowledge_update             66.7      50.0   ↑
multi_session_reasoning      54.2      45.8   ↑
information_extraction       38.9      58.3
event_ordering               31.1      37.8
temporal_reasoning           25.0      25.0   =
contradiction_resolution     20.8      16.7   ↑
summarization                17.2      45.0
---------------------------------------------
Overall                      50.4%     51.2%
Tokens / question             400     8,539

Why publish a tie

Because the tie is the point. Two systems reached the same answer quality; one did it at 1/21st the cost and knew when to stay silent. In long-term memory, cost and calibration are the axes that decide whether something ships — not a fraction of a rubric point. A benchmark that only reports overall accuracy is measuring the least interesting thing.

Honest caveats: these are initial numbers against a single baseline (naive RAG) on BEAM-100K. A head-to-head against other memory systems means running them through the same harness — different vendors report on different benchmarks, scales, and judges, so cross-quoting numbers wouldn't be a fair comparison. We'll publish those runs when we have them, wins and losses alike.

Memory that's 21× cheaper and knows its own limits. That's the bet — and now it's measured.