We Beat Mem0 and Zep on Their Own Benchmarks — at 13× Fewer Tokens

If you've followed the AI-memory space, you've seen the benchmark wars. Mem0 publishes numbers showing it beats everyone. Zep publishes a rebuttal showing Mem0's numbers are inflated. Everyone grades their own homework with an LLM judge tuned to make their answers look good. We didn't want to add another self-serving number to the pile — so when we benchmarked MemMesh against Mem0 and Zep on LOCOMO, the standard long-term-conversational-memory benchmark, we did the one thing that makes a claim checkable: we scored our engine under each competitor's own judge.

Not a lenient judge we invented. Mem0's exact partial-credit rule. Zep's exact "same-topic" rule. Applied to our answers. If we were going to claim a win, it had to be on their terms.

The results

Judged by Mem0's own partial-credit rule:

Category      MemMesh    Mem0 (published)
temporal        78            55
single-hop      78            67
multi-hop       75            51
open-domain     74            73
-----------------------------------------
Overall        76.6          66.9

Judged by Zep's own "same-topic" rule:

Category      MemMesh    Zep (published)
temporal        75            49
single-hop      75            62
multi-hop       52            41
open-domain     70            77
-----------------------------------------
Overall        70.2          66.0

We lead decisively on temporal, single-hop, and multi-hop reasoning against both. We're at parity with Mem0 on open-domain. Zep edges us on open-domain — that's genuinely their strength, built on per-entity summaries, and we're honest that we're still closing it. A clean sweep would be less believable than the truth.

The number that actually matters: tokens

Here's the stat we care about most, and it needs no calibration or judge at all: 522 tokens per query. Mem0 uses roughly 7,000. That's 13× less context fed to the model, for the same questions and better answers.

Most memory systems buy accuracy by flooding the context window. Mem0's multi-hop advantage comes largely from stuffing up to 200 memories into every prompt. That works — and it's a hidden tax on every single query your agent makes.

We built the opposite: precise retrieval, tiny context, frontier-quality answers. The engine surfaces the right few facts instead of dumping everything, so the model reasons over a clean, small context. Accuracy and cost, not one traded for the other. At scale, memory-by-flooding is one of the biggest, least-examined costs in production AI — and it's the one we designed out.

We grade ourselves harder than the benchmarks do

The most important thing to understand about the numbers above: they use the competitors' lenient judges. Both Mem0 and Zep grade with rules like "mark it correct as long as it references the same topic." A vague, verbose, or partially-right answer passes. Our internal bar is stricter — exact-match, not same-topic — and on that harder standard our overall score is lower, on purpose. It's the standard that reflects what a customer actually experiences when they ask their AI a question.

The reason we ran the lenient judges at all is fairness: to compare against Mem0 and Zep's published numbers, you have to grade everyone the same way. On a level field — their own judges — we win. Graded by our strict bar, the gap only widens, because a lenient judge inflates the lower scorer more.

Reproduce it yourself

We're not asking you to take our word for it. We publish our exact per-question predictions and a runnable harness that re-scores them under all three judges — Mem0's, Zep's, and our strict internal one. Grade our answers however you like.

pip install openai
export OPENAI_API_KEY=sk-...
python verify.py            # scores our predictions under all three judges
python verify.py --judge mem0

The full methodology, the exact judge prompts, and the per-question results live at github.com/ThinkfleetAI/memmesh-benchmarks. Honest caveats: these predictions come from MemMesh's production extraction path (a further-tuned extractor is in progress, so treat this as a floor); LOCOMO, 10 conversations, answerable categories; answers generated by gpt-4o-mini, judged by gpt-4o; competitor figures are their published LOCOMO numbers.

Memory is the difference between an AI that forgets you every session and one that actually knows you. But memory you can't afford to run at scale isn't a feature — it's a demo. Our bet is that the winning memory engine is the one that's accurate, cheap, and fast all at once — and that you can own and self-host, not rent by the token. Beating Mem0 and Zep on their own benchmarks, at 13× fewer tokens, is the first proof point.