0online
2026-10-31 · AGENT · SURVEY · LLM NEW! 4 · 8 min read

Ranking 12 agent memory systems by evidence - starting with what 'billion-context' is

TL;DR

  • The task that landed in my lap: "survey agent memory accumulation systems like Honcho and 'blilion contex', then rank them by papers, evidence, and reviews." I surveyed 12 systems. Survey date 2026-10-31; all star counts and creation dates measured directly via the GitHub API
  • blilion contex turned out to be billion-context (ranxianglei). It is not a persistent-memory system - it's an in-session context compression plugin. Zero papers, zero independent benchmarks
  • The conclusion up front: you cannot rank these systems by benchmark scores. So I graded them by the strength of the evidence behind their claims instead of raw accuracy
  • Overall: #1 Hindsight 6.6 · #2 Zep/Graphiti 6.35 · #3 Letta 6.3 · #4 Mem0 6.25 · Honcho, the one I run, at #7 with 5.4
  • The heaviest finding comes from an independent benchmark (MemVerdict): "no memory system clearly beats plain RAG, and they cost 11–31× more"

Where it started: what "blilion contex" is

The request came in garbled: "agent memory accumulation systems like homcho / blilion contex." A few search angles later, GitHub search settled it.

ranxianglei/billion-context. Created 2026-08-06, 3,466 stars (measured Oct 31), TypeScript. The pitch: "month-long single sessions on a 100K context window, 5× fewer tokens." It's an npm plugin that attaches to 25 coding agents - Claude Code, Codex, OpenCode, pi, Gemini CLI, hermes - with a dedicated pi fork, billion-context-pi (266 stars, MIT).

The important part is taxonomy. This is in-session compression (Active Context Pruning), not the cross-session persistent memory Honcho does. The roles are orthogonal - you can combine them, but pitting them against each other in one ranking is a category error. Evidence-wise it's also the weakest entry in this whole survey: zero arXiv papers, zero independent evaluations, and a license GitHub itself can't classify.

Method: evidence tiers

Since the scores can't be trusted, define the layers of evidence first.

  • T0 peer-reviewed venue (EMNLP/ACL/ICLR main track)
  • T1 arXiv paper + reproducibility artifacts (configs, raw data) published
  • T2 vendor self-reported benchmark
  • T3 independent audit/reproduction (third party, same conditions)
  • T4 community hands-on reviews (HN/Reddit/Substack)
  • T5 adoption metrics (stars, funding, API traffic)

Composite weights: academic evidence 20% · benchmark trustworthiness (independent reproduction) 25% · reviews 15% · adoption/ecosystem 20% · self-host & license 10% · operating cost 10%. This is my judgment, and gaps under 1.0 points should be read as ties.

The ranking

# System Score Strongest evidence / weakest spot
1 Hindsight 6.6 Paper + tied #1 in an independent pilot + competitors concede defeat on their own pages / heavy, costly, immature
2 Zep·Graphiti 6.35 Bi-temporal KG paper + fully disclosed bench setup / CE discontinued, dispute history
3 Letta (ex-MemGPT) 6.3 MemGPT paper is a foundational citation + honest benchmark skepticism / no standard bench, churny product strategy
4 Mem0 6.25 Overwhelming adoption (68.1k★, $24M, 186M API calls/mo) / scores are managed-platform-only, not reproducible on OSS
5 Cognee 5.7 Tied #1 at MemVerdict + honestly flags its own 10M result as "exploratory" / production graph store is paid
6 MemoryOS 5.55 The only EMNLP 2025 Oral / academic project, no ecosystem
7 Honcho 5.4 Best-in-class transparency (configs + raw data public) + differentiated user modeling / no paper, 38-pt gap in an independent run
8 MemOS 5.2 Two papers / all numbers from its own harness, docs partly Chinese
9 MemPalace 4.5 60.2k★ in 7 months, LLM-free raw mode / benchmark validity dispute (top_k=50)
10 OpenViking 4.3 ByteDance, 41.2k★, organic HN praise / zero papers or benchmarks, AGPL
11 Supermemory 4.2 Broad editor integrations / closed local binary, self-claimed #1s everywhere
12 LangMem - Convenient, MIT / 33.3% in independent measurement (off-pace from every baseline)

LangMem is the only one I left unscored: in independent measurement it isn't statistically tied with anything else, so applying the same scale would distort the rest.

Why the scores can't settle it

This is the most important axis of the survey - more than the ranking itself.

1. LoCoMo itself is shaky. Penfield Labs' audit (Dec 2025) found 6.4% of the answer key is wrong, and the LLM judge accepted 63% of deliberately incorrect answers. It cannot resolve an 88% system from a 94% one. Only 10 conversations, too.

2. Files + grep score 74%. The Letta team gave an agent nothing but attached files, grep, and vector search - LoCoMo 74.0% with gpt-4o-mini. That beats the 68.5% Mem0 reported for its graph variant in its own paper. Letta's own conclusion: "current memory benchmarks may not be very meaningful."

3. A 25.6-point swing between vendors. Zep's LoCoMo went 84% (Zep) → 58.44% (Mem0 scoring its rival) → 75.14% (Zep's rebuttal). Same system, same benchmark, no independent reproduction on either side.

4. Independent runs are undecided. MemVerdict, funded and run by no memory vendor, ran a pilot (n=30) with the official judge plus a 3-judge frontier panel: Cognee and Hindsight tie for best at 90.0%, statistically tied with a plain RAG baseline at 86.7%. Their words: "no memory system clearly beats plain RAG." Cost: $61.6–172 per 1k questions versus $5.45 for RAG - 11–31×.

5. The benchmarks are saturated. Top claims converge into the high 80s–mid 90s; discrimination is gone. Only BEAM (1M–10M tokens) still separates systems - and there the numbers (Hindsight 64.1 / MemOS 56.75 / Mem0 48.6 / Honcho 40.9) use different judges and configs, so they don't cross-compare either.

So this ranking measures "who has assembled evidence to back their claims," not "who is most accurate." In practice the reader model, the judge, and the config move accuracy more than the architecture does.

Where does Honcho sit

If you've read this blog for a while, you know I self-host Honcho. Which is exactly why I'll be blunt.

The strength is transparency: benchmark configs (deriver gemini-2.5-flash-lite, dreamer haiku-4.5) and raw results are all public on GitHub, backing LongMem S 90.4% · LoCoMo 89.9% · BEAM 100K 0.630. The peer/dialectic approach - modeling the user rather than retrieving facts - is genuinely differentiated.

Two weaknesses. There is no paper. And in the Keep Field Report's independent run, Honcho scored 51.8% on LoCoMo - a 38-point gap from the vendor's 89.9% that has never been reconciled. Add the background "dreaming" LLM calls, which put operating cost in the top group (~1.08M watt-seconds per 1k queries, estimated).

I'm keeping it anyway. My use case isn't a benchmark number - it's "the assistant understands me," and the structural approach still holds there. The v3.0.3 → v3.0.9 upgrade is worth queuing, though.

Patterns that repeat across reviews

  • r/hermesagent, "Memory Providers: I tested them all" (2026-05): "Hindsight is technically the best memory but too heavy, too many API calls, costly, hidden config, too many bugs" / "a thorny road to adoption - stabilized only through local patch overlays" / recurring unexplained server crashes
  • HN Ask HN (2026-02): "Mem0 stores memories but doesn't learn user patterns" / "user corrections are the highest-signal data and nobody capitalizes on them"
  • HN Show HN: Mem0 (2024): fresh accounts posting kudos - an astroturfing suspicion, with moderator flags
  • HN "Agent memory as a file format" (191 pts, 96 comments): "a whole lot of text to say it's markdown + RAG" / "the tool is called bash, the optimized API is grep, the storage is the filesystem" / OpenViking got unsolicited praise in that same thread

Applying it to my stack

  • Keep Honcho, queue the upgrade. It's already integrated with pi via the pi-honcho extension, so switching costs dominate
  • billion-context-pi is worth an experiment. "In-session compression + cross-session memory" are exactly orthogonal. Caveats: ① verify the nonstandard license, ② with zero published evidence, A/B it on my own workload (token usage, cache hit rate, long-task success rate), ③ kill it the moment cache hit rates sag - on the GLM flat-rate plan, prefix-cache policy is the core variable
  • Mind the GLM-only constraint. No embedding API on the plan, so evaluating embedding-dependent systems (Hindsight, Mem0, Cognee) requires local embeddings first (ollama + nomic/bge)
  • Measure the RAG baseline before adopting anything new. Independent runs say files+search is still strong; the rational procedure is checking that a new system actually beats it

Sources

Change the weights and the order changes. By adoption alone Mem0 is #1; by academic rigor alone MemoryOS is #1. This ranking is what falls out when you commit, explicitly, to a single yardstick: the strength of the evidence.

NAV
comments (0)
MENU
TAGS
COUNTER
001065
TOTAL VIEWS
0007
TODAY
© 2026 PINION - Resisting the dead internet theory.
BEST VIEWED AT 1280×1024 ◇ NETSCAPE NAVIGATOR 4.0
v0.2.0 · since 2026