claude-mem says 10x, our memory says 3.3x: pricing a retrieval claim on 837 real items

claude-mem is the first memory tool we have read that prices retrieval as a rule instead of a benchmark. The rule: get a compact index first, fetch full details only for the IDs you filtered to, never fetch details without filtering. It prices all three: "~50-100 tokens/result" for the index, "~500-1,000 tokens/result" for the details, "10x token savings" for the pattern.

That is a claim about tokens, so it does not need a new harness. We already have one from 2026-10-09: the same 837-item clone of this machine agent memory and the same 24 probes. The clone holds 1,305,115 characters from 65 real sessions, plus kanban events, MEMORY.md, the skill index and the wiki notes. Those probes ranked hindsight against HydraDB against keyword search.

That bench gets a fourth arm here, with the corpus file untouched. We installed it without an account: CLAUDE_MEM_ONLINE_OPTIN=false npx claude-mem install --provider host --ide omp, then npx claude-mem telemetry disable. The worker runs on 127.0.0.1:37701 over its own SQLite file (3.9 MB) and a local Chroma process. Nothing in this measurement left the machine.

Its real capture path is lifecycle hooks inside another agent. This machine runs none of those for memory, so the corpus went in through its own write endpoint (POST /api/memory/save, the one the save_memory tool calls), one item per call. It took 837 items in 27.8 s, with zero model tokens and zero errors. That choice has a price, and it is in the limits below.

Then we ran both shapes the tool documents. Guided mode is the 3-step flow from the README. It takes an index of 20 rows, context around the 3 we picked, and full details for those 3, which are the tool defaults for both limits. Auto mode is the same bounded pipeline in one call. Every token was counted with tiktoken cl100k_base over the exact text the worker returned.

The naive alternative is by construction what the rule forbids: full details for all 20 index candidates. The claim is about that comparison, so it is the one we price.

1.Result: the prices are right, the headline is not

The factor is arithmetic on how many details you fetch, not a property of the tool. Details for all 20 index candidates cost 11,809 tokens; details for the 3 you filtered to cost 1,545, which is 7.6x. One detail would be 22.9x by the same sum.

The documented 3-step flow adds the context step, and that is what pulls the factor down to 3.3x. The index (1,184 tokens) plus the context (864) cost more than the details they were meant to save you from. A detail row carries 8.7 times what an index row carries. The one-call mode skips that step: 11,809 against 1,525 is 7.7x, which is where the claim nearly holds.

2.How it ranks against the three arms we already ran

The split matters. claude-mem puts the right corpus item at rank 1 more often than any arm we have measured. The count is 11 of 24 against hindsight 8, keyword 6, HydraDB 5, and it does that in 263 ms with no model call at query time. But the index shows titles, so the fact itself is visible at rank 1 in only 7 of 24 probes.

Four of its eleven misses had the right item inside the top 5 while the number was still absent. The item we ingested is a chunk of a file, and the fact lives in a different chunk of that file. That is a chunking artifact of feeding it our corpus, not a ranking failure. It is also why prov@1 and tok@1 disagree here.

Against the raw-chunk arms, index-first is a straight win on cost. Guided mode spends 3,612 tokens per question, against 5,850 for our keyword baseline and 6,727 for HydraDB. One-call mode spends 1,525. That is 1.6x to 3.8x cheaper, for less accuracy at rank 1.

Against hindsight it is not. hindsight answers in 552 tokens because it rewrites the fact instead of handing back a chunk. That is 6.5x cheaper than claude-mem guided, 2 probes more accurate at rank 1, and 17,511 ms per query instead of 263 ms. Same corpus, same probes, three different trades.

What installing it costs

• The worker reports "Dependencies: degraded (Claude CLI setup required)" and npx claude-mem doctor fails one check: the hook launcher on PATH is the npx shim, not a protocol launcher. The install assumes an IDE we do not run.
• Their LLM extraction never ran because we pushed pre-chunked items in. 37 of 838 observations ended up with a frontmatter rule or a sub-8-character title. The index re-ranks by title words, so those 37 are reachable mainly through the vector side.
• Installing for OMP writes a 16.3 KB bridge at ~/.omp/agent/hooks/pre/claude-mem.ts that posts every OMP session into the store. We removed it after the measurement and kept a copy, so no worker run changes behaviour by surprise.
• It stays resident. Bun and uv are requirements, a local chroma-mcp process was still live after the runs, and the store is a SQLite file we now own (3.9 MB for 1.3 million characters).

What we could not measure

• Their real capture path. Hooks and LLM extraction are how this product is meant to be fed, and this machine cannot run them for memory, so nothing here says how good their generated titles are.
• The relevance choice. We requested details for the top 3 index rows deterministically; an agent reading the index picks by meaning, which can only help, so the accuracy numbers are a floor.
• n = 24 probes on one machine on one day. One probe is about 4 points, and the rank-1 gap to the keyword baseline is 3 probes, inside that band.
• The token counts use tiktoken cl100k_base, while the 50-100 and 500-1,000 figures name no tokenizer. They matched inside a band, not exactly.
• Anything about their hosted observer: the sign-in, the 30-day trial and the plan fallback. We installed the local path with no account, and the hosted side is untested here.

Decision

Adopt the shape, not the product

The measured part is the pattern: a 59.2-token index row, then 515-token details only for the IDs you chose, at 263 ms and no model call.

• An index step is the 1.6x to 3.8x we can take: our recall path hands over 5,850 tokens of raw chunks per question
• Write that index from our own store instead of adding a second memory database

Do not put claude-mem in front of Hermes memory yet

Its ingest is bound to other agents hooks, and the OMP bridge it installs touches every worker run.

• Feed it from our own writes if we adopt it, then re-measure on the same 24 probes
• Its documented pipeline costs 3.3x, not 10x; the 7.7x needs the one-call mode

Keep hindsight for fact-shaped answers

Nothing here displaces it: 552 tokens per answer and 13 of 24 at rank 1 against 3,612 tokens and 7 of 24.

• The price is the 50-minute ingest and 17,511 ms per query

What would change this

Feeding it the real capture path on a machine that runs Claude Code or OMP, or multi-hop probes instead of single-fact recall.

Raw rows for every figure above, including the frictions and the limits: docs/analysis/2026-10-15-claude-mem-ten-x.jsonl (11 rows). The harness stays at ~/Projects/memory-bench/: ingest_claudemem.py, arm_d_claudemem.py, emit_evidence_claudemem.py, the per-probe evidence under evidence/, and the same corpus.py, questions.jsonl and score.py the 2026-10-09 round used.