ai-memory vs the memory Hermes already runs: 837 items, 24 probes

ai-memory ships a git-backed markdown wiki with full-text search, entities and links, and calls it the source of truth for agent memory. We already run something similar in shape and much smaller: a block of text that is injected into every turn. The question worth measuring is not which one sounds better, but which one answers our own questions.

1.What the incumbent is

The arm we call Hermes memory is three tools, and naming them matters because each one has a different cost. memory writes MEMORY.md and USER.md under ~/.hermes/profiles/<profile>/memories/. The whole block is pasted into the system prompt on every turn: 3,551 characters today against caps of 2,200 and 1,375, so it sits at 99% of its budget. It cannot grow without deleting something.

search_files is the second tool: ripgrep over files, no ranking, no index, no model. The third is session_search: SQLite FTS5 (full-text search) over past sessions. This arm measures memory plus a ranked keyword search, which is the same job search_files does with a score, so the comparison is possible. session_search is not in these numbers.

The corpus is the frozen clone from 2026-10-09: 837 items and 1,305,115 characters. It draws on 65 real sessions, kanban tasks and comments, MEMORY.md, USER.md and the LLM Wiki notes. The 24 probes are the same 24, and the scoring is lexical, with no model judge. A gold token inside the k-th result counts for tok@k; prov@k counts the k-th result being the item the fact came from.

2.Result

Read the two columns together. ai-memory wins in the top 5 and loses at rank 1: 18 against 16 in the top 5, but 7 against 10 at rank 1.

ai-memory also returns the right source item more often than keyword search: 11 against 6 at rank 1, because every page is one corpus item. The page path is the item id. Keyword search still puts the answer first more often, which is what an agent reading one line needs.

3.ai-memory was full-text only, and its snippets hide answers

ai-memory 2.0 defaults to local embeddings, and the store told a different story after 837 writes: zero rows in page_embeddings, zero entities, 65 links.

The write path we used, one page per item through write-page, did not compute vectors on this build. So what we measured is its zero-LLM path: SQLite FTS5 ranking, 290.8 ms per question, no model call at any point. Running ai-memory embed afterwards would be a different arm and is not in these numbers.

The JSON the CLI prints carries a truncated snippet, not the page body. Scoring the snippet gives 6 of 24 at rank 1 and 14 of 24 in the top 5, against 7 and 18 when the caller opens the page. Four probes lose their answer inside the snippet, which is a real cost for an agent that reads output and moves on.

4.What we could not measure

• ai-memory with embeddings. The CLI write path left the vector tables empty, so the hybrid path advertised in its README is untested here.
• session_search. The Hermes arm covers memory and file search, not SQLite FTS over past sessions, which would need its own ingest.
• Multi-hop and time ordering. The single question that needs two sources failed in both arms.
• Thai tokenisation. Both indexes segment Thai text differently from the Latin probes and 8 of the 24 questions are Thai; a failure there is not a ranking failure.
• n = 24 probes, so one probe is about 4 points. The 2-probe gap between ai-memory and keyword search at rank 5 is inside that band.
• One machine, one corpus, one day: MacBook Pro M1, 8 GiB, macOS 26.3, 837 items from 65 real sessions, measured 2026-10-25. Nothing left this machine.

Decision

Use ai-memory when

You want a human-editable wiki as the source of truth and 291 ms per question fits the loop.

• 18 of 24 in the top 5, the best of the two arms
• Zero model calls, nothing leaves the machine

Keep the injected block plus search

It answers 20 of 24 today, with no ingest and a 0.1 ms query, and it is already paid for.

• It is at 99% of its 2,200 and 1,375 character caps
• Anything new has to replace something old

What would change this

An embedding pass on the ai-memory arm, a real BM25 (keyword ranking) arm, or probes that need two sources at once.

Raw rows for every figure above, including the per-category split, the limits, and the excluded telemetry arm: docs/analysis/2026-10-25-ai-memory-vs-hermes-memory.jsonl (8 rows). The harness stays at ~/Projects/memory-bench/ (arm_e_aimem.py, arm_c_keyword.py, score.py, and the per-query evidence under evidence/).