Agent memory, three ways: hindsight vs HydraDB vs keyword search on our own 837 items

hindsight claims to be the most accurate agent memory system ever tested, and HydraDB publishes a comparison table against it. Both are vendor material about a hosted product. None of it can be reproduced on this machine. We are choosing where an agent memory lives, so the benchmark that settles it is our own memory, on this machine.

The corpus is what this machine agent remembers: 837 items and 1,305,115 characters. It comes from 65 real sessions in the harness state database (tool payloads excluded). The rest is kanban tasks, comments and events, MEMORY.md and USER.md, the skill index, and the LLM Wiki notes. Nothing was curated down for the test.

Every arm answered the same 24 probes. The set: 4 environment facts, 3 preferences, 6 kanban facts, 6 session facts, 4 facts that changed over time, and 1 question that needs two sources at once. Scoring is lexical and automatic, with no model judging the answers. tok@k means the gold token appears in the text of the k-th result. prov@k means the k-th result is the corpus item the fact came from.

1.Result

2.Where the win is, and where it is not

The 3-probe edge at rank 1 is not spread across the corpus. On environment facts hindsight gains 2 (3 of 4 probes against 1 of 4). On preferences it gains 1 (2 of 3 against 1 of 3). On kanban, sessions and updates it is level with the keyword baseline. In the top 5 it keeps environment and preferences, gains 2 on updates, and gives back one kanban and one session probe.

hindsight also returns the right source item more often: 17 of 24 in the top 5 against 15 for keyword and 12 for HydraDB. It answers in a rewritten, self-contained sentence rather than a raw chunk. That is a different product: the 13 at rank 1 measure a number that survived extraction, not only a retrieval hit.

3.What the accuracy costs

Ingesting 1.3 million characters took 50.0 minutes and 4,553,279 model tokens. Every write is one LLM extraction call, 3,588.8 ms on average and 15,555.7 ms at worst. Recall calls the model too. So the 17,511 ms median is model latency, not database latency. That is why the keyword baseline, at 0.1 ms and no ingest at all, stays the default here.

HydraDB built its store in 180.7 s with no model calls. Steps: 837 chunks in 21.8 s, 6,000 term nodes in 153.0 s (26.0 ms per node), 43,167 edges in 5.9 s (136.3 microseconds per edge). All of it was hand-modelled. It rejects a bare node CREATE ("only one-hop edge patterns are executable in Query engine CREATE", 0 of 837 written). String ids fail too ("node id property must be an integer").

• HydraDB needed 4 failed starts before it would answer: GRAPH_ALLOW_PLAINTEXT, an existing GRAPH_DATA_CACHE_DIR, and a token file. Without the file it exits with Os { code: 2, NotFound }.
• The image tagged 0.2.0 reports itself as version 0.1.0 and warns that build provenance is unknown.
• Batched UNWIND is rejected ("supports one-hop relationships only"). A string id is rejected, and so is a non-integer value in a relationship field.
• hindsight needed a bank per corpus and 92 documents for the 837 items. Its own Postgres held only 7.5 MB because the facts live as 4,951 nodes and 152,484 links.

4.What we could not measure

• Multi-hop traversal, the one thing HydraDB is built for. These probes are single-fact recall, and the single question that needs two sources failed at rank 5 in all three arms.
• Their published numbers (82% on BEAM 1M and 90.79% on LongMemEval-s) come from the hosted product. We could not reproduce them, so they stay vendor-reported, not verified here.
• n = 24 probes: one probe is about 4 points, and a ±20% swing is 5 probes. The rank-1 gaps between the arms are 2 to 3 probes, inside that band.
• The HydraDB arm is our hand-modelled schema, not a vendor-recommended ingest. A better model would likely raise its scores, so this is a floor, not a ceiling.
• hindsight returns rewritten facts and the other two arms return raw corpus items, so tok@k measures survival through extraction as much as retrieval quality.
• One machine, one corpus, one day: MacBook Pro M1, 8 GiB, macOS 26.3, 837 items from 65 real sessions, measured 2026-10-09. In all three arms the memory never left this machine.

Decision

Use hindsight when

The question is fuzzy and phrased your own way.

• It is about this machine or your stated preferences
• A 17 s answer fits the workflow

Keep keyword as the default

It matched 16 of 24 probes in the top 5 with no ingest, no model calls and a 0.1 ms answer.

• hindsight bought 2 more probes for 50 min and 4.55M tokens

Try HydraDB for paths

Its measured edge is 16 ms and a small store.

• Do not buy it for accuracy: these probes return a floor, not a ceiling, for a hand-modelled schema
• Multi-hop traversal, the reason to run a graph engine, is untested here

What would change this

Probes built around multi-hop and time ordering, a real BM25 (keyword ranking) arm, or a queue that hides 17 s of recall latency behind a background job.

Raw rows for every figure above, including the engine frictions and the limits: docs/analysis/2026-10-09-agent-memory-three-arms.jsonl (8 rows). The harness that produced them stays at ~/Projects/memory-bench/ (corpus.py, arm_a_hindsight.py, arm_b_hydradb.py, arm_c_keyword.py, score.py, and the per-query evidence under evidence/).