Hindsight: we installed the 2.89 MB memory binary and checked whether it remembers
Published
Four facts in, then the same four after a restart: all correct. At 5,000 facts on an 8 GiB Mac, answer time grew from 272 ms to 3,088 ms and accuracy slipped to 10/12.
Hindsight stores facts and answers questions phrased your own way. Its README claims the most accurate agent memory ever tested, state of the art on LongMemEval. The eye-catching part is the download: 2.89 MB. So we measured the one question that matters on this machine: does it remember?
1.Result
| Path | Size | What it can do |
|---|---|---|
| Platform binary (darwin-arm64) | 2.89 MB | Nothing alone: Cannot connect to Hindsight API, exit 1 |
| Python embedded (pip install hindsight-all) | 2.13 GiB over 214 packages | Starts a server in-process against a bundled Postgres |
| Docker (documented as the first path) | tested on 2026-10-07 | 2.39 GB image, healthy API in 7 s, 930 MiB idle; 4 / 4 answers before and after a restart |
The download that starts it
2.89 MB
Platform binary alone cannot connect to the API: it exits 1
Answers after a restart
4 / 4
Four synthetic facts stored, then the same questions asked of the restarted server
Answer time at 5,000 facts
3,088 ms
272 ms on a 4-fact bank; the same 12 questions asked back at every stage
Correct at 5,000 facts
10 / 12
Two misses at the top of the scale, with 204.2 MB on disk and 1.15 GiB peak
| Measure | First run | After restarting the server |
|---|---|---|
| Correct answers | 4 / 4 | 4 / 4 |
| Answer time | 41–58 ms | 53–73 ms |
| Larger bank (20 facts, 5 questions) | 5 / 5 | 132–160 ms |
| Facts stored | Answer time | Correct | Store on disk | Peak memory |
|---|---|---|---|---|
| 500 | 1,538 ms | 12 / 12 | 106.2 MB | 1.04 GiB |
| 1,000 | 1,823 ms | 12 / 12 | 111.8 MB | 1.04 GiB |
| 3,500 | 2,933 ms | 12 / 12 | 172.5 MB | 1.15 GiB |
| 5,000 | 3,088 ms | 10 / 12 | 204.2 MB | 1.15 GiB |
Decision
- Use it when
- you need recall over more material than a prompt can hold, and memory must survive a restart. It did: 4/4 after one.
- The bank held across a restart: 4 / 4 before and after the server came back
- It answered questions phrased differently from the stored sentences
- Skip it when
- you only need ten stable facts. A text file and an agent that reads it cost no RAM, and this run peaked at 1.09 GB.
- Ten facts in a text file cost no RAM and no install at all
- The database floor is about 100 MiB before the first fact is stored
- Real cost per fact
- 3.47 s to write, one model call per fact, plus 61.6 KiB on disk on top of a database floor of about 100 MiB.
- 3.47 s per single-fact write, because every fact is a model call
- One call per 20-fact document costs 0.85 s per fact instead
- At scale
- at 5,000 facts an answer took 3,088 ms, against 272 ms on a 4-fact bank. Only 2 of 12 questions missed, and no machine limit appeared at 1.15 GiB.
- Answer time grew ten-fold from 272 ms on 4 facts to 3,088 ms on 5,000
- Accuracy slipped to 10 / 12, and the store reached 204.2 MB on disk
Details worth knowing
- It did more than copy. Asked about the ink, it answered “cyan ink since Tuesday, September 29, 2026”. It also turned 20 stored sentences into 40 facts (world + observation types).
- The binary and the heavy server are one system. Pointed at the embedded server, bank list returned in 48 ms with fact_count 40.
- Write in documents, not one fact at a time. One call per 20-fact document cost about 16.9 s, 0.85 s per fact, against 3.47 s for a single-fact write.
- Writes cannot be parallelised from the Python client. All 4 and 8 concurrent retains failed with Timeout context manager should be used inside a task.
- Version history: release v0.10.2, 25,438 stars, MIT licence, 764 MB repository.
Not tested
- Their benchmark claim (LongMemEval, reproduced by Virginia Tech and The Washington Post) is the project’s own claim. We reproduced none of it.
- MCP (Model Context Protocol) transport, mcp_enabled=True, was never wired to an agent. Whether an agent uses this memory better is still open.
- No machine ceiling appeared. We stopped at a self-imposed 5,000-fact ceiling, with RAM at 1.15 of 8 GiB and 36 GiB still free on disk.
Scale evidence: docs/analysis/2026-10-05-hindsight-memory-at-scale.jsonl (12 rows: per-stage write time, disk growth, recall latency and accuracy, the failed concurrency probe, and the note that no machine limit was reached). Install evidence: docs/analysis/2026-10-05-hindsight-memory-on-a-mac.jsonl (18 rows, including the project’s claim flagged tested_by_us=false). Method: the v0.10.2 binary was verified by SHA-256 (attestations exist for the Python wheels, not the platform binaries). The server came from pip in a clean virtual environment, with bundled Postgres for storage.