caveman: the 129.8x page number is one filtered answer, and our lockfile came back untouched

caveman is three things: a skill, a proxy and caveman browse. The skill makes an agent answer short. The proxy shrinks what the agent reads. caveman browse hands the agent a compressed accessibility tree instead of a full page dump. Its README leads with two numbers.

The claim: 33.2% fewer input tokens through the proxy, and 129.8x smaller web pages for the agent. We installed the CLI (2.0.1) on an M1 Mac and re-ran the browse half against caveman’s own fixture first, then against ten real pages.

1.Their fixture reproduces, almost to the token

Four rows land on the number their benchmark reports. The gap is the third: 129.8x compares a query-filtered answer (their 121 tokens) against the whole Playwright snapshot (15,704). Full against full, caveman is 1.18x smaller, the 14.88% their own BENCHMARK.md already states.

The same asymmetry reverses on a tiny page. The checkout form is 67 Playwright tokens. caveman’s full snapshot is 157 and its focused one 113, because it also returns action UIDs, a recovery handle and exact token accounting. Their README calls this the small-page loss, and it reproduced.

2.On ten real pages

Five of the ten pages were read by both tools on the same rendered page, and the win there is 0.95x to 3.28x with a median of 1.32x. One page came back slightly larger. Nothing here is near three digits.

The large ratios are not compression. On crates.io caveman snapped a page with 0 anchors and 0 characters of text; six seconds later the same tab held 85 anchors and 7,729 characters. Our own homepage came back at 169 tokens against Playwright’s 6,122, but the tab it read held 149 characters of visible text, a skeleton list, not a compressed page.

caveman browse does not wait for a page to settle, so a streaming or JS-rendered page can be captured before it exists. Two more pages returned an outright error: cave_browser_snapshot_uncompressed, thrown when a 2.45M-token (Wikipedia) or 66,828-token (docs.rs) accessibility tree fails to compress. The CLI withholds the raw tree rather than passing it through.

3.The compressor on five real dev files

Their per-file table claims 98.5% to 99.1% off CSV, logs, YAML, test output and JSON. Their own method note says the measurement passes each file through the engine with the case question as the query. Each fixture also hides one record in 61 to 72 KB of noise. Fed whole files with no query, only the Docker log moved.

The proxy’s own entry point agrees: caveman shrink -- docker logs --tail 4000 bestofgit-cron-1 gave 2,132 to 1,272 tokens (40.3%). The same command on the lockfile saved 1% (103,796 to 102,755).

Decision

Read 129.8x as a query result

The number is one filtered answer against a whole page. Full against full was 1.18x on caveman’s own fixture; on five real pages, 1.32x at the median.

• Full against full on their own fixture: 1.18x, the 14.88% their BENCHMARK.md states
• On five real pages the win ran 0.95x to 3.28x, median 1.32x

Do not expect your files to shrink

A 222,998-byte lockfile, a 93,696-byte CSV, a 54,008-byte JSON and a 115,027-byte test log all came back unchanged. Only the log halved.

• The lockfile, the CSV and the JSON all saved 0% with no query in the call
• caveman shrink on the Docker log saved 40.3%, on the lockfile 1%

It fails loudly, which helps

Wikipedia and docs.rs returned an error instead of dumping two million tokens into the context. A silent full-page dump would have been worse.

• The 2.45M-token Wikipedia tree and the 66,828-token docs.rs one both refused
• The CLI withholds a tree it cannot compress instead of passing it through

Watch what it captured

Two of our ten pages were captured before they rendered, so their ratios measure an empty page. Check the snapshot, not the ratio.

• crates.io snapped a page with 0 anchors; six seconds later it had 85
• Our homepage read 169 tokens, but the tab held 149 characters of text

What we did not measure

• The 33.2% proxy number. It is provider-reported input tokens over whole Claude Code sessions, and this machine has no signed-in agent CLI. caveman’s WRAP-BENCHMARK.md says the result cannot be reproduced from the checkout and ran on an older engine.
• Provider billing. Every caveman figure here is an inferred o200k count from its own counter, which caveman itself labels inferred. No API invoice was involved.
• The query-aware compressor. caveman-engine compress has no query flag, so the path that produced their 98.5% file numbers is not reachable from the CLI we tested.
• Their five-run medians. Our browse rows are n=1 per arm per page; their BENCHMARK.md runs five Chrome rounds and reports min–max ranges.
• caveman’s own Chrome launch. It failed in this session, so every browse row attaches to a headless Chrome we started by hand on port 9334.

Raw rows (30) including the refusals and the blank captures: docs/analysis/2026-10-07-caveman-token-and-page-size.jsonl. Tool: @caveman-ai/cli 2.0.1, binary release bin-v2.0.2. This is our measurement of their claims, not a score for a tool.