headroom: the 10,144 → 1,260 demo reproduces at 86% on our worker logs, and at 2.8% through the shipped pipeline

Published 2026-10-07 09:25

The same 10,476-token log dump went to 1,464 tokens (86.03%) when headroom’s log compressor ran, 10,180 (2.83%) through the shipped router, and 10,476 (0%) while its 261 MiB model was cold. Six of ten answers died with the lines.

Headroom is the fastest-rising compressor in our corpus this week: 74,524 stars (+423 in 7 days), Python, 88 MB on disk. Its pitch is one number from the README GIF: a 10,144-token log dump compresses to 1,260 tokens and the same FATAL is still found. The library itself ships (compress(messages)), so the claim is checkable on this machine.

We fed it logs of ours: the worker container’s own output (docker logs bestofgit-cron-1) plus the ops logs of this pipeline, 31,825 bytes, 1,048 lines, 10,476 tokens. Then we asked ten questions whose answers are single log lines, before and after.

1.Result

One machine: MacBook Pro M1, 8 GiB, macOS 26.3, headroom-ai 0.40.0 on Python 3.13.13, tokens counted with tiktoken o200k_base (headroom’s own gpt-4o mapping), measured 2026-10-07. “Router” is the shipped compress() pipeline; “LogCompressor” is the log component it dispatches to.
Path	Tokens	Saved	Lines kept
Their README GIF	10,144 → 1,260	87.58%	—
Our payload · shipped compress(), model cold	10,476 → 10,476	0.00%	274 sections, all plain text
Our payload · shipped compress(), model warm	10,476 → 10,180	2.83%	1,286
Our payload · LogCompressor directly	10,476 → 1,464	86.03%	100 of 1,049

Their README demo, same kind of payload

87.58%

A 10,144-token log dump to 1,260 tokens, the number this article set out to check

Our log through the log compressor

86.03%

10,476 tokens to 1,464, with both the FATAL and the ERROR lines intact

The same log through the shipped router

2.83%

Model warm, 274 sections, every one called plain text and sent down the ML branch

Answers left after the 86% pass

4 of 10

Ten questions answerable only from a log line, asked once per arm at temperature 0

Their headline is not marketing arithmetic, it reproduces. On bytes of the same order and the same kind (a log dump of about ten thousand tokens) the log compressor lands at 86.03% against their 87.58%. That gap is the difference between one payload and another, not between a claim and a fudge.

2.The compressor is real; the dispatcher decides whether you get it

Headroom’s pipeline is not one compressor. A content router classifies each section of the payload and picks a strategy. Logs go to the Rust log compressor. Prose goes to the ML model (kompress-v2-base). Code goes to an AST-aware path. Our payload never reached the log path: the router called it plain text and sent all 274 sections down the ML branch.

The same two paths on five payloads of ours, in the same warm process. magika 1.0.3 (the detector behind the router) is the last column. Our logs are plain text at low confidence, and the one payload that reached the log compressor is the one it called CSV.
Payload we own	Tokens	Shipped router	LogCompressor	magika label
Worker + ops logs (10 questions here)	10,476	2.83%	86.03%	txt 0.27
Full test-suite output (713 tests)	24,054	42.82%	95.81%	txt 0.31
Ops logs on their own	8,004	16.17%	80.27%	—
Worker container log on its own	2,460	0.04%	79.63%	—
next build --webpack output	1,069	8.51%	97.47%	txt 0.32
A 42,112-byte debug log from this box	11,836	89.28%	89.28%	csv 0.82

The spread is the finding. Across six payloads of real logs from one machine, the shipped pipeline rescues 89.28% from the one it classified as structured data. On the other five it saves 0.04% to 42.82%, while the component their demo shows does 79.63% to 97.47% on every one of them. The same bytes move from 2.83% to 89.28% on a classifier decision alone.

The 42,112-byte row is reported as a measurement only: those bytes are a chat-gateway log and contain identifiers, so we publish the number and not the payload. Every other payload is reproducible from this repository.

3.What 86% costs the answers

Ten questions answerable only from a line in the log, asked once per arm to deepseek-v4.1-flash at temperature 0. The log compressor kept both error lines verbatim, the promise their demo makes, and dropped the numbers around them.
Arm	Tokens sent	Correct	What went wrong
Raw log	10,476	10 of 10	—
Shipped router (ML text path)	10,180	9 of 10	“Worst stage” answered “snapshot:stars 33”, not “discover:trending 38”
LogCompressor directly	1,464	4 of 10	Six answered NOT_IN_LOG; the seventh produced 8344, a number in no log

Both error-bearing lines survived the 86% pass. unknown stage "deps:backfill" and ERROR: aggregate function calls cannot be nested are there byte for byte, which is exactly what their demo promises. What did not survive: 52312, 49035, 10450, redditBlockedFailures: 13, the timestamp 04:30:18 and the stage/hours table.

Their own accuracy table is careful about this, and ours should be too. SQuAD v2 holds 97% “at 19% compression” and BFCL 97% at 32%. Our four-of-ten arm ran at 86%. The claim is not wrong. It is a claim about keeping errors at a low ratio, and the demo number is a high ratio.

4.What it costs to run

Disk: the venv is 331 MB for the base package, 539 MB with the [proxy] extra the detector needs, and 1.1 GB with the [ml] extra (torch) the text model needs. A 274,049,435-byte ONNX artifact arrives on first warmup. The README installs headroom-ai[all].
Speed: the log compressor is 4.6 ms median on 10,476 tokens. The shipped router on the same bytes is 1,175.7 ms median, because the ML branch runs per section (274 of them). Their published 0.21 ms p50 is quoted for a JSON search result, not for this path.
Memory: peak RSS 124 MB for the log path, 698 MB once the ML model is loaded, against a 119 MB interpreter floor, on an 8 GiB machine.
Cold start matters: an unwarmed install answers with “Kompress model not ready; requests will not be compressed” and passes the payload through untouched, which is how our first run measured 0.00%.

Their privacy sentence holds up in the strong sense. With HuggingFace offline and every proxy pointed at a dead port, both paths returned byte-identical output to the online run (same sha256). Nothing in the payload leaves the process. The one network call is the model download itself, on first warmup.

Decision

Worth it for log dumps: 86.03% on our worker logs with both error lines intact, at 4.6 ms and 124 MB.
The log compressor is the real thing
Do not expect it to fire by itself: The shipped router sent 6 of our 6 payloads to the text path: 0.04% to 42.82%.
Call the log path, or measure before believing the headline
Check the ratio against your questions: At 86% our ten log questions fell to four.
Their 97% accuracy numbers are measured at 19% and 32%
Against context-mode: context-mode measured 99.90% by never putting the bytes in the conversation.
headroom rewrites them in place and keeps errors
Different trade, both honest

What we did not measure

One machine, one accuracy payload, ten questions, one answering model at temperature 0: the 4/10 and 9/10 are counts, not a benchmark.
No agent integration: no headroom wrap, no proxy, no Claude Code or Codex session, so “my agent works better wired to it” is untested.
We installed base + [proxy] + [ml]; the README’s [all] adds extras we did not install (evals, image, reports, spreadsheet, voice, memory, mcp).
Routing is version-dependent: headroom-ai 0.40.0 on 2026-10-07 decided every route here, and a detector update can move all of them.

Evidence: docs/analysis/2026-10-07-headroom-compression.jsonl (61 rows). It holds the install sizes, the claims as quoted, the payload hashes and every ratio. It also records the presence or absence of each key line, plus all 30 answers. The offline egress test and the traps we hit ourselves are in there too. This article measures their claim; it is not a score for a tool.

Repositories this covers