e2e: seven replays, no model calls, and the 25 seconds a renamed button costs

tester-army/e2e drives a browser with a model, then tries to stop paying for it. Its README makes one claim a reader can check. An agent step that a later assertion verifies records its actions, and the next run replays them with no model calls until the app changes.

We wrote one case against our own site: open /search, type playwright, submit the form, then assert a result link is visible. Four arms on one machine, with model calls counted from the runner’s own report events, not from a provider bill.

1.Result

Seven replays in a row made no model calls, spent no tokens, and never rewrote the recording. The median replay took 2.43 s against 11.85 s for the run that recorded it. It is also 1.35x the same flow written as plain locators, and that gap is what the route and anchor checks cost.

2.The bill for renaming one button

Then we changed the app: the submit button’s label went from Search to Find repos. The recording says tap button Search, so the next run replayed the first action and could not find the control. It handed the step over, and the agent found the new name and finished the test.

An entry remembers the control it recorded, not the intent behind it, so changing the label back is another miss. Both directions cost 2 model calls and about 25 s. The next passing run re-records, and the run after that replays again. A recording of tap button Search is stale the moment that name changes.

3.With no model key at all

The README’s last sentence is the cheapest one to check: tests without agent steps need no model. We removed the key from the environment and ran the same files again.

A recorded step replays with no credentials at all: the entry on disk is all it needs. Only a live step reaches for a model, and that failure left the entry untouched. The replay after it still replayed, with no key.

Two problems on the way are worth recording. The OpenCode key stored in the CLI’s auth file answered 429 “Go usage limit exceeded” on both routes. Port 3000 replied HTTP 200 with a different app entirely (OWASP Juice Shop), and a status code alone would have measured the wrong site.

Decision

Keep the cache on

Seven replays cost 0 model calls and 0 tokens.

• Against 28,627 tokens for the one run that recorded the step

Expect a miss on every UI rename

One renamed button handed the step back twice, once on the change and once on the revert.

• At 24.64 s and 28.52 s

Use --strict-cache in CI

A stale recording is silent by default.

• The test passes and every run pays the model again until someone re-records it

Budget the extra 0.63 s

A replay is 1.35x a plain locator run.

• The route and anchor checks are not free. They are far cheaper than a model call

What we did not measure

• The n behind the interesting number: one recording run, one hand-off per direction. The 4.9x speed-up rests on a single recorded run.
• Their example models: ours was opencode.go/deepseek-v4.1-flash, not the gateway or Anthropic models the project documents, so turns per step are model-specific.
• A production build: the app under test was next dev on port 3100, so all four arms share the same dev-server overhead.
• Provider billing: model calls come from the runner’s own report events, not from the provider’s usage page.
• The data: /search reads our database, so the 96 matches and the card anchors the recording captured are not fixed inputs.

Raw rows, including the runs with no key and the failures above: docs/analysis/2026-10-07-tester-army-e2e-replay-without-model-calls.jsonl. This is our measurement of the claim, not a score for the tool.