strix never finished one run, and our hand-run probes produced their own false positives
Published
We pointed strix 1.7.0 at two deliberately vulnerable apps here. Its own model quota died first (HTTP 429), so the agent produced nothing. The 15 findings came from manual curl and python, and 3 of 9 probes on the first target did not reproduce.
strix is an autonomous AI pentester written in Python. You point it at a target and it runs its own exploit loop, deciding what to try next and reporting what it broke in. Our corpus has it at 66,831 stars, 1,319 of them added in the last seven days.
Its README makes a claim about quality rather than speed: findings are validated through working proofs-of-concept, "without the false positives of static analysis tools".
That claim is checkable without touching anyone else’s site. This laptop already runs two apps built to be broken: OWASP Juice Shop on port 3000 and DVWA on port 8081 at security level low. Both publish their own vulnerability checklists, so an answer key exists before we start.
1.Result: 15 findings, none of them from strix
strix 1.7.0 with its sandbox image (1.3.0) did not complete a single run. Its own model call failed with HTTP 429 "Go usage limit exceeded". The agent needs its own LLM credits, and they were gone before the first exploit step.
One attempt also pointed strix at a paid model on the same key and spent what was left. The evidence instead holds a manual pass with curl and python requests against the same two targets. Every row carries the exact request, and every finding was re-run by hand.
| Finding | Severity | Endpoint | Re-run with |
|---|---|---|---|
| SQL injection in login (auth bypass) | critical | POST /rest/user/login | email: admin@juice-sh.op' OR 1=1-- |
| IDOR on another user’s basket | high | GET /rest/basket/{id} | valid token + incremented id |
| NoSQL injection in product search | high | GET /rest/products/search | q[$gt]=x |
| Sensitive file listing under /ftp/ | medium | GET /ftp/ | plain GET |
| Prometheus metrics exposed | low | GET /metrics | plain GET |
| Malware samples left in /ftp/quarantine/ | medium | GET /ftp/quarantine/ | plain GET |
| Probe | Payload | What the app returned |
|---|---|---|
| Path traversal to /etc/passwd | ../../../etc/passwd | the Juice Shop frontend HTML, traversal is normalised away |
| XSS in product search | <script>alert(1)</script> | the string comes back escaped, no script executes |
| SQL injection in product search | ' OR 1=1-- | an empty result set, not the full table |
2.DVWA: same method, different answer sheet
DVWA at localhost:8081, security level low, logged in as admin/password. Eleven probes ran, nine reproduced and two did not. The two that did not are the interesting rows, because DVWA at this level is famous for both.
| Finding | Payload | Verified |
|---|---|---|
| SQL injection, error-based | 1' OR '1'='1 | yes |
| SQL injection, UNION-based | 1' UNION SELECT user,password FROM users-- - | yes |
| Reflected XSS | <script>alert("XSS")</script> | yes |
| Stored XSS in the guestbook | <script>alert("XSS_Stored")</script> | yes |
| Command injection | 127.0.0.1; id | yes |
| Local file inclusion | /etc/passwd via page= | yes |
| Unrestricted file upload | shell.php executed | yes |
| Brute force on the login form | admin:password | yes |
| IDOR by user id | id=1 | yes |
| CSRF on the password change | password_new=hacked | no, a token is required |
| Insecure CAPTCHA bypass | password change with no CAPTCHA | no, validation enforced |
3.What the attempt itself cost
The measurement belongs next to the findings, because that is where the time went. Across the cards for these two files, the scheduler spent 264 minutes of worker wall time to produce 16 minutes of results.
The rest breaks down by state. The model provider walled us off with rate limits for 163 minutes. Workers crashed for 45 minutes, one card waited 25 minutes on a repository another worker held, and 15 minutes went to locks reclaimed by hand.
| State | Minutes | What it means |
|---|---|---|
| rate_limited | 163 | provider quota wall; the run was requeued without counting a failure |
| crashed | 45 | the worker process exited without closing the card |
| blocked | 25 | one repository, one writer, a second worker waits |
| reclaimed | 15 | stale locks taken back before the next attempt |
| done | 16 | the two cards that produced the evidence below |
Provider rate-limit wall
163
Minutes of the 264 the scheduler spent here, the longest of the five states
Worker crashes
45
Processes that exited without closing their card, read from our own board database
Minutes that produced anything
16
Two cards of the 264; the evidence above came from these and strix contributed none
Two configuration faults in the attempt, kept because they cost real time: strix’s LLM_EXTRA_HEADERS was written as a string where the tool expects an object. One run also pointed strix at a paid model on the same key, which drained the account. We changed configuration only, with no patch to strix’s code, so nothing here is about its exploit logic.
4.What this does and does not say
- Do not repeat the no-false-positives claim
- strix never ran to completion here, so this article cannot test that claim in either direction.
- Three false positives in the evidence belong to our hand-run probes, not to strix
- Budget the agent’s own model credits
- strix calls a model between steps; on this account that failed before any exploit did, with HTTP 429.
- An autonomous pentester is a model workload as much as a security tool
- Keep a hand-run baseline
- The answer key was right: 15 of 20 probes reproduced in minutes with curl and python.
- Each row is a request a reader can re-run against their own local copy
- Pin a cheap model if you try it
- The drained credits came from a paid model on the same key.
- Point it at the cheapest model that can follow the loop before letting it loose
What we did not measure
- strix in autonomous mode: not one completed run, so no findings, no tokens and no wall time from the tool itself.
- Whether strix would have reported the same 15 findings, different ones, or any of our 3 false positives, which is the question this article set out to ask.
- The repair half of its claim: it also advertises fixing what it finds, and we ran it read-only.
- DVWA above security level low, and any target other than these two containers on this laptop.
- Token accounting for the agent, because the calls that would have produced it never succeeded.
One machine: MacBook Pro M1, 8 GiB, macOS 26.3, on 2026-10-07. Juice Shop and DVWA ran in Docker on this laptop; nothing outside localhost was touched. strix 1.7.0 and its sandbox image 1.3.0 are the versions installed here.