A field report on using Qwen3.8-Max (via the Qwen Code CLI) as an autonomous blackbox web-app pentester – including the moment it started testing things I didn’t tell it to, and what that taught me about running these tools safely.
The setup
The brief I gave it was one sentence: pentest this internal web app, blackbox, dev environment, I’m already on the VPN. That’s it. No wordlists, no scope document, no list of tools to use. I wanted to see what a general-purpose coding agent – not a purpose-built pentest tool, just Qwen3.8-Max running in the Qwen Code CLI with shell access, file access, and a todo-list tool – would actually do with that.
The target (I’ll call it TARGET-01 here – real IP and app branding redacted, since it’s a real client system with real findings) was a FastAPI app on a Windows box, sitting behind a client-side rule that said: full port scan first, but only active testing on port 80. Everything else stayed off-limits.
What follows is reconstructed from the full session transcript – every command it ran, every file it wrote, every piece of internal reasoning it logged before acting. I didn’t watch it live; I read the receipts afterward. That turned out to be the interesting part.
Act 1: The environment fights back
Before any actual hacking happened, the agent spent its first few minutes just trying to get curl to run. It typed curl -sS -D - ... and got:
'curl' is not recognized as an internal or external commandSame for where, same for powershell. That’s odd, because echo %PATH% clearly showed C:\Windows\system32 – right where those binaries normally live – already on the path. Most tools would either retry the same command a few times or just give up and tell the user “curl isn’t installed.” This one didn’t do either. It reasoned through it out loud, essentially: the PATH looks fine, so maybe the files aren’t actually missing – maybe bare-name resolution itself is broken in this shell. Then it tested that specific theory: dir C:\Windows\System32\curl.exe. The file was right there. So it tried invoking it by full path instead – C:\Windows\System32\curl.exe --version – and that worked.
From that point on, every single shell command in the session used a fully-qualified path. It didn’t rediscover the problem five separate times; it fixed it once and generalized.
It also discovered there was no nmap and no python anywhere on the box. A lot of “AI pentester” demos quietly assume Kali Linux with a full tool belt. This was a bare Windows dev machine. So the agent just… wrote its own tools. A raw TCP port scanner in Node.js (since Node was on PATH). A hand-rolled MySQL greeting-packet parser to grab a version banner off port 3306 without a MySQL client installed. A timing-instrumented HTTP client for testing login latency. None of these are exotic – any competent tester would write the same 30-line scripts – but the agent didn’t ask permission to improvise, it just did, and kept a clean paper trail of what it wrote and why.
Act 2: The scan comes back bigger than expected
The full port sweep (allowed, per scope) returned eight open ports: 80 the actual web app and then 135, 139, 445 (SMB/RPC), 3306 (MySQL, directly reachable from outside), 3389 (RDP), and two unidentified HTTP services on 5000 and 8080.
Its own internal note at this point read almost like a human tester rubbing their hands together:
“This is a Windows server… 3306 — MySQL directly exposed! 3389 — RDP.”
It queued up next steps: grab a banner off MySQL, probe the two mystery HTTP ports, keep going with the username-timing test it already had planned.
And this is where I stepped in. Those three tool calls MySQL banner grab, port 5000 probe, port 8080 probe – were sitting in the queue, about to run. I sent one line:
“Focus only port 80.”
They never executed. The transcript shows them cancelled before running – zero packets sent to that out-of-scope surface. The agent immediately rewrote its own todo list and a persistent project note to bake in the restriction, and for the rest of the engagement it didn’t touch those ports again, not even implicitly its final report just flagged them as “not tested per client instruction, worth reviewing separately.”
Here’s the thing I keep coming back to: from a pure “find more bugs” standpoint, what it was about to do was completely reasonable. A real pentester who just found exposed MySQL and RDP on a target would absolutely want to poke at them. The instinct was good. It just wasn’t authorized, and the agent had no way to know that on its own – it only knew because I was watching the queue and said something before the commands fired. If I’d stepped away from the keyboard at that exact moment, or if the harness had been set to auto-approve everything, that out-of-scope traffic goes out. That’s not a knock on the model’s judgment about what’s interesting to test. It’s a reminder that scope enforcement can’t live in the model’s head – it needs to be a hard boundary in the tooling around it (an approval gate, a network allow-list, something a human or a system controls, not something the model has to remember to respect on its own).
Act 3: Finding the real bug — carefully
Back on port 80, the most interesting find wasn’t the loudest one, it was the quietest: a timing side-channel on the login endpoint. The error message for a bad login was completely generic – no “user not found” vs. “wrong password” distinction anywhere in the visible response. But the agent noticed the response times weren’t generic at all.
It didn’t jump to a conclusion off one slow request. First it ran eight alternating rounds – real username candidate, then a made-up one, back and forth – and got a clean split: nonexistent usernames landed at 120–160ms, every single time, while one specific username consistently took 415–435ms. Then, instead of declaring victory, it wrote a proper 44-candidate script, throttled to a deliberate ~180ms between requests (not a burst – it wasn’t trying to trip a rate limiter, it was trying to be a careful statistician), and ran the whole list. Across more than 50 timed requests total, exactly one username fell in the “slow” bucket. Zero overlap between the two clusters. That’s a textbook timing oracle – the backend is almost certainly only doing password-hash verification when the username actually exists, and that hash comparison is what’s showing up as the extra ~300ms.
It also fuzzed the same login endpoint with garbage-typed payloads – arrays where strings were expected, null, integers, and a {"$ne": null} payload specifically shaped to test for a NoSQL-injection auth bypass. That last one is a nice detail: it got a 500 error just like the other malformed payloads, and the agent explicitly checked that a 500 is not the same thing as “logged in as admin” before writing it up as “unhandled exception from bad input validation” rather than “NoSQL injection bypass.” Two bugs that would look identical from the outside (both come back as some kind of error) got correctly told apart.
One more small thing worth mentioning because it’s exactly the kind of step a rushed human tester skips: an admin endpoint returned an HTTP 307 redirect when hit with a trailing slash, which can sometimes mean a routing rule is stripping the auth check on the “wrong” version of a path. Instead of treating the 307 itself as a red flag, the agent followed the redirect (curl -L) and confirmed the endpoint it actually landed on still demanded auth. No bypass. Boring, correct, and the kind of thing that’s easy to hand-wave past when you’re moving fast.
What it actually found
Nine things, in short:
- Full API map exposed to anyone: unauthenticated, the auto-generated docs endpoint handed over all 217 routes, including every admin function, on a single request.
- No rate limiting on login at all: 100+ straight failed attempts against a known-valid account, zero throttling, zero lockout.
- The timing oracle above: confirming exactly one valid username out of 44 candidates tested.
- An unauthenticated device webhook: that accepted arbitrary JSON with no signature check – anyone reaching the app could inject fake device events.
- CORS wildcard: on API responses.
- No security headers, no TLS at all.
- Type-confusion crashes: on the login endpoint (500s instead of clean validation errors).
- Minor info leaks: on two unauthenticated endpoints (health/status details).
- An OTP step it couldn’t finish testing: flagged honestly as “unverified, needs valid credentials” rather than guessed at.
And just as important – what didn’t break:
Every one of the 100+ protected endpoints it hammered with path tricks, verb tampering, and header spoofing stayed locked behind auth. No SQL injection anywhere. No open redirects. The OAuth login flow correctly rejected a forged callback. No weak or default password got in among roughly 60 tried.
That “held up” list matters as much as the findings list – it’s the difference between a report you can trust and one that just lists scary-sounding things.
The numbers, because they’re honestly the most surprising part
That’s a full recon-to-report cycle, including writing custom tooling from scratch because the standard tools weren’t installed, from one sentence of direction – the opening brief, plus the single “focus only port 80” correction, with 796 lines of scripts and report content written along the way.
A cheaper alternative to Claude and ChatGPT – and a genuinely usable one
I’ll say the quiet part out loud: I went into this expecting a toy. What came back was a real findings report – correctly severity-ranked, methodologically sound, with a statistically validated timing oracle I’d be happy to see in a report from a human tester. I’ve used Claude and ChatGPT-class models for adjacent security work, and on this task Qwen3.8-Max held its own against them while costing noticeably less to run. And I mean noticeably: the entire engagement – recon, tooling improvisation, the timing-oracle statistics, access-control fuzzing across 100+ endpoints, and the final report – ran on the Individual Lite token plan, $8 USD. That’s not a per-engagement rate card, that’s a monthly subscription tier, and it covered this with room to spare. For a security team or a solo consultant deciding where to spend a model budget, that’s not a footnote, that’s the headline: you no longer need the most expensive frontier model on the market – or even a meaningful budget line – to get a competent first-pass pentest out of an agentic CLI.
That said – “competent output” and “competent unsupervised operation” are two different claims, and this transcript only supports the first one. Every good decision the model made was still operating inside boundaries a human set: what target, what scope, and – critically – the one moment (Act 2, above) where I had to physically say “stop, port 80 only” before it wandered into MySQL and RDP. Hand this tool to someone with genuine offensive-security judgment, and you get a fast, cheap, well-documented first pass. Hand it to someone without that judgment, and you get the same fast, cheap tool with nobody catching the moment it should have stopped. The model supplies capability; it does not supply scope discipline or the judgment to know when a finding is actually severe versus merely noisy. That part is still on the human.
You don’t need Kali. You don’t even need to know what Kali is.
Here’s the detail that actually unsettled me a little: this ran on a stock Windows dev box with no nmap, no Python, not even a working where command. A real pentester’s instinct in that situation is “install my toolkit first.” This agent’s instinct was “write a 30-line Node.js script that does the one thing I need instead” – and it did that three separate times (a port scanner, a MySQL banner grabber, a timing-instrumented HTTP client) without ever being asked to.
That means the operator didn’t need Kali, didn’t need a security-tooling background, and didn’t need to know what a MySQL greeting packet looks like. They needed to know how to type a sentence and, later, how to type a second sentence telling it to stay on port 80. That’s the whole skill floor for getting a genuinely usable vulnerability assessment out of this. A technically competent Windows user – a sysadmin, a developer, an IT generalist with no offensive-security background at all – can point this at a target and walk away with real findings, including ones (like the timing oracle) that most people wouldn’t think to test for manually.
Why that should worry us a little
Lower the skill floor for “run a competent recon-to-report pentest” this far, and you don’t just get more defenders doing more self-assessment. You also get more people who have no business testing anyone’s systems doing exactly that – cheaply, quickly, and with a tool that writes its own scripts when the built-in ones are missing. The gap between “curious person with a Windows laptop” and “someone capable of mapping 217 endpoints and finding a real auth weakness” just got a lot smaller – and it got there for $8 a month. I’d expect this to translate directly into more opportunistic, low-skill attack activity in the wild – more script kiddies, except the “script” now writes and adapts itself, and the entry fee is less than a movie ticket. That’s not a reason to avoid these tools; it’s a reason to expect the baseline level of unsophisticated attack traffic against anyone with an internet-facing footprint to keep climbing.
Which is exactly why pentesting-as-a-service isn’t going anywhere
That’s the part I keep coming back to. If the cost and skill barrier to attempting a competent-looking intrusion just dropped this much – an $8 plan, a stock Windows box, one sentence of direction – the “we’re too small to be a target” assumption that a lot of startups and SMEs still operate on gets weaker, not stronger. It used to take real budget and real expertise to seriously probe a company’s perimeter. This transcript is evidence that it now takes one sentence, a Windows laptop, and an $8 model subscription to get most of the way there.
That doesn’t make professional, authorized penetration testing less relevant – it makes it more necessary, especially for smaller organizations that could never justify an enterprise-grade security budget but are now squarely inside the reach of cheap, capable, semi-autonomous tooling. The value a security firm brings isn’t “we have access to an LLM” anymore – clearly, anyone can get that. It’s the judgment layer this whole case study kept coming back to: knowing what’s in scope, knowing which finding actually matters, knowing when to stop. That’s still a services business, and if anything, this makes the case for it stronger than it was a year ago.
Full findings report, redacted, is filed separately with the client. This piece describes methodology and process, not exploit details.
Red Team Specialist – SecStrike.ai