What is automated penetration testing, and what does it miss?

“Automated penetration testing” is attached to products that do very different jobs. A scanner that matches version banners against a CVE feed, a platform that drops an agent on your internal network and tries to reach domain admin, a tool that replays MITRE ATT&CK techniques to see whether your endpoint protection notices, and an LLM agent that logs into your web app and tries to read another tenant's invoices are all sold under the same label. It is easy to buy the wrong one, and easier still to read a clean report as meaning more than it does.

Start with what a penetration test is. The UK National Cyber Security Centre defines it as “a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system's security, using the same tools and techniques as an adversary might.” NIST's SP 800-115 definition adds the part that separates a test from a scan: most penetration tests involve “looking for combinations of vulnerabilities on a single system or multiple systems that can be used to gain more access than could be achieved through a single vulnerability.” Automated penetration testing, then, is software that attempts that work itself. This guide covers the four kinds of tool that use the name, what automation is good and bad at, where it fits next to manual testing and compliance requirements, and what to check before you trust one.

Where a scan ends and a test begins

The PCI Security Standards Council's penetration testing guidance draws the line by purpose. A vulnerability scan exists to “identify, rank, and report vulnerabilities that, if exploited, may result in” a compromise. A penetration test exists to “identify ways to exploit vulnerabilities to circumvent or defeat the security features of system components.” One produces a list of possibilities; the other tries them.

Most tools sit somewhere between the two, so it helps to place any product with three questions. Does it attempt exploitation, or stop at detection? Does it adapt, using what one step returned to decide the next one? And does its report show what it achieved (data read, code run, an account taken over) or only what might be possible? A tool that answers yes to all three is doing penetration testing. A tool that answers no to all three is a scanner, whatever the brochure calls it.

Four kinds of tool sold under the name

Vulnerability scanners and DAST

Network scanners such as Nessus and OpenVAS, and dynamic application scanners such as ZAP and Burp Scanner, send large numbers of known checks and match the responses. They are fast, broad and cheap to run often, and they are what PCI DSS calls scanning, not penetration testing. NIST's SP 800-115 warns that “vulnerability scanners can have a high false positive error rate (i.e., reporting vulnerabilities when none exist).” Nearly every product in the other three categories runs a scanner as its first step; the difference is what happens after a hit.

Breach and attack simulation

Breach and attack simulation (BAS) is, in AttackIQ's definition, “the process of using software to emulate adversary tactics, techniques, and behaviors” in an IT environment. Agents on endpoints and servers run scripted techniques, usually mapped to ATT&CK, and record whether your controls blocked, logged or missed them. MITRE's Caldera is the open-source reference point, described as “an open-source cybersecurity platform for automated adversary emulation” built on the ATT&CK framework. BAS answers “would we notice or stop this technique?” It measures your defences against known behaviour; it does not go looking for new vulnerabilities in your own code. That makes it valuable for detection engineering and no substitute for testing an application.

Automated network and infrastructure pentesting

Platforms such as Horizon3.ai's NodeZero and Pentera try to turn what they find into access: harvest credentials, reuse them, move laterally and escalate, often inside Active Directory. NodeZero's product page says internal tests run from a Docker host or virtual appliance and external tests from Horizon3's cloud, “chaining weaknesses to demonstrate impacts far beyond CVEs.” Pentera says its platform “is designed to run safely in live production environments.” This is where automated pentesting is most mature, and the PCI guidance explains why: because network protocols are well defined, “network-layer testing is more suitable for automated testing.”

AI agents for web application pentesting

The newest category puts a large language model in the loop. The agent reads responses, forms a hypothesis, writes a payload, calls a tool, and changes its plan when the result surprises it. That suits the parts of a web test that never fitted a signature: understanding a custom API, adapting a payload to a filter, noticing an ID in one response and trying it in another request.

The evidence so far cuts both ways. The PentestGPT paper (USENIX Security 2024) found LLMs proficient at sub-tasks such as “using testing tools, interpreting outputs, and proposing subsequent actions,” while they “encounter difficulties maintaining a whole context of the overall testing scenario.” In the field, XBOW reported in June 2025 that it had reached the top of HackerOne's US leaderboard after submitting nearly 1,060 vulnerabilities. Its own breakdown is worth reading: at the time, 130 were resolved and 303 triaged, while 208 were duplicates, 209 informative and 36 not applicable, and its security team reviewed every finding before submission to comply with HackerOne's policy on automated tools. Open-source agents in this category include Xalgorix (ours), Strix, Shannon and PentestGPT; our comparison pages set out how they differ.

What automation does well

Breadth and repetition come first. A tool tests every host, route and parameter the same way on every run, and it does not get bored on the four-hundredth endpoint. That makes automation the natural fit for frequency: a yearly manual test samples one version of an application that may ship daily, while an automated run can follow each release and catch the regression that reopened last quarter's bug.

Retesting is the quiet win. PCI DSS requirement 11.4.4 says that after fixes, “penetration testing is repeated to verify the corrections.” Re-running a specific exploit against a patched endpoint is tedious for a person and cheap for a machine.

Automation is also strongest where success can be observed without judgment: a payload that actually executes in a headless browser (the check XBOW describes for its XSS validator), an out-of-band callback carrying a value the target computed, a database version read back through an injection, a file pulled from disk. When the evidence is machine-checkable, a tool can confirm its own work. When it is not, someone has to decide.

What it still misses

Business logic

OWASP's Web Security Testing Guide is blunt about logic flaws: “This type of vulnerability cannot be detected by a vulnerability scanner and relies upon the skills and creativity of the penetration tester,” because “automated tools find it hard to understand context.” LLM agents loosen that, since reading context is what they do. The harder problem is that the rule being broken usually is not written anywhere in the application. Whether coupons may stack, whether a support role may see billing data, whether a refund may exceed the purchase: those rules live in policy documents and in people's heads. A tool can notice that a discount applied twice; calling it a bug requires knowing it shouldn't. If you want logic tested, give the tool the rules and a test account for each role, or expect it to miss them. The PCI guidance makes the same point for human testers, who should test “from the perspective of the defined roles of the application” with credentials the organization supplies.

Long chains

NIST's definition puts combinations of vulnerabilities at the centre of a penetration test. Network platforms chain well because the moves are standard: a password hash leads to a host, the host to a domain credential. Web chains are less regular. An information leak that reveals an ID format, an access-control gap that accepts the ID, and a password-reset flow that trusts the result can sit in three different parts of an app and be found days apart. That is the kind of “whole context” the PentestGPT authors found models struggle to keep. Expect tools to find individual links and sometimes short chains; long chains across roles and sessions are still where experienced testers earn their fee.

Scope and safety

SP 800-115 says systems “may be damaged or otherwise rendered inoperable during the course of penetration testing,” and that while experienced testers can mitigate the risk, “it can never be fully eliminated.” Automation removes the pause in which a person thinks “this endpoint deletes accounts” or “this form emails real customers.” Agents add a newer risk: improvising their way past the boundary they were given, which we looked at in when an AI pentester escapes the lab. Scope has to be enforced by the tool and the network around it, not by a sentence in a prompt.

Silence

A report with no findings means nothing was found in what was tested. Pages the crawler never reached, flows behind a login the tool was not given, and a WAF that starts blocking halfway through all shrink what “tested” covers. That applies to Xalgorix as much as any tool: a target that blocks the scanner mid-run cuts the run short.

How it fits next to manual tests and compliance

The NCSC's framing is useful here. It says penetration testing “should be viewed as a method for gaining assurance in your organisation's vulnerability assessment and management processes, not as a primary method for identifying vulnerabilities,” and compares it to a financial audit. Day-to-day bookkeeping is where automation has always lived. Good agents now reach into work the audit used to own, but someone still has to sign the audit.

PCI DSS spells that out. In version 4.0, as printed in the council's SAQ D, requirement 11.3 covers vulnerability scans every three months (external ones by an Approved Scanning Vendor), and 11.4 covers penetration testing. The methodology must include application-layer testing for, at a minimum, the attacks listed in requirement 6.2.4, a list that names “attacks on business logic,” plus network-layer testing. Internal and external tests happen “at least once every 12 months” and after any significant change, “by a qualified internal resource or qualified external third-party,” with “organizational independence of the tester.” The current v4.0.1 was a limited revision that, in the council's words, has “no additional or deleted requirements.”

The council's 2017 guidance goes further: “Penetration testing is essentially a manual endeavor,” and “simply running an automated tool does not satisfy the penetration testing requirement.” That supplement predates v4.0 and today's AI agents, and it is guidance rather than a requirement, but the requirement text points the same way: a qualified, independent person owns the methodology and the interpretation. Automated output can feed that test. Check with your assessor before relying on a tool's output alone.

SOC 2 prescribes no specific test. In the AICPA's 2017 Trust Services Criteria, penetration testing appears as an example in a point of focus under CC4.1 (management uses “a variety of different types of ongoing and separate evaluations, including penetration testing”), and vulnerability scanning is a point of focus under CC7.1. The same document says use of the criteria “does not require an assessment of whether each point of focus is addressed.” The Cloud Security Alliance's summary is that a penetration test “isn't required to complete a SOC 2 report, but it is highly advisable.” Whether an automated test satisfies your auditor, and the customers reading your report, is a question to settle with them early.

In practice, the two kinds of testing fit together like this:

Point the human-led test at the work automation is weakest at: business logic, multi-role authorization and long chains. Rank the fixes by whether a finding was exploited, not by its score alone; we covered why in ranking vulnerabilities by exploitability.

How to evaluate an automated pentesting tool

Ask for evidence, not detections

For every finding, you should see the exact request, the response, and what it achieved: the row read, the command output, the callback, the other user's data. “Possible” and “confirmed” should be labelled differently, and a version banner match should never be reported as an exploited vulnerability. Our sample reports show the format we hold ourselves to, and we have written about why a detection alone falls short. Xalgorix runs a 22-phase methodology, exploits what it finds, then has an independent verifier re-test each finding: findings it disproves are dropped, and ones it cannot reproduce are flagged for review rather than reported as confirmed.

Ask how independent that verification really is. SP 800-115 notes that checking one tool's results with a second tool is quick, but “these comparison tools can often produce similar results—including the same false positives.” An automated verifier, ours included, can share blind spots with the system it checks. Treat anything flagged as inconclusive as work for a person.

Measure noise and misses on targets you know

Vendor demos run against targets the vendor chose. Run the tool against something where you already know the answers. The OWASP Benchmark provides a test suite of “actually exploitable” cases for SAST, DAST and IAST tools; deliberately vulnerable apps help too. The best test is a staging copy of your own application with bugs you have already found or planted, because it has your auth, your framework and your quirks. Count the misses as carefully as the false positives. And look at what reaches triage, not just headline counts: in XBOW's numbers above, duplicates and informative reports together came to about 40% of submissions.

Check the safety controls

• Scope enforced by the tool: hosts, ports and paths it will not leave, plus exclusions for endpoints that delete, pay, or message real users.
• Rate limits and automatic backoff when the target starts returning errors, and a kill switch you can reach mid-run.
• A complete log of every action. SP 800-115 recommends “mechanisms for automatically logging assessor actions,” and that matters more when the assessor is software.
• For agents: restricted network egress, and no access to secrets beyond the test accounts you hand over.

Look for coverage you can see

Ask whether it tests authenticated, with more than one role; whether it handles APIs, GraphQL and JavaScript-heavy front ends; and whether the report says what was not tested and why. A tool that cannot tell you what it skipped cannot tell you what a clean result means.

Know where your data goes

A pentesting tool sees your application's responses, which can include customer data. With an LLM agent, those responses also pass through a model provider unless you run the model yourself. If that matters, look for a self-hostable option: the Xalgorix engine is open source under Apache-2.0 and runs on your own infrastructure with your own LLM provider key, or with local models through Ollama.

Read the pricing model, not just the price

Tools are priced per asset or IP, per seat, per test, or per scan, and self-hosted agents add the model tokens you pay for directly. Ask what counts as a unit, whether retests cost extra, what happens when a run fails, and how cost grows on a large target. For reference, Xalgorix Cloud uses credits: a scan uses 1 credit, up to 2 on heavy targets, and 10 credits cost $10. Self-hosting is free to run, but the tokens are not, as we worked through in what self-hosting an agentic pentester costs.