AI pentesting vs DAST vs vulnerability scanners, and what each one misses

The OWASP Top 10:2025 keeps Broken Access Control at number one and reports that 100% of the applications tested had some form of it. Meanwhile ZAP, the open-source DAST tool, says in its own documentation that “logical vulnerabilities, such as broken access control, will not be found by any active or automated vulnerability scanning.” Most of the current argument about AI pentesting follows from those two facts. The most common serious bug class sits in the blind spot of the tools most teams automate with, and LLM agents are being sold as the thing that finally reaches it.

Whether they do, and at what cost, comes down to a detail most vendor comparisons skip: how each kind of tool decides that a response means “vulnerable.” A signature scanner matches what it sees against known fingerprints. DAST sends attack payloads and reads the reaction through generic, per-request checks. An agent forms a hypothesis about the application, tries to exploit it, and then judges the evidence. That decision rule, the tool's oracle, determines what each one finds and misses, how often it is wrong, and how safely you can run it. This comparison covers all three with sources, including the places where the older tools still win.

Three ways to decide that something is a bug

Signature and rule-based vulnerability scanners

NIST's testing guide, SP 800-115, states the mechanism plainly. The target's “behaviors and outputs in response to attack patterns submitted by the scanner are compared against those that characterize the signatures of known vulnerabilities, and the tool reports any matches that are found.” Network vulnerability scanners and template engines such as ProjectDiscovery's Nuclei work this way. Each check is a request plus a matcher, and Nuclei's documentation describes matchers as “flexible comparisons on protocol responses.” A typical check looks like this (an illustrative template, not one copied from the public library):

id: exposed-git-config
info:
  name: Exposed .git/config
  severity: medium
http:
  - method: GET
    path:
      - "{{BaseURL}}/.git/config"
    matchers-condition: and
    matchers:
      - type: status
        status: [200]
      - type: word
        words: ["[core]"]

The strengths come straight from that design. Checks are cheap, so a scanner can sweep thousands of hosts quickly. The same templates give the same answer on the same target. When a new CVE gets a signature, every user of the tool can check for it at once. The limits come from the same place. A scanner knows only what is in its repository, which NIST notes “requires the assessors to update these signatures frequently.” Version inference also misfires: a banner can say a package is old when the distribution has already backported the fix. The same guide warns that application scanners “typically have high false positive rates” and suffer from “high false negative rates” too. A scanner reports what matches and says nothing about anything else.

DAST: crawl, attack, read the response

OWASP describes DAST tools as scanners that test web applications “normally from the outside” for flaws such as cross-site scripting, SQL injection, command injection and path traversal. Rather than matching known products, DAST crawls the application, finds its inputs and mutates them. PortSwigger's documentation splits Burp Scanner's audit into three phases: a passive phase that only watches traffic, an active phase that “inserts modified input into requests,” and a JavaScript analysis phase for DOM-based bugs. Every active check needs an observable effect, such as a database error, a reflected marker, a timing difference, or an out-of-band DNS or HTTP callback for bugs that leave nothing in the response. Our guide to out-of-band callbacks covers how to read that last kind of signal.

Calling DAST “detection only” undersells it. Burp's intrusive checks “attempt to confirm whether critical vulnerabilities, such as SQL injection, can be exploited,” and its dynamic JavaScript analysis runs payloads in a headless browser to see whether they reach a dangerous sink. What DAST lacks is a generic oracle for intent. It can tell that a payload changed a response. It cannot know that user B should never see invoice 1043, or that a coupon must apply only once per order. Those are facts about the business, and HTTP doesn't encode them.

AI pentesting agents: hypothesis, exploit, judgement

An agent-based pentester wraps a language model in a loop with tools: an HTTP client or browser, a shell, recon utilities. It reads what it finds and forms a hypothesis (“this export endpoint takes an account ID, so try another tenant's”). Then it acts, reads the result and decides what to try next. Because it carries state across steps, it can work through multi-request flows that a per-request scanner has no way to express.

Published research supports both the capability and its limits. In a 2024 study, Fang et al. gave a GPT-4 agent a browser and six web-security documents. Allowed five attempts per vulnerability, it exploited 11 of 15 vulnerabilities planted in sandboxed websites without being told which flaw was present, including a SQL union attack that took 38 actions to extract the schema and then the data. Every open-source model the authors tested scored 0%. The PentestGPT paper at USENIX Security 2024 found that models are proficient at sub-tasks such as running tools and interpreting their output, but “encounter difficulties maintaining a whole context of the overall testing scenario.” In an agent, the oracle is the model's own reading of the evidence. That gives agents their reach, and it is also the source of their new failure modes.

AI pentesting vs DAST vs vulnerability scanners, side by side

The table summarizes typical behaviour. Individual products vary, and the categories are starting to overlap: some DAST vendors now ship multi-user authorization checks, and an agent can call a conventional scanner as one of its tools.

What each one finds, and what slips past it

Known vulnerabilities and misconfigurations

Scanners win here, and it isn't close. An agent can spot an outdated component too, but paying a model to reason its way to something a template finds with one request is a waste of tokens. A sensible agent hands that part of the job to a conventional scanner and spends its budget on what comes after.

Injection with a clear signal

Reflected XSS, error-based and time-based SQL injection, and command injection that shows up in a callback all have observable effects, so DAST finds them systematically and repeatably at low cost. Agents find them too, at a higher price per run. Where an agent adds something is in adapting a payload to a filter it has just watched block the first attempt, and in following a hit through to impact, such as extracting data rather than stopping at a suspicious delay.

Access control: IDOR and BOLA

Authorization bugs need two identities and a notion of who should see what. One DAST vendor's own documentation shows why. StackHawk's business logic testing docs ask you to “configure two profiles representing different authorization levels” and to supply an OpenAPI spec. They also explain that earlier single-identity authorization rules were removed because “those rules couldn't confirm an authorization flaw and raised false HIGH alerts.” The lesson applies to every tool. Without a second identity, nothing can tell a legitimate 200 from a leak.

Give an agent two sessions and it can request account A's objects as account B, compare what comes back, and reason about which fields actually matter. It can also follow object references that a spec never lists, such as IDs in a JavaScript bundle, an export job or a WebSocket message. It still needs those test accounts. With one login, an agent is guessing too. Our IDOR and BOLA testing guide walks through the two-account method in detail.

Business logic and chained issues

The OWASP Web Security Testing Guide says business logic flaws “cannot be detected by a vulnerability scanner” and that “automation of business logic abuse cases is not possible and remains a manual art.” That was written before LLM agents. Agents have made a dent in it: an agent can read a checkout flow and try applying a coupon twice or skipping the payment step. They haven't closed the gap, though. PortSwigger notes that finding logic flaws often requires “an understanding of the business domain”, and an agent has that only if you provide it, through documentation, role descriptions and test accounts.

Chains are the same story. NIST says scanners “are unable to detect vulnerabilities that are revealed only as the result of potentially unending combinations of attack patterns,” while penetration tests look for “combinations of vulnerabilities on one or more systems that can be used to gain more access than could be achieved through a single vulnerability.” Agents can attempt chains, such as an SSRF that leads to cloud metadata credentials, but benchmarks keep expectations in check. In CVE-Bench, published in 2025, the best agent framework tested could exploit “up to 13%” of critical-severity real-world web application CVEs in its sandbox. A 2025 study on a live university network of about 8,000 hosts reported a stronger result: the authors' agent, ARTEMIS, placed second overall against ten professional penetration testers, outperforming nine of them. The same study found that AI agents struggled with GUI-based tasks.

Blind bugs

DAST and agents both rely on timing and out-of-band callbacks for bugs that never show up in a response. PortSwigger lists blind SQL injection, external service interaction and out-of-band resource loads among the issues Burp Collaborator is used to detect, and a scanner can plant those payloads in every insertion point it audits. Agents are selective: they chase leads. For breadth, systematic beats selective. Selectivity earns its keep when it comes to turning a weak callback into demonstrated impact.

How false positives change from one tool to the next

Each category has its own kind of noise. Scanners misread version strings and banners. DAST flags heuristic matches and can pad reports with low-severity header findings, although mature tools grade confidence; Burp, for example, rates static-only JavaScript findings lower than those confirmed dynamically.

Agents add a newer kind of false positive: the model asserting an impact that its evidence doesn't show, such as reading a 500 as SQL injection or an echoed string as code execution. In the network study above, the authors report that AI agents “exhibit higher false-positive rates” than the human testers, and ARTEMIS's own valid submission rate was 82%. An agent's finding should therefore be treated as a claim until its evidence has been reproduced.

That is the reason to require exploitation evidence and an independent re-test. Xalgorix exploits each finding, then has an independent verifier re-test it from scratch. The verifier drops findings it disproves and flags inconclusive ones for human review, and we've written up how that pipeline works. Verification has a clear boundary, though. It cuts false positives, but it does nothing about bugs the agent never tried.

Speed, cost and repeatability

Once set up, scanners and DAST are cheap to run. ZAP's baseline scan spiders for one minute by default and takes “a few minutes at most.” Its full scan performs real attacks with no spider time limit by default and “can potentially run for a long period of time.”

Agents pay per token. Fang et al. estimated about $9.81 per website attempt, failures included, at 2024 GPT-4 prices, against an estimated $80 for the same effort from a human. The ARTEMIS authors put some of their agent's configurations at $18 an hour, against $60 an hour for professional penetration testers. These figures come from specific research setups, and model prices change, so treat them as orders of magnitude. On Xalgorix Cloud, a scan costs 1 credit, which covers up to 40 million tokens of AI analysis, and never more than 2 credits on heavy targets; 10 credits cost $10. If you self-host the open-source engine, you pay your LLM provider directly, and token use varies a lot by target. Our self-hosting cost breakdown has the details.

Repeatability is the subtler issue. Fang et al.'s agent succeeded on 42.7% of single attempts but on 73.3% of vulnerabilities when given five tries, with the same agent and the same targets. Run an agent twice and you may get different findings. That is fine for exploration and poor for a regression gate. When a fixed bug comes back, you want a check that fails the build every time, and DAST rules and scanner templates give you exactly that. ZAP, for instance, lets you set each rule to FAIL, WARN or IGNORE. When an agent finds something real, turn it into a deterministic test so that it stays found.

Where each fits in a CI/CD pipeline

• Every pull request: a passive DAST baseline, which ZAP describes as “ideal to run in a CI/CD environment, even against production sites,” plus static and dependency checks. A diff-scoped AI review is cheap enough for this stage too. The free Xalgorix GitHub App reviews the changed code on each PR. It reasons about the diff and doesn't exploit anything.
• Nightly, or on each deploy to staging: full active DAST, plus signature scans of the infrastructure.
• Before releases and after changes to auth, tenancy or payments: an agent run against staging with real test accounts. The Xalgorix GitHub Action can start one from CI and fail the build only on findings it reproduced, at the severity you choose. That works as a release gate, but it is too slow and too variable to run on every commit.
• On a regular cadence: human testers, for scoping, business-risk judgement and the engagements compliance requires.

Running them against production

Scanners are the gentlest, but not harmless. NIST notes that network vulnerability scanning generates significantly more traffic than port scanning and that built-in denial-of-service tests “can have a marked negative impact on scanned hosts” when an inexperienced assessor runs them, so switch those tests off.

DAST's active checks are attacks by design. ZAP puts it bluntly: “Active scanning is an attack on those targets.” Burp's documentation warns that medium-intensity checks “may trigger security alerts, log users out, or cause other unintended effects,” and that intrusive checks “may cause permanent data loss, service outages, or other irreversible impacts.” StackHawk tells users to run its multi-user scans against staging and “never against production systems with real customer data.”

Agents add a different risk: the tool chooses its own next action. The guardrails therefore have to live outside the model: scope allowlists enforced at the network layer, rules against destructive payloads, rate limits and a kill switch. Our analysis of AI pentesters that reached systems outside their lab explains why. NIST's line on penetration testing applies here too: the risk can be mitigated but “can never be fully eliminated.” Xalgorix limits itself to non-destructive payloads, backs off on bursts of 5xx errors and has a kill switch on the dashboard, and our advice is still to prefer staging with production-like data. If you must test production, use dedicated test accounts and tell the people who watch the alerts.

Choosing the mix

These tools complement each other, and the right mix depends on what you are protecting:

• Many hosts running off-the-shelf software, where most of the risk is known CVEs: start with a signature scanner.
• A web app that ships weekly through CI: add DAST as the regression net.
• Multi-tenant SaaS, APIs keyed by object IDs, or workflows that move money: add agent-based testing with real test identities on a regular cadence, and turn what it finds into regression tests.
• High-stakes logic or pentests required for compliance: hire human testers, and let automation clear the breadth first.

Xalgorix sits in the agent column. The engine is open source under Apache-2.0 and can be self-hosted with your own LLM key or with local models through Ollama. It works through a 22-phase methodology and re-tests what it exploits before reporting it. You can see what that output looks like in the sample report, and see how it stacks up against other agents on the comparison page. It shares the limits described above: it costs more per run than DAST, results vary between runs, it can miss what it never thinks to try, and it needs test accounts before it can say anything useful about access control. Run it alongside your scanner and your DAST pipeline.