← Blog
November 19, 202610 min readAppSecSecurity EngineeringMethodologyAccess Control

How to turn a fixed vulnerability into a regression test that fails CI

Log4j’s 2.15.0 fix was incomplete four days after Log4Shell, and Craft CMS shipped a bypass of its own SSRF fix in 2026. A hands-on guide to security regression tests: replaying the report, widening it to the invariant, guarding the route list, and a Nuclei check against the preview build, including why Nuclei’s exit code alone will pass a CI job when the bug is back or the target is down.

By The Xalgorix Team

Log4Shell, CVE-2021-44228, was published on 2021-12-10, and Log4j 2.15.0 was the fix. Four days later the project published CVE-2021-45046, whose description opens: “It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations.” Lookup patterns that arrived through the Thread Context Map could still reach JNDI, and 2.16.0 removed message lookups and disabled JNDI by default.

It is not a one-off. On 2026-01-05 Craft CMS published an SSRF in its GraphQL asset mutation as CVE-2025-68437; on 2026-02-24 came CVE-2026-27127, described in its CVE record as “a bypass of the security fix for CVE-2025-68437” through DNS rebinding. Our ruby-saml teardown covers a run of signature-bypass CVEs in one library between 2024 and 2025. A security fix is a claim about a whole class of inputs, and a patch often covers only the input in the report.

The durable answer is a test that replays the attack and fails the build when the bug comes back. This guide builds one in two layers on a small lab app: a unit-level test that runs in milliseconds with the rest of your suite, and a Nuclei check that runs against the deployed preview. We ran everything below on 2026-11-19 (Python 3.13, Flask 3.1.3, pytest 9.1.1, Nuclei v3.11.1), including one Nuclei behaviour that turns a naive CI job green when the target is down.

The lab: one invoice IDOR and three builds

The app serves invoices to authenticated users. Each invoice has two routes: JSON at /api/invoices/<id> and a PDF export at /api/invoices/<id>/pdf. The bug is a textbook broken object level authorization (API1:2023): Bob's token can read Alice's invoice by changing the id. A MODE setting produces three builds of the same code:

  • vuln: no ownership check anywhere.
  • partial: the developer added the check to the JSON route named in the report. The PDF route still loads the invoice without it. This is the incomplete fix.
  • fixed: both routes go through one loader that checks ownership and returns 404 to anyone else.
def load_invoice(invoice_id, check_owner):
    inv = INVOICES.get(invoice_id)
    if inv is None or (check_owner and inv["owner"] != g.user):
        abort(404)
    return inv

@app.get("/api/invoices/<int:invoice_id>")
def get_invoice(invoice_id):
    return jsonify(load_invoice(invoice_id, check_owner=MODE in ("partial", "fixed")))

@app.get("/api/invoices/<int:invoice_id>/pdf")
def invoice_pdf(invoice_id):
    inv = load_invoice(invoice_id, check_owner=MODE == "fixed")
    return f"%PDF-1.7 invoice {inv['id']} {inv['memo']}", 200, {"Content-Type": "application/pdf"}

Alice's seeded invoice carries the memo ALICE-PRIVATE-101. That marker matters later: a security test needs something that can only appear in the response if the attacker really read the victim's data, rather than a status code that a login page or an error handler could also return.

Layer one: replay the report, then widen it to the invariant

The natural first test is the request from the report, word for word. Write it, because it documents the bug. Then write the test that states what the fix is supposed to guarantee: no route that renders an invoice serves it to someone who does not own it.

import pytest
from app import app

ALICE = {"Authorization": "Bearer alice-token"}
BOB = {"Authorization": "Bearer bob-token"}

@pytest.fixture
def client():
    return app.test_client()

# 1. The narrow test: replays the exact request from the report.
def test_report_replay_bob_cannot_read_alice_invoice(client):
    resp = client.get("/api/invoices/101", headers=BOB)
    assert resp.status_code == 404
    assert b"ALICE-PRIVATE-101" not in resp.data

# 2. The invariant: no route that renders an invoice serves it to a non-owner.
INVOICE_ROUTES = ["/api/invoices/{id}", "/api/invoices/{id}/pdf"]

@pytest.mark.parametrize("route", INVOICE_ROUTES)
def test_non_owner_gets_nothing(client, route):
    resp = client.get(route.format(id=101), headers=BOB)
    assert resp.status_code in (403, 404)
    assert b"ALICE-PRIVATE-101" not in resp.data

# 3. Positive control: the owner still can, so the test cannot pass by breaking the feature.
@pytest.mark.parametrize("route", INVOICE_ROUTES)
def test_owner_still_reads_own_invoice(client, route):
    resp = client.get(route.format(id=101), headers=ALICE)
    assert resp.status_code == 200
    assert b"ALICE-PRIVATE-101" in resp.data

Running the same file against the three builds:

MODE=vuln     3 failed, 2 passed   (replay + both invariant cases fail)
MODE=partial  1 failed, 4 passed   (only test_non_owner_gets_nothing[.../pdf] fails)
MODE=fixed    5 passed

The replay test passes against the partial fix. If it were the only test, the incomplete patch would ship with a green build and a regression test that looked like it was guarding something. The parametrised invariant is what catches the PDF route, and it is the same move that would have mattered for Log4j and Craft: the fix covered the input in the report, and the bypass came through a neighbouring input to the same sink.

The positive control looks redundant until someone “fixes” a flaky test by returning 404 for every invoice. Without it, a broken feature and a secure feature produce the same green run.

Bob requests Alice's invoice 101
vuln build
Replay fails, invariant fails on JSON and PDF
partial build
Replay passes; invariant fails on /pdf only
fixed build
All tests pass, owner control still reads both routes
The report replay only sees the route in the report. The invariant test enumerates every route that renders the object, so it fails on the partial fix.

Keep the route list from going stale

The invariant is only as good as INVOICE_ROUTES. Next quarter someone adds a CSV export, forgets the shared loader, and the list does not know the route exists. You can make the framework tell you. Flask exposes its routing table as app.url_map, so a fourth test can require every route with an invoice id to be listed:

# 4. Coverage guard: a new invoice route fails the build until it joins INVOICE_ROUTES.
def test_every_invoice_route_is_covered():
    registered = {r.rule for r in app.url_map.iter_rules() if "<int:invoice_id>" in r.rule}
    covered = {r.replace("{id}", "<int:invoice_id>") for r in INVOICE_ROUTES}
    assert registered == covered

We added an unchecked /api/invoices/<id>/csv route to the fixed build and this test failed with “Extra items in the left set”; the other five tests stayed green, because none of them knew the route existed. Most frameworks expose their route table in some form (FastAPI's app.routes, Rails' route set, Spring's handler mappings), so the same guard ports. It depends on a naming convention (here, the invoice_id parameter), which is worth making a convention on purpose.

Choosing the invariant for other bug classes

The question to ask of every fix is: what family of inputs reaches the same sink, and which of those did the patch actually handle?

  • Object-level authorization: every representation of the object (JSON, export, GraphQL node, bulk and search endpoints) and every HTTP method on it.
  • SSRF: run the validator over a corpus of address spellings and redirect cases, and assert that the client connects to the IP the validator approved. The URL parser confusion and DNS rebinding articles have the cases; the Craft bypass above is the rebinding one.
  • Injection: test the sink with each payload family (quote styles, encodings, second-order storage) and assert on parameterisation or output encoding, not on one string being rejected.
  • Signature and token checks: feed the verifier the structural variants (extra elements, alternate algorithms, keys supplied by the token) and assert on what the application ends up trusting.

Layer two: a Nuclei check against the deployed build

A unit test runs against code. The bug can also come back through things the code does not control: a gateway route that maps an old path to a new handler, a CDN rule that caches an authenticated response, a feature flag that turns on a legacy endpoint in one environment. A black-box check against the preview deployment covers that gap. Nuclei is a reasonable choice because its templates are small YAML files that live in the repo and can be reviewed like code.

id: invoice-bola-regression

info:
  name: Invoice BOLA regression (non-owner reads another tenant's invoice)
  author: appsec-team
  severity: high
  description: Replays the invoice IDOR against every route that renders an invoice.
  tags: regression,bola,internal

variables:
  attacker_token: "set-with-var"
  victim_invoice: "101"
  victim_marker: "set-with-var"

http:
  - method: GET
    path:
      - "{{BaseURL}}/api/invoices/{{victim_invoice}}"
      - "{{BaseURL}}/api/invoices/{{victim_invoice}}/pdf"
    headers:
      Authorization: "Bearer {{attacker_token}}"

    matchers-condition: and
    matchers:
      - type: status
        status:
          - 200
      - type: dsl
        dsl:
          - "contains(body, victim_marker)"

The matchers require both a 200 and the victim's marker in the body (matchers-condition: and), so an error page or a login redirect cannot count as a hit. The marker and tokens are template variables overridden with -var at run time, so the secrets stay out of the file; in our run the dsl matcher read victim_marker correctly. nuclei -validate -t invoice-bola-regression.yaml reported “All templates validated successfully”, which is worth running in the PR that changes a template.

Pointed at the lab with Bob's token, it reported both routes on the vuln build, only /api/invoices/101/pdf on the partial build, and nothing on the fixed build. So far, so good.

The exit code stays at 0, findings or not

Then we checked what a CI runner would see. Nuclei v3.11.1 exited with status 0 on all three builds, including the vuln build with two findings. It also exited 0, with an empty output file, when we pointed it at a port where nothing was listening. Our reading of nuclei -h for that version found no flag that turns findings into a non-zero exit. A job that runs nuclei and trusts its status passes when the bug is back, and passes when the preview never deployed.

Both problems are solved by a short wrapper. It runs the template twice: once with the owner's token as a control, which must match every route (if it does not, the target, auth or seed data is broken and the run tested nothing), and once with the attacker's token, which must match nothing.

#!/usr/bin/env bash
set -euo pipefail
T=security/regressions/invoice-bola-regression.yaml
common=(-t "$T" -u "$TARGET" -var victim_marker="$VICTIM_MARKER" -duc -silent -omit-raw -jsonl)

# Control: the owner's token must hit both routes, or the scan tested nothing.
nuclei "${common[@]}" -var attacker_token="$OWNER_TOKEN" -o control.jsonl >/dev/null
if [ "$(wc -l < control.jsonl)" -ne 2 ]; then
  echo "control failed: target unreachable, auth broken or seed data missing" >&2
  exit 1
fi

# Regression: the attacker's token must hit nothing.
nuclei "${common[@]}" -var attacker_token="$ATTACKER_TOKEN" -o findings.jsonl >/dev/null
if [ -s findings.jsonl ]; then
  echo "regression: invoice BOLA is back" >&2
  jq -r '."matched-at"' findings.jsonl >&2
  exit 1
fi
echo "ok: owner reads 2 routes, non-owner reads 0"
vuln     exit 1  regression: invoice BOLA is back (/api/invoices/101, /api/invoices/101/pdf)
partial  exit 1  regression: invoice BOLA is back (/api/invoices/101/pdf)
fixed    exit 0  ok: owner reads 2 routes, non-owner reads 0
down     exit 1  control failed: target unreachable, auth broken or seed data missing
Preview deployed
seeded test tenants
Control run
owner token: 2 matches
Regression run
attacker token: 0 matches
Gate
fail on either mismatch
The control run must match before the regression run means anything; only then does an empty findings file count as a pass.

One more detail: without -omit-raw, each JSONL finding included the full request/response pair, and the response held Alice's data. Nuclei v3.11.1 masked the Authorization header as *** in our output, but a CI artifact full of the victim's response bodies is still a leak, so keep raw pairs out of anything uploaded.

Wiring both layers into CI

The pytest file runs with the rest of the unit suite on every push. The Nuclei job runs after the preview deploy, using tokens for two dedicated, seeded test tenants stored as CI secrets. Never point it at real customer accounts. A GitHub Actions sketch (Nuclei v3.11.1 declares Go 1.26 in its go.mod, hence the version):

security-regressions:
  runs-on: ubuntu-latest
  needs: deploy-preview
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-go@v5
      with:
        go-version: "1.26"
    - run: go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.11.1
    - run: echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
    - run: ./security/regressions/run-invoice-bola.sh
      env:
        TARGET: ${{ needs.deploy-preview.outputs.url }}
        OWNER_TOKEN: ${{ secrets.REGRESSION_OWNER_TOKEN }}
        ATTACKER_TOKEN: ${{ secrets.REGRESSION_ATTACKER_TOKEN }}
        VICTIM_MARKER: ${{ secrets.REGRESSION_VICTIM_MARKER }}
  • Pin the scanner version. A template engine upgrade can change matcher behaviour; take it deliberately and rerun the four cases above when you do.
  • Keep templates next to the code they guard and name them after the advisory or ticket, so a reviewer who deletes one has to say why.
  • Keep blocking checks deterministic. Checks that depend on an out-of-band callback can fail because of DNS or a slow interaction poll; for those, a local listener in the test network is steadier than a public OAST service.
  • Nothing destructive. The regression run should read, not write; a replayed mass-assignment or delete bug belongs in the unit layer against a throwaway database.

What a green regression suite does and does not tell you

These tests keep a known door closed. They say nothing about the door next to it. The coverage guard helps when new routes follow a convention, but it cannot see a GraphQL resolver that returns invoices under a different name, or an authorization bug in a completely different object. A dependency fix like Log4j's is better guarded by a version floor in your dependency policy plus a test of your own logging configuration than by replaying JNDI strings at production.

That is the division of labour worth being explicit about. Exploratory testing, by people or by autonomous tools such as Xalgorix, finds new paths; it is also not exhaustive, and an agent that missed the CSV export last week can miss it again. Regression tests turn each confirmed finding into a check that runs on every build, so that a bug, once found, stays fixed. Each covers the other's weak spot.

Checklist for every security fix: replay the reported request; write the invariant and parametrise it over every route, representation and payload family that reaches the same sink; add a positive control so the test cannot pass by breaking the feature; guard the route list against drift; and, for the deployed check, fail on findings and on a failed control, because the scanner's exit code will not do either for you.

Ready to see it prove a bug?

Start scanning on Cloud →