How to turn a fixed vulnerability into a regression test that fails CI
Log4j’s 2.15.0 fix was incomplete four days after Log4Shell, and Craft CMS shipped a bypass of its own SSRF fix in 2026. A hands-on guide to security regression tests: replaying the report, widening it to the invariant, guarding the route list, and a Nuclei check against the preview build, including why Nuclei’s exit code alone will pass a CI job when the bug is back or the target is down.
By The Xalgorix Team
Log4Shell, CVE-2021-44228, was published on 2021-12-10, and Log4j 2.15.0 was the fix. Four days later the project published CVE-2021-45046, whose description opens: “It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations.” Lookup patterns that arrived through the Thread Context Map could still reach JNDI, and 2.16.0 removed message lookups and disabled JNDI by default.
It is not a one-off. On 2026-01-05 Craft CMS published an SSRF in its GraphQL asset mutation as CVE-2025-68437; on 2026-02-24 came CVE-2026-27127, described in its CVE record as “a bypass of the security fix for CVE-2025-68437” through DNS rebinding. Our ruby-saml teardown covers a run of signature-bypass CVEs in one library between 2024 and 2025. A security fix is a claim about a whole class of inputs, and a patch often covers only the input in the report.
The durable answer is a test that replays the attack and fails the build when the bug comes back. This guide builds one in two layers on a small lab app: a unit-level test that runs in milliseconds with the rest of your suite, and a Nuclei check that runs against the deployed preview. We ran everything below on 2026-11-19 (Python 3.13, Flask 3.1.3, pytest 9.1.1, Nuclei v3.11.1), including one Nuclei behaviour that turns a naive CI job green when the target is down.
The lab: one invoice IDOR and three builds
The app serves invoices to authenticated users. Each invoice has two routes: JSON at /api/invoices/<id> and a PDF export at /api/invoices/<id>/pdf. The bug is a textbook broken object level authorization (API1:2023): Bob's token can read Alice's invoice by changing the id. A MODE setting produces three builds of the same code:
- vuln: no ownership check anywhere.
- partial: the developer added the check to the JSON route named in the report. The PDF route still loads the invoice without it. This is the incomplete fix.
- fixed: both routes go through one loader that checks ownership and returns 404 to anyone else.
def load_invoice(invoice_id, check_owner):
inv = INVOICES.get(invoice_id)
if inv is None or (check_owner and inv["owner"] != g.user):
abort(404)
return inv
@app.get("/api/invoices/<int:invoice_id>")
def get_invoice(invoice_id):
return jsonify(load_invoice(invoice_id, check_owner=MODE in ("partial", "fixed")))
@app.get("/api/invoices/<int:invoice_id>/pdf")
def invoice_pdf(invoice_id):
inv = load_invoice(invoice_id, check_owner=MODE == "fixed")
return f"%PDF-1.7 invoice {inv['id']} {inv['memo']}", 200, {"Content-Type": "application/pdf"}Alice's seeded invoice carries the memo ALICE-PRIVATE-101. That marker matters later: a security test needs something that can only appear in the response if the attacker really read the victim's data, rather than a status code that a login page or an error handler could also return.
Layer one: replay the report, then widen it to the invariant
The natural first test is the request from the report, word for word. Write it, because it documents the bug. Then write the test that states what the fix is supposed to guarantee: no route that renders an invoice serves it to someone who does not own it.
import pytest
from app import app
ALICE = {"Authorization": "Bearer alice-token"}
BOB = {"Authorization": "Bearer bob-token"}
@pytest.fixture
def client():
return app.test_client()
# 1. The narrow test: replays the exact request from the report.
def test_report_replay_bob_cannot_read_alice_invoice(client):
resp = client.get("/api/invoices/101", headers=BOB)
assert resp.status_code == 404
assert b"ALICE-PRIVATE-101" not in resp.data
# 2. The invariant: no route that renders an invoice serves it to a non-owner.
INVOICE_ROUTES = ["/api/invoices/{id}", "/api/invoices/{id}/pdf"]
@pytest.mark.parametrize("route", INVOICE_ROUTES)
def test_non_owner_gets_nothing(client, route):
resp = client.get(route.format(id=101), headers=BOB)
assert resp.status_code in (403, 404)
assert b"ALICE-PRIVATE-101" not in resp.data
# 3. Positive control: the owner still can, so the test cannot pass by breaking the feature.
@pytest.mark.parametrize("route", INVOICE_ROUTES)
def test_owner_still_reads_own_invoice(client, route):
resp = client.get(route.format(id=101), headers=ALICE)
assert resp.status_code == 200
assert b"ALICE-PRIVATE-101" in resp.dataRunning the same file against the three builds:
MODE=vuln 3 failed, 2 passed (replay + both invariant cases fail) MODE=partial 1 failed, 4 passed (only test_non_owner_gets_nothing[.../pdf] fails) MODE=fixed 5 passed
The replay test passes against the partial fix. If it were the only test, the incomplete patch would ship with a green build and a regression test that looked like it was guarding something. The parametrised invariant is what catches the PDF route, and it is the same move that would have mattered for Log4j and Craft: the fix covered the input in the report, and the bypass came through a neighbouring input to the same sink.
The positive control looks redundant until someone “fixes” a flaky test by returning 404 for every invoice. Without it, a broken feature and a secure feature produce the same green run.
Keep the route list from going stale
The invariant is only as good as INVOICE_ROUTES. Next quarter someone adds a CSV export, forgets the shared loader, and the list does not know the route exists. You can make the framework tell you. Flask exposes its routing table as app.url_map, so a fourth test can require every route with an invoice id to be listed:
# 4. Coverage guard: a new invoice route fails the build until it joins INVOICE_ROUTES.
def test_every_invoice_route_is_covered():
registered = {r.rule for r in app.url_map.iter_rules() if "<int:invoice_id>" in r.rule}
covered = {r.replace("{id}", "<int:invoice_id>") for r in INVOICE_ROUTES}
assert registered == coveredWe added an unchecked /api/invoices/<id>/csv route to the fixed build and this test failed with “Extra items in the left set”; the other five tests stayed green, because none of them knew the route existed. Most frameworks expose their route table in some form (FastAPI's app.routes, Rails' route set, Spring's handler mappings), so the same guard ports. It depends on a naming convention (here, the invoice_id parameter), which is worth making a convention on purpose.
Choosing the invariant for other bug classes
The question to ask of every fix is: what family of inputs reaches the same sink, and which of those did the patch actually handle?
- Object-level authorization: every representation of the object (JSON, export, GraphQL node, bulk and search endpoints) and every HTTP method on it.
- SSRF: run the validator over a corpus of address spellings and redirect cases, and assert that the client connects to the IP the validator approved. The URL parser confusion and DNS rebinding articles have the cases; the Craft bypass above is the rebinding one.
- Injection: test the sink with each payload family (quote styles, encodings, second-order storage) and assert on parameterisation or output encoding, not on one string being rejected.
- Signature and token checks: feed the verifier the structural variants (extra elements, alternate algorithms, keys supplied by the token) and assert on what the application ends up trusting.
Layer two: a Nuclei check against the deployed build
A unit test runs against code. The bug can also come back through things the code does not control: a gateway route that maps an old path to a new handler, a CDN rule that caches an authenticated response, a feature flag that turns on a legacy endpoint in one environment. A black-box check against the preview deployment covers that gap. Nuclei is a reasonable choice because its templates are small YAML files that live in the repo and can be reviewed like code.
id: invoice-bola-regression
info:
name: Invoice BOLA regression (non-owner reads another tenant's invoice)
author: appsec-team
severity: high
description: Replays the invoice IDOR against every route that renders an invoice.
tags: regression,bola,internal
variables:
attacker_token: "set-with-var"
victim_invoice: "101"
victim_marker: "set-with-var"
http:
- method: GET
path:
- "{{BaseURL}}/api/invoices/{{victim_invoice}}"
- "{{BaseURL}}/api/invoices/{{victim_invoice}}/pdf"
headers:
Authorization: "Bearer {{attacker_token}}"
matchers-condition: and
matchers:
- type: status
status:
- 200
- type: dsl
dsl:
- "contains(body, victim_marker)"The matchers require both a 200 and the victim's marker in the body (matchers-condition: and), so an error page or a login redirect cannot count as a hit. The marker and tokens are template variables overridden with -var at run time, so the secrets stay out of the file; in our run the dsl matcher read victim_marker correctly. nuclei -validate -t invoice-bola-regression.yaml reported “All templates validated successfully”, which is worth running in the PR that changes a template.
Pointed at the lab with Bob's token, it reported both routes on the vuln build, only /api/invoices/101/pdf on the partial build, and nothing on the fixed build. So far, so good.
The exit code stays at 0, findings or not
Then we checked what a CI runner would see. Nuclei v3.11.1 exited with status 0 on all three builds, including the vuln build with two findings. It also exited 0, with an empty output file, when we pointed it at a port where nothing was listening. Our reading of nuclei -h for that version found no flag that turns findings into a non-zero exit. A job that runs nuclei and trusts its status passes when the bug is back, and passes when the preview never deployed.
Both problems are solved by a short wrapper. It runs the template twice: once with the owner's token as a control, which must match every route (if it does not, the target, auth or seed data is broken and the run tested nothing), and once with the attacker's token, which must match nothing.
#!/usr/bin/env bash
set -euo pipefail
T=security/regressions/invoice-bola-regression.yaml
common=(-t "$T" -u "$TARGET" -var victim_marker="$VICTIM_MARKER" -duc -silent -omit-raw -jsonl)
# Control: the owner's token must hit both routes, or the scan tested nothing.
nuclei "${common[@]}" -var attacker_token="$OWNER_TOKEN" -o control.jsonl >/dev/null
if [ "$(wc -l < control.jsonl)" -ne 2 ]; then
echo "control failed: target unreachable, auth broken or seed data missing" >&2
exit 1
fi
# Regression: the attacker's token must hit nothing.
nuclei "${common[@]}" -var attacker_token="$ATTACKER_TOKEN" -o findings.jsonl >/dev/null
if [ -s findings.jsonl ]; then
echo "regression: invoice BOLA is back" >&2
jq -r '."matched-at"' findings.jsonl >&2
exit 1
fi
echo "ok: owner reads 2 routes, non-owner reads 0"vuln exit 1 regression: invoice BOLA is back (/api/invoices/101, /api/invoices/101/pdf) partial exit 1 regression: invoice BOLA is back (/api/invoices/101/pdf) fixed exit 0 ok: owner reads 2 routes, non-owner reads 0 down exit 1 control failed: target unreachable, auth broken or seed data missing
One more detail: without -omit-raw, each JSONL finding included the full request/response pair, and the response held Alice's data. Nuclei v3.11.1 masked the Authorization header as *** in our output, but a CI artifact full of the victim's response bodies is still a leak, so keep raw pairs out of anything uploaded.
Wiring both layers into CI
The pytest file runs with the rest of the unit suite on every push. The Nuclei job runs after the preview deploy, using tokens for two dedicated, seeded test tenants stored as CI secrets. Never point it at real customer accounts. A GitHub Actions sketch (Nuclei v3.11.1 declares Go 1.26 in its go.mod, hence the version):
security-regressions:
runs-on: ubuntu-latest
needs: deploy-preview
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.26"
- run: go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.11.1
- run: echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- run: ./security/regressions/run-invoice-bola.sh
env:
TARGET: ${{ needs.deploy-preview.outputs.url }}
OWNER_TOKEN: ${{ secrets.REGRESSION_OWNER_TOKEN }}
ATTACKER_TOKEN: ${{ secrets.REGRESSION_ATTACKER_TOKEN }}
VICTIM_MARKER: ${{ secrets.REGRESSION_VICTIM_MARKER }}- Pin the scanner version. A template engine upgrade can change matcher behaviour; take it deliberately and rerun the four cases above when you do.
- Keep templates next to the code they guard and name them after the advisory or ticket, so a reviewer who deletes one has to say why.
- Keep blocking checks deterministic. Checks that depend on an out-of-band callback can fail because of DNS or a slow interaction poll; for those, a local listener in the test network is steadier than a public OAST service.
- Nothing destructive. The regression run should read, not write; a replayed mass-assignment or delete bug belongs in the unit layer against a throwaway database.
What a green regression suite does and does not tell you
These tests keep a known door closed. They say nothing about the door next to it. The coverage guard helps when new routes follow a convention, but it cannot see a GraphQL resolver that returns invoices under a different name, or an authorization bug in a completely different object. A dependency fix like Log4j's is better guarded by a version floor in your dependency policy plus a test of your own logging configuration than by replaying JNDI strings at production.
That is the division of labour worth being explicit about. Exploratory testing, by people or by autonomous tools such as Xalgorix, finds new paths; it is also not exhaustive, and an agent that missed the CSV export last week can miss it again. Regression tests turn each confirmed finding into a check that runs on every build, so that a bug, once found, stays fixed. Each covers the other's weak spot.
Ready to see it prove a bug?
Start scanning on Cloud →