OWASP LLM Top 10 Scanning Checklist: What to Test at Each Layer

The OWASP Top 10 for LLM Applications is the closest thing the industry has to a shared vocabulary for LLM application risk, but the list itself is not a test plan. This OWASP LLM Top 10 scanning checklist translates the 2025 edition into concrete scanning tasks: what to probe for each entry, which layer the scan belongs at, and which tools currently cover it. One caveat up front: no single scanner covers all ten entries, and at least three of them (supply chain, excessive agency, poisoning) are only partially scannable at all. Treat scan results as one input to a layered control set, not a clean bill of health.

Know your three scanning layers first

Every entry on the list maps to one of three places in the pipeline, and picking the wrong layer is the most common way teams waste an evaluation.

Build-time (static). Scanning model artifacts, dependencies, and datasets before anything runs. This is where supply-chain and some poisoning checks live. It looks more like traditional SCA than LLM testing.

Pre-deployment (adversarial probing). Firing structured attack payloads at a candidate model or application and grading the responses. This is the home turf of the open-source scanners: garak, NVIDIA’s Apache-2.0 scanner that, per its repo, “checks if an LLM can be made to fail in a way we don’t want” with probe modules for prompt injection, jailbreaks, data leakage, toxicity, and XSS-via-output; PyRIT, Microsoft’s risk-identification framework for generative AI red teaming; and promptfoo, whose docs map its red-team plugins explicitly onto the 2025 list, down to an owasp:llm shorthand that activates coverage for all ten entries.

Runtime (guardrails and monitoring). Input/output filtering, tool allowlists, and rate controls on the live system. Scanning validates these controls exist and hold; it does not replace them. For the guardrail side of that equation, guardml.io tracks the defensive tooling landscape in more depth than we will here.

The checklist, entry by entry

• LLM01: Prompt Injection. Probe direct injection (jailbreak templates, encoding tricks, role-play overrides) and indirect injection (payloads planted in documents your RAG pipeline will retrieve). garak’s injection and jailbreak probe families and promptfoo’s owasp:llm:01 plugin set both cover the direct case; indirect injection needs test corpora seeded into your actual retrieval path. The offensive side of this technique moves fast — aisec.blog covers current injection and agent-exploitation tradecraft if you need attacker-realistic payloads.

• LLM02: Sensitive Information Disclosure. Seed known canary strings (fake PII, fake API keys) into system prompts, fine-tuning data, and RAG context, then scan for their reappearance in outputs. promptfoo ships dedicated pii:direct, pii:api-db, and pii:session plugins per its docs; garak’s leakage probes cover the model-memorization angle.

• LLM03: Supply Chain. Build-time, not prompt-time. Scan model artifacts for unsafe deserialization (pickle-based formats remain the classic vector), verify checksums and provenance for third-party models and adapters, and keep an SBOM for the serving stack. promptfoo’s ModelAudit component does static artifact scanning; conventional SCA tooling covers the Python dependency tree.

• LLM04: Data and Model Poisoning. The weakest link in any scanning checklist. Behavioral probing can surface crude backdoors (trigger phrases producing anomalous outputs), but a competently poisoned model will pass. Dataset provenance controls and training-data filtering do more here than any scanner; be suspicious of vendors implying otherwise.

• LLM05: Improper Output Handling. Treat model output as untrusted input to downstream systems. Scan for outputs that survive into HTML, SQL, or shell contexts unescaped — garak includes XSS-oriented probes — and add output-validation assertions (JSON-schema enforcement where structure is expected) to your CI evals.

• LLM06: Excessive Agency. Mostly a design-review item, not a scanning item. promptfoo’s excessive-agency and rbac plugins can probe whether an agent exceeds its intended permissions, but the primary control is a reviewed tool allowlist with least-privilege scopes. A scan can confirm the allowlist holds; it cannot tell you the allowlist is sensible.

• LLM07: System Prompt Leakage. Run prompt-extraction probes (promptfoo ships one; garak’s leakage probes overlap) and, separately, audit the system prompt itself. The durable mitigation is architectural: nothing secret belongs in a system prompt, because extraction defenses fail eventually.

• LLM08: Vector and Embedding Weaknesses. Scan the RAG layer specifically: indirect injection through retrieved chunks, cross-tenant retrieval (does user A’s query surface user B’s documents), and access control on the vector store. promptfoo’s bola, bfla, and indirect-injection plugins target this entry per its docs.

• LLM09: Misinformation. This is evaluation more than scanning — hallucination checks need ground truth to grade against. Build a domain-specific eval set and run it on every model or prompt change; generic hallucination probes (both garak and promptfoo include them) catch only the egregious cases.

• LLM10: Unbounded Consumption. Test rate limits, token caps, and cost ceilings under adversarial load: repetition-inducing prompts (promptfoo’s divergent-repetition plugin), oversized inputs, and concurrent-request floods. This overlaps with ordinary load testing; the difference is that the attacker controls per-request cost.

What this checklist will not catch

Scanning is point-in-time; LLM behavior is not. A model version bump, a system-prompt edit, or new retrieval content can reopen a finding you closed last quarter, so the checklist only earns its keep wired into CI as a regression gate rather than run as an annual exercise. That framing matches the Measure function in the NIST AI Risk Management Framework, whose Generative AI Profile (NIST-AI-600-1, released July 2024) is the reference we would hand an auditor asking why scan cadence matters.

The residual risk is concentrated in the entries scanners handle worst: LLM03 and LLM04 are provenance and process problems, LLM06 is an architecture problem, and LLM09 needs domain ground truth no vendor can ship you. Expect a scanner to give you strong coverage on injection, leakage, extraction, and output handling — roughly six of the ten entries — and plan governance controls for the rest.