Best LLM Vulnerability Scanners 2026: Garak, PyRIT, Promptfoo

The best LLM vulnerability scanners in 2026 occupy a narrower category than most procurement guides acknowledge: tools that actively probe your model or application for exploitable weaknesses before an attacker does. This is distinct from runtime filters that stop attacks in flight and from general SAST tools that scan your Python rather than your prompts. If you’re evaluating pre-deployment scanning for an LLM application this year, four tools appear on serious shortlists — Garak, PyRIT, Promptfoo, and Mindgard — and the selection turns on deployment model, attack surface scope, and CI/CD integration depth.

Those four are the core. Around them sit two further groups that any complete stack needs and that buyers routinely conflate with the scanners themselves: narrow specialists that outperform the generalists inside a single attack class, and the runtime guards that catch what pre-deployment scanning cannot. This guide covers all three layers.

Pre-Deployment Scanning and Runtime Filtering Are Not the Same Purchase

Conflating the two is the most common mistake teams make when evaluating tools, and it produces stacks with three overlapping input scanners and no red-team coverage at all.

Pre-deployment scanners run before or alongside a release. They fire structured adversarial prompts at a model or application, collect the responses, and report which attack classes succeeded and at what rate. Their output is an audit trail and a release decision. They probe mostly LLM01, LLM02, and LLM06.

Runtime guards sit in the request path in production. They inspect live prompts and responses and block or redact in flight. Their output is an enforcement action and a log line. They cover the real-time surface: a public endpoint receiving untrusted user input.

A scanner finds holes; it does not seal them. A guard seals holes; it does not find the ones it was never configured to look for. A complete stack needs both, and the sections below are ordered accordingly.

What LLM Vulnerability Scanners Actually Test

The reference threat model is the OWASP Top 10 for LLM Applications 2025, which updated significantly from the 2023 version. Two new categories were added and several others substantially reworked. The risks most relevant to scanner coverage:

• LLM01:2025 Prompt Injection — crafted inputs hijack model behavior; the attack surface expanded considerably with agentic architectures where the model processes external data.
• LLM02:2025 Sensitive Information Disclosure — training-data extraction and context leakage from both the model and the application layer.
• LLM06:2025 Excessive Agency — an agent with tool access takes unauthorized or consequential actions beyond intended scope.
• LLM07:2025 System Prompt Leakage — extraction of confidential system instructions through adversarial prompting.
• LLM08:2025 Vector and Embedding Weaknesses — retrieval poisoning and adversarial manipulation of RAG pipeline inputs.

No single scanner covers all ten categories at equal depth. The practical question before any procurement is: which OWASP categories does your deployment actually expose? A chat interface sitting in front of a single model has a different threat surface than a multi-agent pipeline with web browsing, code execution, and CRM write access.

The Best LLM Vulnerability Scanners for 2026

Garak (NVIDIA)

Garak — generative AI red-teaming and assessment kit — is the broadest open-source scanner available and the closest analog to Nessus for LLM infrastructure. Version 0.15.0, released May 2026, added a multi-turn GOAT probe, an Agent-breaker probe for testing tools available to LLM agents, a system-prompt-extraction probe targeting LLM07, a ModernBERT refusal detector, and native NeMo Guardrails server support.

The tool ships with 50+ probe modules and 28 detector types. Probe categories cover prompt injection, jailbreaks, encoding bypasses (Base64, ROT-13, visual encoding), data leakage, package hallucination, and toxic content generation. It supports 23 generator backends: OpenAI, Anthropic, Hugging Face, AWS Bedrock, Replicate, Cohere, Groq, llama.cpp, and custom REST endpoints.

Installation is pip install -U garak, and the architecture is modular in a way that matters operationally: probes are decoupled from detectors, so adding a domain-specific probe does not mean rewriting scoring logic. Results log as JSONL, which drops cleanly into whatever reporting pipeline already exists. Independent coverage reaches the same conclusion most practitioner reviews do: breadth is the reason to run it.

Trade-offs: CLI-first workflow requires scripting effort to integrate into CI/CD pipelines at scale. A full suite run takes hours; targeted probe subsets bring that to minutes and are the practical choice for every-commit gating. Its probes are also single-turn by default, so the patient multi-turn attacker is better simulated by PyRIT below. Garak is a pre-deployment scanner — it finds holes, it doesn’t seal them at runtime. For the complementary guardrail layer, guardml.io covers open-source runtime defenses in detail.

Garak fits teams that want maximum probe breadth and are comfortable building around a command-line tool. It does not fit teams expecting a turnkey enterprise dashboard. For the full probe/detector/generator architecture and CLI workflow, see our garak vulnerability scanner walkthrough.

Microsoft PyRIT

PyRIT (Python Risk Identification Toolkit) is Microsoft’s open-source automation framework for adversarial probing of generative AI systems. The framework ships with 53+ attack datasets — HarmBench, AdvBench, XSTest, AIRT, and others — and 20+ response scorers including LLM-as-judge, Azure AI Content Safety integration, Likert scales, and true/false classifiers.

The standout capability for agentic deployments is XPIAOrchestrator, which runs cross-domain prompt injection attacks by embedding malicious instructions in external data sources: document stores, email bodies, and web content retrieved by the agent. This directly targets LLM08 vector weaknesses and LLM01 indirect injection — the two attack classes hardest to catch with naive probe suites that only test direct user input.

The orchestrators are the other half of the story. RedTeamingOrchestrator runs multi-turn conversations in which an attacker model generates follow-ups from the target’s replies; CrescendoOrchestrator escalates gradually through benign-looking turns, which is how real attackers approach well-guarded systems; TreeOfAttacksWithPruningOrchestrator explores several attack paths in parallel and prunes the dead ends. A memory layer logs every prompt, transform, and result to SQLite or Azure SQL, so runs are comparable release over release. PyRIT also handles multi-modal targets (text, image, audio, video) and is MIT-licensed, so the only cost is the models used for attack generation.

PyRIT requires more configuration effort than Garak for comparable coverage, but the scoring infrastructure and dataset library are more mature for producing structured OWASP or NIST AI RMF evidence maps. Recommended for teams who need auditable test artifacts tied to specific control frameworks, and for any team running RAG-augmented or agentic systems. Our PyRIT framework explainer breaks down the target/dataset/orchestrator/converter/scorer architecture and the multi-turn strategies that single-shot scanners miss. For a deeper look at the indirect injection mechanics PyRIT targets, aisec.blog covers the attack patterns in operational detail.

Promptfoo

Promptfoo’s red-team mode auto-generates adversarial prompts using 50+ attack plugins spanning prompt injection, jailbreaks, PII leakage, SSRF, SQL injection through tool calls, excessive agency, and hallucination. The platform ships OWASP LLM Top 10 and NIST AI RMF presets, and it has the cleanest native CI/CD integration of any tool in this category: a YAML config wraps your LLM endpoint, a GitHub Action gates the release, and failing builds block on configurable vulnerability thresholds.

In March 2026, OpenAI acquired Promptfoo for undisclosed terms, having built a user base of 350,000 developers and deployments at 25% of Fortune 500 companies. Per OpenAI’s announcement, the project remains open source under its current MIT license. The acquisition confirms the tool’s production credibility but introduces the standard governance questions for any open-source project absorbed by a commercial entity.

Underneath the red-team mode is an assertion engine (semantic similarity, regex, and LLM-graded criteria run against model outputs), which is what makes Promptfoo double as a regression harness. Teams define expected behaviours once and gate every deployment against them, catching the case where a model or prompt-template update quietly reopens a jailbreak that was previously closed. The free core is open source; the commercial tier adds a hosted dashboard and team features.

Promptfoo occupies the sweet spot for development teams who want automated security gates without dedicated red team staffing. Garak covers more attack categories in depth; Promptfoo integrates more cleanly into developer workflows from day one. For how these three slot into a CI pipeline as a per-commit gate, scheduled campaign, and PR-blocking check, see Automated LLM Red-Teaming in CI: garak vs PyRIT vs Promptfoo.

Mindgard

Mindgard is the enterprise option: annual subscription (five figures), managed adversarial testing, continuous scanning rather than point-in-time runs, and a reporting layer that maps findings directly to MITRE ATLAS and OWASP LLM categories. The web interface is designed for security teams rather than developers, and the output format is built for compliance artifact handoffs.

The trade-off is cost and integration overhead. Mindgard fits organizations with dedicated AppSec budget, a compliance requirement (SOC 2 AI addendum, EU AI Act high-risk classification), and a preference for vendor-managed tooling over maintained open-source stacks. It does not fit teams who want developer-led security gates or who are early enough in their LLM deployment to still be validating threat model assumptions.

Specialists That Beat the Generalists Inside One Attack Class

The four scanners above are breadth plays. Several narrower tools outperform all of them within a single vulnerability class, and they are worth layering in once you know which class actually dominates your threat model. Published comparisons and vendor documentation agree on the shape of the trade-off even where they disagree on the exact numbers, so read the following as relative positioning rather than a scoreboard.

Rebuff targets prompt injection and almost nothing else. It stacks a canary-token layer, a vector-similarity layer matched against a store of known injections, and an LLM-based classification layer, and it is consistently reported at the top of the field on injection detection specifically. The cost of that focus is large blind spots: jailbreak and exfiltration coverage are marginal, so it belongs in a layered stack rather than as a sole control. Note also that the repository has been archived since May 2025: the design is sound, but nobody upstream is updating the heuristics. aisecreviews.com’s Rebuff review covers the maintenance posture in detail.

Vigil scans at the request/response level and adds YARA-based custom rules, which is the differentiator. Rules written against internal vocabulary (the names of internal systems, proprietary product terms, specific system-prompt strings) let Vigil catch extraction attempts aimed at your organization specifically, which no general-purpose classifier trained on a public corpus will do. That level of customization is genuinely hard to get elsewhere.

Prompt Armor is narrower still: its detection engine is trained on system-prompt exfiltration and confidentiality violations, and it leads that category while trailing badly on general injection and jailbreak coverage. Correct choice only when protecting the system prompt is the primary threat model.

Cisco AI Defense (formerly Robust Intelligence) is the one tool in this group that reaches below the prompt layer. It tests models directly through their inference API and flags attack surfaces in fine-tuned weights, which makes it the only meaningful option here for supply-chain risk: the case where you are pulling fine-tunes from Hugging Face or an internal model registry and need to know what is inside them before they load. adversarialml.dev covers the weight-level attack background.

The Runtime Layer: Blocking Live Attacks After the Scan Is Done

Everything above runs before deployment. These run in production.

LLM Guard is the leading open-source runtime option. It sits in front of your API and applies a configurable analyzer pipeline to both directions of traffic: on input, prompt-injection detection, PII anonymization, toxicity classification, and regex blocklists; on output, sensitive-data detection, relevance scoring, and factual-consistency checks. It runs locally, so nothing leaves your environment, which is the deciding factor for regulated deployments. Its PII detection is widely regarded as one of the better open implementations. The trade-off is that you own the latency budget: each additional analyzer stacks processing time, so benchmark the pipeline before committing to it inline. Our LLM Guard input/output scanning walkthrough covers the full scanner catalog and where the latency goes.

Vigil appears again here, because its layered detection (vector similarity, YARA matching, transformer classification, canary tokens) works inline as well as in a scan. It complements LLM Guard rather than replacing it: Vigil’s multi-method injection detection paired with LLM Guard’s output analysis covers more surface than either alone.

Lakera Guard is the dominant commercial runtime option for teams that want protection without operating the infrastructure. Single-line integration, sub-50ms processing per the vendor’s documentation, and a proprietary threat-intelligence feed updated from observed attack traffic. The case for it is operational simplicity; the case against it is the standard managed-security trade: your prompts transit a third party, your latency depends on their uptime, and pricing scales with volume.

For a head-to-head of the dedicated injection detectors in this group, see Prompt Injection Detectors: Rebuff vs Vigil vs LLM Guard; for the guardrail models that sit alongside them, see Llama Guard vs NeMo vs OpenAI Moderation.

Matching Tool to Role

No single tool covers the full OWASP LLM10 surface. The right stack depends on where you sit. To turn your own deployment, threat, integration point, and budget into a ranked shortlist, run our interactive scanner picker, which knocks out tools that fail your hard constraints and scores the rest with a transparent pick-if / skip-if for each:

• Security engineers running quarterly red team assessments: Garak for breadth across direct attacks, PyRIT for agent and RAG-specific indirect injection chains.
• DevSecOps teams implementing CI/CD gates: Promptfoo has the lowest setup cost and surfaces high-severity issues fastest against a developer workflow.
• Enterprise AppSec with compliance mandates: Mindgard provides the audit trail; layer Garak underneath for probe depth the managed platform doesn’t reach.
• Teams that have not yet implemented pre-deployment scanning: start with Promptfoo — the OWASP preset configuration takes minutes and will find LLM01 and LLM06 exposures in most applications immediately.

How to Build the Stack

The standard starting point is Garak for pre-deployment scanning plus LLM Guard in front of the production API. That mirrors the SAST-plus-WAF pattern from web application security: test before you ship, filter at runtime. From there the additions are conditional rather than cumulative:

• Add PyRIT if you are shipping an agentic system that uses tools or takes external actions, or if your threat model includes a multi-turn adversary.
• Add Promptfoo if you have a CI/CD pipeline and want automated regression testing on every deploy.
• Add Rebuff or Vigil if prompt injection specifically, rather than the whole OWASP surface, is what keeps you up at night.
• Add Cisco AI Defense or an equivalent supply-chain scanner if you ingest third-party model weights without an internal scanning gate. This is the layer teams most often skip and least often can justify skipping.
• Enterprise buyers evaluating the full commercial landscape should also look at WhyLabs LLM Security for observability and Protect AI’s Guardian for model supply chain.

None of these replace red teaming by people who understand your specific application, user base, and abuse surface. They automate the repeatable parts. What to block, at what threshold, and what false-positive rate is acceptable remain engineering decisions. For a structured way to measure that cost and tune the threshold without gutting detection, see False Positive Cost in Refusal Systems: Measure and Tune.

The residual risk after any scanner is the attack surface the tool’s probe library does not yet cover. That gap expands with every new agentic deployment pattern — multi-agent handoffs, tool-augmented reasoning, retrieval over untrusted content. The correct response is layered scanning at build time and runtime filtering at the production edge, not faith in any single tool’s coverage claims.