How to Scan an LLM for Prompt Injection: Tools, Method, and Limits

The practical answer to how to scan an LLM for prompt injection is that you need two different scans, run at two different points in the lifecycle: an offline probe suite that fires known injection payloads at your model or application before deployment, and a runtime classifier that inspects prompts and retrieved documents in production. Prompt injection is LLM01:2025 in the OWASP Top 10 for LLM Applications, and OWASP’s own guidance lists adversarial testing as one mitigation among seven, not a fix. This guide walks through the build-time pass with garak, PyRIT, and promptfoo, the runtime pass using Azure Prompt Shields as a worked example, and, just as important, what a clean scan report does not tell you.

Decide what you are actually scanning

OWASP defines prompt injection as occurring when user-supplied input alters the model’s behavior or output in unintended ways, and it notes the input does not even need to be human-readable, only parseable by the model. The category splits in two. Direct injection is the user typing the attack into your chat box. Indirect injection arrives through content the application retrieves on its own: web pages, uploaded files, email bodies, RAG chunks.

The distinction matters because it changes the scan target. Greshake et al. demonstrated in 2023 that real LLM-integrated applications, including Bing’s GPT-4-powered chat, could be compromised through retrieved content alone, with no access to the user interface. Their framing is the one to internalize: LLM-integrated applications blur the line between data and instructions. A scan that only probes the foundation model through its chat endpoint misses the retrieval path, the tool-calling layer, and the system prompt, which together form most of the real attack surface. Scan the application, not just the model.

Baseline with an offline scanner: garak

garak (Generative AI Red-teaming and Assessment Kit), maintained by NVIDIA, is the closest thing this category has to a general-purpose vulnerability scanner. Per its README, it checks whether an LLM “can be made to fail in a way we don’t want,” and its probe library covers the injection-relevant ground: a promptinject module implementing the PromptInject attack framework, encoding-based injection probes, and the DAN-style jailbreak families, alongside probes for data leakage and toxicity.

Two properties make it useful as a first pass. It speaks to most things you would deploy, including Hugging Face models, the OpenAI API, AWS Bedrock, Cohere, Groq, local ggml/gguf weights, and generic REST endpoints, and that REST support is what lets you point it at your own application rather than the bare model. And its output is a JSONL report with per-probe results plus a hit log recording every payload that actually landed, which gives you concrete failing prompts to replay after a fix.

A minimal run looks like garak --target_type openai --target_name <model> --probes promptinject, and garak --list_probes enumerates the full probe set. Treat the resulting pass rate as a sample against known public attacks, not a security score.

Orchestrate application-level attacks: PyRIT and promptfoo

Single-shot probes undertest the indirect path, so the second build-time layer is an orchestration framework. PyRIT, Microsoft’s open-source Python Risk Identification Tool, is built, in the project’s words, “to empower security professionals and engineers to proactively identify risks in generative AI systems.” Its architecture maps well onto injection testing: converters mutate payloads through encodings and obfuscations, orchestrators drive multi-turn and multi-step attack flows, and scorers grade whether an attempt succeeded. That structure is what you want for indirect injection: seed your document store or mock retrieval layer with hostile content, run the agent’s normal workflow, and score whether the planted instructions were followed. Note the repository moved from the Azure GitHub organization to microsoft/PyRIT in early 2026; the old location is archived.

promptfoo’s red-teaming mode covers similar ground from a CI-friendly angle. Per its documentation it generates adversarial inputs against your application, evaluates responses with deterministic and model-graded metrics, and explicitly supports testing RAG architectures and agents end to end. In practice it slots in as a regression gate: once garak or PyRIT has surfaced failing payloads, a scoped promptfoo suite re-runs them on every prompt or model change. For a deeper catalog of the attack techniques these suites draw from, the offensive side is tracked in detail at aisec.blog.

Add runtime scanning for what offline probes miss

Build-time scanning samples yesterday’s attacks. The production complement is an input classifier in the request path, and Microsoft’s Prompt Shields in Azure AI Content Safety is a well-documented example of the pattern. It is a unified API with two detectors that mirror the direct/indirect split: a user-prompt shield (formerly “jailbreak risk detection”) that flags system-rule overrides, embedded conversation mockups, role-play persona swaps, and encoding attacks, and a document shield that scans third-party content for manipulated instructions, exfiltration commands, and malware-spreading directives before the model ever sees them.

The documented limits are as instructive as the capabilities. Per Microsoft’s docs, the models are trained and tested on eight languages (Chinese, English, French, German, Spanish, Italian, Japanese, Portuguese) with degraded quality elsewhere, and the same page states plainly that the shields “may not catch all attack vectors or may flag legitimate prompts,” recommending additional validation layers. That caveat generalizes to every classifier in this category, whichever vendor ships it. The broader guardrail tooling landscape, including open-weight and self-hosted options, is covered at guardml.io.

What a clean scan proves, and what it leaves open

A green report from all three layers means your application resists the payloads those tools know about, in the languages their classifiers cover, as of the day you ran them. It does not mean the injection problem is closed. Greshake et al. wrote in 2023 that effective mitigations for these threats were lacking, and the honest reading of OWASP’s LLM01:2025 entry is that this is still architecture guidance, not a solved detection problem: constrain model behavior in the system prompt, validate outputs with deterministic code, enforce least privilege on tool access, require human approval for high-impact actions, and segregate external content so the model can distinguish it from instructions.

Concretely, the residual risks after scanning are novel payloads that postdate your probe corpus, cross-language gaps in runtime classifiers, and semantic attacks phrased innocuously enough that no pattern matcher fires. The workable posture is to re-run the offline suite on every model, prompt, or retrieval change, keep the runtime classifier’s verdicts flowing into audit logs rather than treating them as silent drops, and design the application so that a successful injection has little to reach. Scanning tells you where the fence is weak. It does not replace the fence.

Sources

• OWASP LLM01:2025 Prompt Injection: the canonical definition, the direct/indirect taxonomy, and the seven-item mitigation list referenced throughout this guide.
• NVIDIA garak: README and probe documentation for the open-source LLM vulnerability scanner, including the promptinject and encoding probe families and supported targets.
• Microsoft PyRIT: the Python Risk Identification Tool’s self-description and its converter/orchestrator/scorer architecture.
• Greshake et al., arXiv:2302.12173: “Not what you’ve signed up for,” the paper that established indirect prompt injection against real LLM-integrated applications.
• Prompt Shields in Azure AI Content Safety: Microsoft’s documentation of runtime user-prompt and document attack detection, including language coverage and stated limitations.
• promptfoo red-teaming documentation: vendor docs for adversarial test generation and model-graded evaluation against RAG and agent applications.