How to Detect Prompt Injection Vulnerabilities in LLM Apps

Knowing how to detect prompt injection vulnerabilities is not optional once you ship a production LLM application — it is the baseline from which every other defense follows. OWASP ranks prompt injection as LLM01:2025, the top risk for LLM applications, and NIST assigns it two separate formal attack IDs in AI 100-2e2025: NISTAML.018 for direct injection (user-controlled input) and NISTAML.015 for indirect injection (content retrieved from external sources). Both share a root cause — the model treats externally sourced text as trusted instruction — but they surface at different points in the request path and require different detection strategies. This guide covers those strategies in order: input classification, output validation, structural controls, and adversarial testing to validate coverage.

Direct vs. Indirect Injection: Why the Distinction Matters for Detection

Detection architecture should be shaped by the injection surface you’re actually defending.

Direct injection arrives in the user-controlled fields: the chat message, the API payload, the query string. Attackers craft text that overwrites system instructions, extract context, or redirect the model toward unauthorized actions. This is the attack pattern most security teams think of first, and it is well covered by input-layer controls.

Indirect injection is more operationally dangerous. The malicious instruction does not come from the user — it is embedded in content the model retrieves during operation: a web page fetched by a browsing agent, a PDF indexed into your RAG pipeline, a calendar event processed by an AI assistant. The model never “sees” the attack as user input; it encounters it as trusted context. Because indirect injection bypasses input scanners pointed only at user fields, teams that test direct adversarial inputs without testing retrieved content leave their most exposed attack surface unmonitored.

aisec.blog covers the mechanics of indirect injection attacks in detail, including documented cases where LLM agents were manipulated through environmental content.

Because input classifiers pointed at the user field cannot see this traffic at all, two research directions have emerged that detect from inside the model rather than in front of it. The Attention Tracker method (NAACL Findings 2025) observes that a successful injection measurably diverts the model’s attention away from the original instruction toward the injected one, and flags that shift directly from attention patterns with no separate classifier in the path. A 2026 pre-trained-model-plus-heuristic-features approach takes the hybrid route instead, combining a fine-tuned detector with hand-built features rather than trusting a single learned signal. Both are early, and neither is a drop-in product yet, but they address the surface where the deployed controls are weakest.

Input Scanning: Classifiers, Perplexity Filters, and Semantic Heuristics

Four detection approaches dominate the input-scanning layer, and they complement rather than replace each other.

Prompt injection classifiers are fine-tuned models trained on labeled datasets of benign and adversarial prompts. The classifier runs as a pre-filter before the payload reaches the LLM. Lakera Guard — now part of Check Point’s AI security platform after a September 2025 acquisition — uses a proprietary classifier trained on large-scale adversarial datasets from real-world red-teaming and from Lakera’s public Gandalf challenge. Per Lakera’s documentation, Guard scans fetched content, attachments, and URLs for embedded or indirect instructions, including content hidden in HTML and PDFs. Classifier-based detection is fast and broadly effective against known attack patterns but struggles with novel paraphrasing.

The open-weight equivalent is worth knowing because it is the cheapest control in this entire guide to stand up. Meta’s Llama Prompt Guard 2 is the reference implementation: per Meta’s model card it is a BERT-style classifier built on the DeBERTa family, shipping in an 86M-parameter multilingual variant and a 22M English-only variant, emitting a binary benign-versus-attack label. Protect AI’s open deberta-v3-base-prompt-injection models occupy the same slot. Both run in single-digit-to-tens-of-milliseconds on CPU and wire in as a gate with no infrastructure beyond a model server. Their weakness is exactly what you would expect from a classifier trained on known attacks: novel phrasings, obfuscation, and adversarial tokenization (whitespace tricks, fragmented tokens) slip past, which is why Prompt Guard 2 added adversarial-resistant tokenization and an energy-based loss term specifically to harden against those evasions. Treat the score as a signal, not a verdict.

Known-answer detection (KAD) attacks the problem from a different direction and generalizes better than any classifier. You append the untrusted text to a detection instruction that embeds a random secret string known only to your detector, then check whether the model reproduces that secret. If the output fails to echo the key, the model followed an instruction hidden in the data instead of yours, and you flag it. KAD keys on the behavior an injection produces rather than on recognizing the attack’s surface text, so unseen payloads are still caught. The cost is an extra model call per check and sensitivity to where the canary sits in the prompt.

Perplexity filtering exploits the statistical signal that injection payloads often have abnormal token distributions relative to normal user queries. A simple perplexity filter checks overall prompt perplexity against a threshold; a windowed variant analyzes contiguous chunks of text to catch sub-prompt injection within an otherwise normal message. The technique generalizes to obfuscated attacks that evade keyword-based rules, though it produces more false positives on technical content and multi-lingual inputs.

Embedding-based classifiers generate vector representations of incoming text and compare them to a trained binary boundary separating benign from malicious samples. The approach allows detection of semantically similar injection variants without exact-match rules, but it requires a labeled training set that reflects your specific deployment’s attack surface — a generic classifier trained on open benchmarks may underperform on domain-specific injections your application actually faces.

A critical finding from WAInjectBench, a systematic benchmark of detection approaches for web agents, should temper confidence in any single method: detection accuracy is “moderate to high” against attacks with explicit textual instructions, but existing detectors “largely fail against attacks that omit explicit instructions or employ imperceptible perturbations.” Sophisticated attackers will explore that gap.

Output Validation: Catching What Passes the Input Gate

Input classifiers reduce the attack surface; they do not eliminate it. Output validation provides a second detection layer.

The RAG Triad — context relevance, groundedness, and answer relevance — is a structured evaluation framework that checks whether a model’s response is consistent with the retrieved context and the original user request. A response that suddenly references an “override instruction” or attempts to call a tool not relevant to the query is a detection signal. OWASP recommends “use deterministic code to validate adherence to” expected output formats, and explicit JSON schema enforcement can catch responses that deviate from expected structure before they reach downstream systems.

The rest of the output-side checklist is short and worth running in full: structured-output mode so a manipulated model cannot emit free-form instructions; secret and PII scanning to catch exfiltration in progress; and a tool allowlist plus human approval for any high-impact action. That last item is the one that converts a detection miss into a contained event rather than an incident, and OWASP recommends it for exactly that reason.

guardml.io covers production implementations of output guardrails including schema enforcement and semantic output classifiers across common LLM deployment patterns.

Structural Controls: Spotlighting and Content Segregation

The highest-confidence detection-adjacent control in the current literature is not a classifier at all — it is structural prompt design that makes injections easier to detect and harder to execute. Microsoft Research’s “spotlighting” technique uses explicit delimiters and XML tags to mark the boundary between trusted instructions and untrusted retrieved content. Published evaluations showed this reduced indirect injection success rates from above 50% to below 2% in the tested scenarios. The mechanism is simple: when the model is explicitly told that everything inside a <user_content> tag is untrusted data and not instruction, the model is less likely to treat embedded instructions as legitimate.

Content segregation at the architecture level — processing untrusted content in a separate context from trusted instructions — extends this further. Applying least-privilege access controls to what tools an agent can invoke after processing external content limits the blast radius when detection fails.

Red-Team Testing to Validate Coverage

No detection stack is complete without adversarial validation. NIST AI 600-1 (the Generative AI Profile) requires organizations to run adversarial testing, log every interaction, and review detection rates against defined benchmarks quarterly. In practice, this means:

• Testing both direct and indirect injection surfaces, including RAG-indexed documents, email processors, and web-browsing agent inputs.
• Using a red-team tool such as Garak or PyRIT to probe your input scanner with encoding variations, paraphrases, and multi-lingual payloads that exploit known classifier blind spots.
• Validating that output validators catch responses that comply syntactically but deviate semantically from expected behavior.
• Logging all flagged events with enough context to reconstruct the original payload, the response, and what action (if any) the model attempted — this audit trail is mandatory under the NIST AI RMF MEASURE function.

NIST is explicit that “current mitigations do not offer full protection against all attacker techniques.” Treat detection coverage as a percentage, not a binary, and define an explicit residual risk threshold your organization accepts.

Assembling the Layers

Put end to end, a defensible detection stack sits in the request path roughly like this: the inbound prompt hits a classifier gate and, on sensitive flows, a KAD probe; retrieved context passes a perplexity pre-filter and is structurally tagged as untrusted with spotlighting delimiters; the primary model runs under least privilege with a scoped tool allowlist; the output passes schema validation, groundedness scoring, and secret/PII scanning before any tool fires or any text returns to the user.

Two caveats keep that honest. First, every detector has a false-positive rate, and stacking them multiplies friction, so tune thresholds against your own traffic rather than a vendor’s benchmark, because over-defensive guardrails that block legitimate prompts are a documented failure mode with its own cost. Our false-positive cost guide covers how to put a number on that. Second, every layer here is bypassable. Empirical evasion studies show classifier and jailbreak detectors fall to adversarial perturbation, so detection is a control that raises attacker cost and narrows the window, not one that closes the class. Pair it with privilege restriction and human approval, log every flagged event for audit, and red-team the whole chain on a schedule with garak or PyRIT.

For the classifier technology itself (how embedding-based detectors, fine-tuned guardrail models, and intrinsic-feature monitoring differ mechanically), aisecreviews.com’s detection explainer goes a layer deeper than this guide does.