LLM Pentesting Checklist: 8 Test Areas Mapped to OWASP

A usable LLM pentesting checklist covers eight test areas: direct prompt injection and jailbreaks, indirect injection, hidden-context and data leakage, output handling, tool and agent authorization, RAG and vector-store isolation, resource abuse, and model supply chain. Scope the system first, seed canaries, automate the repeatable probes with garak or promptfoo, and test authorization and agent logic by hand.

The list is built for an engagement, not a scanner run. Test IDs come from the OWASP AI Testing Guide, which defines 32 numbered tests across application, model, infrastructure and data pillars. Each area also maps to the OWASP GenAI LLM Top 10 2026 and MITRE ATLAS, so findings land in the vocabulary a risk register already uses. For a per-entry scanner view instead, see the OWASP LLM Top 10 scanning checklist.

What do you need to scope before testing an LLM app?

Scope the whole request path, not just the model: model and version, system prompt, every tool the model can call and the credentials behind it, every retrieval source, conversation memory, the sinks where output lands, and any guardrail in front. A test that skips one of these reports on a system nobody actually runs.

The OWASP GenAI Red Teaming Guide splits the same ground into model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. A model-only engagement covers one of the four. Before the first prompt:

• Inventory the context window. System prompt, developer instructions, retrieved chunks, tool schemas and memory all reach the model. Treat every piece as discoverable.
• List each tool with its real permissions. Record what it can read, write, delete or send, whose credentials it runs under, and whether authorization is enforced server-side or left to the model.
• Map output sinks. Browser rendering, markdown images and links, SQL, shell, email and downstream agents each need their own payloads.
• Record the guardrails in path. Decide whether you are testing the model, the prompt-scanning layer or the composite, and test with the filter bypassed where the rules of engagement allow.
• Agree rules of engagement. Set a token and cost ceiling, use a non-production tenant, list which tool actions may execute for real, and fix transcript retention.
• Seed canaries. Plant unique fake secrets in the system prompt, one RAG document and a second test tenant’s data. A canary in output is an unambiguous finding; a model that merely seems to leak is not.

The checklist: eight areas mapped to OWASP, ATLAS and AITG

OWASP IDs follow the 2026 edition. On 2025 numbering, LLM08 Hidden Context Exposure was LLM07 System Prompt Leakage, and LLM03 Excessive Agency was LLM06. “n/a” marks areas where no single ATLAS technique maps cleanly, so tag those findings by OWASP ID.

1. Direct prompt injection and jailbreaks

AITG-APP-01 lists 23 payload families, from role-play and Base64 encoding to payload splitting, low-resource languages and best-of-N sampling. Cover at least instruction override, persona attacks, encoding and obfuscation, and multilingual variants. The guide notes that responses vary between identical requests, so repeat each payload and record a success rate rather than one pass or fail.

Include multi-turn tests. Crescendo, described by Russinovich, Salem and Eldan, opens with benign questions and escalates by referencing the model’s own replies, so a single-turn probe set cannot find it. PyRIT is the usual choice for scripted multi-turn campaigns, and detecting prompt injection vulnerabilities covers the defensive follow-up.

2. Indirect prompt injection

Plant payloads everywhere the model reads text the user did not type: uploaded files, web pages an agent browses, inbound email, tool responses, RAG documents and messages from other agents. Greshake et al. showed in 2023 that retrieved prompts can control whether and how an application calls other APIs. Two checks matter most. Can a retrieved document trigger a tool call? Can it make the model emit a markdown image or link whose URL carries conversation data? garak’s xss probes target that exfiltration pattern, per its README. Hide payloads in HTML comments, white-on-white text and, for multimodal apps, inside images.

3. Hidden context and data leakage

Ask for the system prompt directly and obliquely: translate it, summarize it, emit it as JSON. Ask the model to list its tools and their parameters, then search every output for your canaries. Test cross-session and cross-user leakage through memory features, and run garak’s leakreplay probes if training-data replay is in scope. Severity follows content: a disclosed prompt with no secrets is low, while one holding an API key or serving as the only access control is high.

4. Output handling

Treat model output as untrusted input and render it into the real sinks. Check for script execution through HTML or markdown, SQL and shell injection wherever output reaches an interpreter, and SSRF wherever output becomes a URL the server fetches. promptfoo ships shell-injection, sql-injection and ssrf plugins, per its MITRE ATLAS mapping docs. If the application writes code, garak’s packagehallucination probe checks for references to packages that do not exist, which an attacker can then register.

5. Tools and agent authorization

The highest-impact findings usually sit here. AITG-APP-06 tests direct tool invocation with attacker-chosen arguments, stop and shutdown overrides, budget and circuit-breaker limits, sandbox escape through code tools, use of the current session to reach another user’s data, and memory poisoning. Its pass criteria are concrete: the agent halts on termination commands, uses only permitted tools, enforces step, time, token and cost limits, and requires explicit authorization for sensitive operations. promptfoo’s excessive-agency, rbac and hijacking plugins automate a first pass. The core question is whether authorization lives in the tool or in the prompt; if only the system prompt stops the model calling delete_user for anyone, the control is bypassable.

6. RAG and vector-store isolation

Query as tenant B for tenant A’s canary. Confirm document ACLs are enforced at retrieval time, not filtered after generation. If users can write to the corpus, test whether a crafted document ranks for common queries and carries an injected instruction into other users’ sessions; AITG-APP-08 covers embedding manipulation.

7. Resource abuse

Send oversized inputs, repetition-inducing prompts and requests that push agents into loops, then confirm per-user token caps, rate limits and cost ceilings trigger. ATLAS separates Denial of AI Service (AML.T0029) from Cost Harvesting (AML.T0034). On pay-per-token APIs the second is often the bigger exposure, because the attacker sets per-request cost.

8. Model supply chain

This is a build-time check, not a prompt-time one. Verify hashes and provenance for base models and adapters, flag pickle-based model formats that can execute code on load, and keep an SBOM for the serving stack.

Which checks can you automate?

Automate most of areas 1 through 4, plus area 7, as a regression suite, and keep areas 5 and 6 largely manual. Probe libraries handle known injection, encoding, leakage and output payloads well, but they cannot know which tool call is unauthorized for which user in your business logic, or which tenant boundary guards regulated data.

garak splits work into probes, detectors and generators under an Apache-2.0 license, which suits a gate that runs every probe on every model change. promptfoo’s mitre:atlas shorthand runs plugins grouped by ATLAS tactic, and its docs list the AI Model Access tactic with no plugins, a coverage gap worth copying into your report. Run each automated probe several times at production temperature so the success rate means something, and wire the suite into CI as described in garak vs PyRIT vs promptfoo in CI. If remediation tightens a guardrail, measure what it starts refusing too; refusal false-positive cost is the usual blind spot.

How should you write up LLM pentest findings?

Write each finding so another tester can reproduce it and a risk owner can rank it: full transcript, model and version, temperature, a hash of the system prompt, guardrail configuration and the success rate over repeated attempts. Rank severity by what the model could do or reach, not by how offensive the text was.

A jailbreak that yields rude text from a tool-less chatbot is usually a lower-severity finding than an indirect injection that makes an agent send email. Tag each finding with its OWASP 2026 ID and ATLAS technique so remediation tracking and threat mapping share the same keys.

State the residual risk plainly. A clean result is point-in-time: a model upgrade, a system-prompt edit or new retrieved content can reopen a closed finding, which is why the automated half belongs in CI and the manual half on a change-triggered schedule. For governance mapping, the NIST Generative AI Profile (NIST AI 600-1), published in July 2024, applies the AI Risk Management Framework to generative systems, and recurring pentests sit under its Measure function.

FAQ

what is llm pentesting

LLM pentesting is an authorized, scoped attack on an application built around a large language model, aimed at proving concrete impact: leaked secrets, unauthorized tool calls, cross-tenant data access or injected output reaching a browser or database. It differs from model red teaming for harmful content because it targets the whole system around the model.

how is llm pentesting different from web app pentesting

The biggest difference is nondeterminism: the same payload can fail repeatedly and then succeed, so findings are reported as success rates over repeated attempts. The input space is natural language rather than a parameter grammar, and the model can act as a confused deputy for every tool and data source it reaches. Web tests still apply around it.

what tools are used for llm penetration testing

Most LLM penetration tests combine an open-source probe library with manual work. NVIDIA’s garak and promptfoo cover known injection, jailbreak, leakage and output-handling payloads; Microsoft’s PyRIT scripts multi-turn campaigns. An intercepting proxy still handles the HTTP layer, and tool authorization and tenant isolation are usually tested by hand.

how often should you pentest an llm application

Run a full manual LLM pentest before launch and after any major architectural change, such as new tools, a new retrieval source or a model swap, and run the automated portion on every deploy. Model updates, system-prompt edits and new retrieved content can reopen findings a quarterly or annual test would miss.