PyRIT — the Python Risk Identification Tool for generative AI — is Microsoft’s open-source framework for proactively finding risks in generative AI systems. Where a scanner like garak ships a fixed library of probes and runs them, PyRIT is closer to a programmable red-teaming SDK: you compose attack strategies, target adapters, prompt transformations, and scorers in Python, and the framework runs the loop. It came out of the work of Microsoft’s AI Red Team, which has used it across the company’s own generative AI systems, and it is MIT-licensed and actively maintained.
If garak answers “run the standard probes against my model,” PyRIT answers “automate the specific attack campaign I want to run, including multi-turn strategies, and score the results my way.”
Architecture: the composable pieces
PyRIT’s design is a small set of abstractions that snap together. Understanding them is most of understanding the tool. One naming note first: releases before 0.10 called the attack layer orchestrators (CrescendoOrchestrator, RedTeamingOrchestrator and so on), and the 0.10 refactor replaced them with executors and attacks, which is the model described here.
Targets are adapters to the system under test. PyRIT can drive OpenAI, Azure OpenAI, Anthropic, Google, and Hugging Face models, custom HTTP endpoints and WebSockets, and even web-app targets via Playwright. The target abstraction is what lets the same attack run against a hosted API today and a self-hosted model tomorrow without rewriting the attack.
Datasets are the seed prompts and attack objectives — the harmful behaviors or jailbreak goals you want to test. PyRIT ships datasets and lets you bring your own, which matters because real coverage comes from domain-specific objectives, not just the public set.
Attacks are the attack strategies: the logic that decides what to send, in what order, and how to react to responses. Current releases ship them in pyrit.executor.attack. PromptSendingAttack is the simple single-turn send-and-score case. The multi-turn attacks are where PyRIT’s depth lives: Crescendo escalates a conversation gradually toward the harmful goal so no single turn trips a filter, the red-teaming attack has an adversarial model generate each follow-up from the target’s replies, and TAP (Tree of Attacks with Pruning) explores several attack paths at once and prunes the dead ends. Skeleton Key is packaged as an attack in the same way.
Executors run attacks at scale: an attack executor takes a batch of objectives, runs the chosen attack against each one, and collects the results.
Converters are prompt transformations applied before a prompt is sent — Base64 or other encodings, translation, ASCII art, tone shifts, and dozens more. Converters are the fuzzing layer: they test whether a safety measure that catches a plain attack survives a surface-level reformulation of the same intent. A model that refuses a direct request but complies with its Base64-encoded form has a real vulnerability, and converters surface it.
Scorers evaluate responses. PyRIT supports true/false scorers, Likert-scale scorers, classification scorers, and custom logic, backed by an LLM-as-judge, Azure AI Content Safety, or your own code. The scorer is what turns a pile of responses into a measurable attack success signal.
Memory records every prompt, converted prompt, response and score, persisting conversations to SQLite or Azure SQL so a run is reproducible and reviewable afterward.
The current documentation also covers attack techniques and scenarios, the layer above these pieces; a scenario packages a set of attacks and objectives into a repeatable assessment you can rerun against a new target or model version.
A red-teaming run wires the pieces together: an executor feeds objectives from a dataset to an attack, the attack optionally runs prompts through converters, sends them to a target and hands the responses to a scorer, and memory records every exchange. Swap any one piece and the rest keep working.
Single-turn vs multi-turn is the real differentiator
Most automated LLM scanning is single-turn: send an adversarial prompt, score the response, move on. Many real attacks are not single-turn. Crescendo, for example, never asks for the harmful thing directly — it walks a conversation through a series of innocuous-looking turns, each building on the last, until the model produces the target content without any single message looking like an attack. Single-turn scanners structurally cannot find this class of vulnerability.
PyRIT’s multi-turn attacks carry conversation state, which is exactly what multi-turn strategies need. That is the strongest reason to reach for PyRIT over a single-shot scanner: if your threat model includes a patient adversary working a conversation, you need a tool that can simulate one. For the mechanics of Crescendo and related multi-turn attacks, aisec.blog covers the technique families in operational detail.
A minimal run
PyRIT installs from PyPI:
pip install pyritThe framework is driven from Python rather than a single CLI invocation — you instantiate a target, choose an attack, optionally attach converters, and attach a scorer. Microsoft maintains runnable notebooks and docs at the PyRIT documentation site that walk through single-turn sending and scoring, the Crescendo attack, and custom scorers. Because it is code, a PyRIT campaign is naturally version-controllable: commit the attack configuration, the dataset, and the scorer, and the run is reproducible — a property the AI-security benchmarking discipline cares about deeply, covered at aisecbench.com.
PyRIT also ships a GUI for human-in-the-loop red teaming, for the cases where an operator wants to interact with the target directly, track findings, and collaborate, rather than run a fully automated campaign.
Where PyRIT fits next to garak
These tools are complementary, not competitive:
- garak is the breadth-first probe scanner: a large library of pre-built probes mapped to known attack classes, run with a single command, ideal as a CI gate. See our garak walkthrough for the full breakdown.
- PyRIT is the depth-first red-teaming SDK: programmable, multi-turn, custom-scored campaigns, ideal when you need to model a specific adversary or cover agentic and multi-turn territory garak leaves thinner.
A mature pipeline uses both: garak as the automated regression gate that runs on every model change, and PyRIT for targeted red-team campaigns — multi-turn strategies, domain-specific objectives, custom scorers — that the probe library doesn’t cover. For the broader landscape of where each tool sits, see Best LLM Vulnerability Scanners 2026.
This page is the operational explainer: what each abstraction does and how to compose a campaign from them. For the scored buying verdict, including who should choose PyRIT over a probe scanner and where it is the wrong fit, see aisecreviews.com’s PyRIT review.
What PyRIT is not
PyRIT is a pre-deployment and assessment tool, not a runtime guard. It finds risks before you ship and helps you measure them; it does not sit in the request path filtering live traffic. For runtime input/output screening you need a separate layer — guardml.io covers the guardrail and content-filtering tools that occupy that role. PyRIT also expects you to bring judgment: the scorers, the objectives, and the definition of “harmful” for your product are yours to specify. The framework automates the loop; it does not decide your policy.
Practical Recommendation
Adopt PyRIT when single-turn probe scanning is no longer enough — when your threat model includes multi-turn adversaries, when you need custom scoring tied to your own policy, or when you’re red-teaming an agent rather than a chatbot. Keep garak as the always-on gate. Commit your PyRIT attack configurations, datasets, and scorers to version control so campaigns are reproducible, and pin the target model to a dated snapshot so a result means something next month. For converting attack-success findings into deployment thresholds and business cost, see False Positive Cost in Production Refusal Systems: How to Measure and Tune.
PyRIT is a build-time instrument. It finds problems; it does not enforce anything at runtime, and it never looks at the weight file the model was loaded from. Those are different categories of tool, mapped in LLM Security Tools: The 2026 Category Map. For the broader open-source framework field, see Open Source LLM Red Teaming Tools.
