Hawestra/scenario blog by hannahwestra25 · Pull Request #3 · hannahwestra25/PyRIT

hannahwestra25 · 2026-06-04T16:27:59Z

Description

Tests and Documentation

Covers AttackTechnique abstraction (microsoft#1592), AttackTechniqueRegistry (microsoft#1611), standardized attack args (microsoft#1608), and v0.14.0 scenario improvements including the new TextAdaptive scenario and EpsilonGreedyTechniqueSelector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restructure around the framework/scanner/GUI three-layer story, add a tour of the scenario catalog with a RapidResponse highlight, tighten the v0.13/v0.14 updates section, and shorten the adaptive scenarios section to the demo-script framing (registry + selector + ASR + cross-run learning). Note GUI roadmap for scenarios in the wrap-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove the 'Where scenarios fit' section; fold the scanner mention into the intro. - Drop the colored dots from scenario flavor headers. - Reframe the RapidResponse opener as a broad starter scan / jumping-off point for deeper testing, rather than an urgent-incident scenario. - Expand the v0.13/v0.14 section with concrete usage detail for AttackTechnique, AttackTechniqueRegistry, Better Scenario Tracking, and SequentialAttack. - Audit baseline language so baseline reads as a generic Scenario feature, not a RapidResponse- or adaptive-specific behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T16:30:15Z

+## What's in a scenario
+
+Cracking one open, every scenario bundles four things:
+


what about a strategy ?

Good call — added a new 'Strategies — the runtime knob' bullet here so the strategy/--strategies concept gets introduced up front instead of only appearing in the RapidResponse section.

hannahwestra25 · 2026-06-04T16:32:25Z

+
+### A closer look: `RapidResponse`
+
+`RapidResponse` is the broadest of the AIRT scenarios — a comprehensive sweep across the most common techniques and the full AIRT harm-category catalog. It's a natural jumping-off point: you run it to get a wide, shallow read on where a target is strong and where it's soft, then pivot to the more focused AIRT scenarios (or to `TextAdaptive`) to dig into the categories or techniques that came back interesting.


"where it's soft" sounds weird

Reworded to 'which categories the target handles well and which ones come back concerning.'

hannahwestra25 · 2026-06-04T16:32:49Z

+
+`RapidResponse` is the broadest of the AIRT scenarios — a comprehensive sweep across the most common techniques and the full AIRT harm-category catalog. It's a natural jumping-off point: you run it to get a wide, shallow read on where a target is strong and where it's soft, then pivot to the more focused AIRT scenarios (or to `TextAdaptive`) to dig into the categories or techniques that came back interesting.
+
+It crosses two axes: **techniques × harm categories**. The technique side pulls from seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`). The harm side covers seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.


don't all scenarios cross two axes ?

True — every scenario crosses techniques × datasets, so that framing was misleading. Dropped 'It crosses two axes' and reworded so the actually-RR-specific bit (results grouped by harm category rather than by technique) is what's called out as the distinguishing feature.

hannahwestra25 · 2026-06-04T16:33:55Z

+
+## What's improved in v0.13.0 and v0.14.0
+
+A lot of the recent work has been less about new scenarios and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.


Suggested change

A lot of the recent work has been less about new scenarios and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.

A lot of the recent work has been less about building out our scenario library and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.

Applied. Also added a follow-on sentence noting that v0.14 did still ship two new scenarios (RapidResponse and AdversarialBenchmark) so the section doesn't undersell the catalog growth — but the framing is now that machinery was the bulk of the work.

hannahwestra25 · 2026-06-04T16:34:51Z

+
+A lot of the recent work has been less about new scenarios and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.
+
+**A real abstraction for "an attack you can drop into a scenario."** Before v0.13.0 a scenario glued attacks together by hand, knowing how to construct each one and what arguments it needed. [`AttackTechnique`](https://github.com/microsoft/PyRIT/pull/1592) replaced that with a single bundle: the attack strategy class, the `seed_technique` configuration that tells the attack how to mutate prompts (jailbreak template, encoding, role-play wrapper, etc.), plus any technique-specific defaults like which adversarial chat to use. A scenario now composes a *list* of `AttackTechnique`s and hands them to the executor — the scenario doesn't need to know the internals of TAP versus Crescendo versus a converter-based attack, just that it has techniques to run.


remove all PR links

Done — stripped every github.com/.../pull/ link throughout the post.

hannahwestra25 · 2026-06-04T16:36:45Z

+
+**A catalog those techniques live in.** [`AttackTechniqueRegistry`](https://github.com/microsoft/PyRIT/pull/1611) is where techniques register themselves with metadata: name, description, tags like `default` / `single_turn` / `multi_turn` / `light`, modality, what kinds of targets they work against. Scenarios pull techniques out via tag queries — `TagQuery.any_of("default")`, `TagQuery.all_of("multi_turn", "text")` — instead of importing each one by name. The scanner CLI uses the same registry to list and describe what's available (`pyrit_scan list-techniques`). And to add your own, you write a factory and register it with a tag; every scenario that queries by that tag picks it up for free. Attack arguments were [standardized in the same release](https://github.com/microsoft/PyRIT/pull/1608), which is what lets the registry instantiate techniques generically — every constructor speaks the same dialect.
+
+**Attribution that survives runs.** [Better Scenario Tracking](https://github.com/microsoft/PyRIT/pull/1758) added a scenario-run ID that gets stamped onto every `AttackResult` row the run produces. That sounds small but unlocks a lot: when you resume a partially-completed scenario (via `scenario_result_id`), the framework can ask memory "which objectives already have results for this run?" and skip them without double-counting; cross-run analytics like "how did `RedTeamAgent` do on this target across our last ten scans?" stop needing manual labeling; and the printer can roll results up to the correct scenario invocation instead of mixing in unrelated history sitting in the same database. This is also what makes the adaptive selector's cross-run learning trustworthy — it can scope its history queries cleanly.


it might be helpful to have a diagram of how these interact

Added a mermaid diagram inline in the Better Scenario Tracking paragraph showing scenario_run_id getting stamped onto AttackResult rows and flowing through memory to resume / cross-run analytics / printer rollup / adaptive selector scoping.

hannahwestra25 · 2026-06-04T16:36:58Z

+
+**Sorted breakdowns and a new compound primitive.** Scenario printers now [sort the per-group breakdown by success rate](https://github.com/microsoft/PyRIT/pull/1809), so the categories the target is most vulnerable on float to the top of the report instead of being buried alphabetically. And [`SequentialAttack`](https://github.com/microsoft/PyRIT/pull/1819) shipped as a compound `AttackStrategy`: give it a list of attacks and a `SequenceCompletionPolicy` (`FIRST_SUCCESS` stops as soon as one lands, `EXHAUSTIVE` runs them all, `FIRST_DECISIVE` stops on the first clear pass-or-fail), and it runs them in priority order. Adaptive scenarios use this under the hood, but it's available as a standalone building block any time you want "try the cheap thing first, escalate only if needed."
+
+**Smaller polish worth knowing about.** `include_baseline` moved from the `Scenario` constructor to `initialize_async` ([#1700](https://github.com/microsoft/PyRIT/pull/1700)) so you decide at run time, not construction time. `BASELINE_POLICY` was renamed to `BASELINE_ATTACK_POLICY` ([#1763](https://github.com/microsoft/PyRIT/pull/1763)) so the name says what it controls. The `FoundryScenario` alias was removed in favor of `RedTeamAgent` ([#1623](https://github.com/microsoft/PyRIT/pull/1623)). And — possibly the most useful change for new users — [scenario doc pages for the seven scenarios that didn't have them](https://github.com/microsoft/PyRIT/pull/1558) finally exist.


hannahwestra25 · 2026-06-04T16:37:39Z

+
+Three pieces make it work:
+
+- **The registry** — the same `AttackTechniqueRegistry` from v0.13.0. It's the catalog of available techniques the scenario can pick from.


mention something about asr ?

Pulled ASR into the opening paragraph of the adaptive section so it's introduced before the 'three pieces make it work' bullets (which now reference ASR by its acronym instead of spelling it out for the first time).

hannahwestra25 · 2026-06-04T16:38:30Z

+
+- The scenarios docs landing page: [`doc/code/scenarios/`](../code/scenarios/0_scenarios.ipynb).
+- The end-to-end walkthroughs from the scanner side: [`pyrit_scan`](../scanner/1_pyrit_scan.ipynb) and [`pyrit_shell`](../scanner/2_pyrit_shell.md).
+- The adaptive scenarios notebook (shipped alongside `TextAdaptive`) is the fastest way to see the bandit in action against a real target.


add link to notebook

Done — pointed the bullet at doc/code/scenarios/3_adaptive_scenarios.ipynb.

- Drop 'two people running the same pass...' sentence; switch opener to 'Enter scenarios.' - Add a 'Strategies - the runtime knob' bullet so the runtime-selection concept is introduced before it's used in RapidResponse / adaptive sections. - Add adaptive scenarios as a fifth flavor in the catalog with a link to its section; mark RapidResponse and AdversarialBenchmark as new in v0.14. - Rework RapidResponse closer-look: replace 'where it's soft' phrasing, drop the 'crosses two axes' framing (every scenario does that), keep the grouping detail as the actually-RR-specific bit. - Apply the 'less about building out our scenario library' wording, then explicitly surface the two new v0.14 scenarios so the section isn't misleading about scope. - Strip every github.com PR link. - Add two new paragraphs: 'Configuration from the CLI and from YAML' and 'Parallel execution within a scenario' to cover microsoft#1680 and microsoft#1783. - Add a mermaid diagram to the Better Scenario Tracking paragraph showing scenario_run_id flowing into memory and out to resume / analytics / printer / adaptive selector. - Drop the 'Smaller polish worth knowing about' paragraph. - Pull ASR mention into the adaptive section's opening paragraph. - Point the 'Where to go next' bullet at the real adaptive notebook path (doc/code/scenarios/3_adaptive_scenarios.ipynb). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…, and TextAdaptive bandit trail Three TODO IMAGE blocks added (all HTML-commented so the docs build stays green): 1. After the 'There are five flavors' opener: pyrit_scan list-scenarios catalog screenshot (2026_06_04_scan_list_scenarios.png). 2. End of the RapidResponse 'A closer look' section: ConsoleScenarioResultPrinter summary showing baseline + per-harm-category breakdown sorted by success rate (2026_06_04_rapid_response_output.png). 3. After the TextAdaptive code block: per-objective technique trail + wins/picks/ rate summary, captured from doc/code/scenarios/3_adaptive_scenarios.ipynb (2026_06_04_text_adaptive_output.png). To activate any of them: drop the PNG into doc/blog/ with the indicated filename and uncomment the markdown image line inside the comment block. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ructure Moved the diagram from the 'Attribution that survives runs' section (where the surrounding prose already covers the same ground) to the 'A catalog those techniques live in' paragraph, where it explains the v0.13 abstraction story visually: techniques register with tags into AttackTechniqueRegistry, TagQuery subsets become ScenarioStrategy enum members, --strategies picks one at run time, and the scenario fans the chosen techniques out across its datasets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…taset_names (microsoft#1911) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ario (microsoft#1936) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…icrosoft#1938) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T18:57:09Z

+
+<small>4 Jun 2026 - Hannah Westra</small>
+
+When we first introduced scenarios in PyRIT a few releases back, the pitch was pretty simple: most of the operators we talked to were assembling the same set of pieces — a target, a curated dataset of objectives, a few attack strategies, a scorer — and running them in a loop. The framework gave them all the right Lego bricks, but every team was clicking those bricks together by hand. There was no clean unit of work to point at when you wanted to say, "run this exact configuration, then compare it against that one."


is it obvious who operators are ? is that a common term for red teamers ? do we use that terminology in the rest of the repo / website ?

Good catch. operator is only used once in the broader docs and never with much consistency — the repo otherwise leans on red teamer / red teaming. Swapped both occurrences (the opening paragraph and the strategies bullet) so the terminology is consistent with framework.md and the rest of the site.

hannahwestra25 · 2026-06-04T19:05:21Z

+
+Cracking one open, every scenario bundles five things:
+
+- **Techniques — the *how*.** How are we going to attack? Maybe we just send the prompt directly. Maybe we wrap it in a role-play scenario. Maybe we escalate over multiple turns with Crescendo or TAP. Techniques include the attack strategy plus its converters, jailbreak templates, and adversarial-chat configuration — basically all the knobs that affect how the attack is crafted and delivered.


i don't think role-play is a scenario ? this is confusing.

You're right — scenario is already a defined term in this post, so saying wrap it in a role-play scenario overloads it mid-sentence. Changed to wrap it in a role-play prompt template which matches the actual mechanism (RolePlayAttack with a RolePlayPaths template) without colliding with the scenario terminology.

hannahwestra25 · 2026-06-04T19:06:16Z

+
+Scenarios can also opt in to running a **baseline pass** — sending the raw, unmodified prompts as a control group so the report can show "what the model does without any attack" next to "what the model does once attacked." Baseline behavior is generic to the `Scenario` base class (toggled per run via `include_baseline` and governed by `BASELINE_ATTACK_POLICY`); individual scenarios just decide whether it makes sense for their workload.
+
+The whole point is that you don't have to wire any of this up yourself. Pick a scenario, point it at a target, and the scenario handles the rest.


add a screenshot of a pyrit_scan command to run a scenario and explain what it's doing

Done. Added a fourth screenshot placeholder right after the pick a scenario, point it at a target paragraph: 2026_06_04_scan_run_scenario.png. The capture instructions in the placeholder ask for the command at the top + the initializer log lines + the first technique progress bars (the final summary is already captured in 2026_06_04_rapid_response_output.png, so this one is specifically the what does it look like to kick one off frame). I also added an explanatory paragraph immediately after the image that walks through what the one CLI command actually does end-to-end (initializers populate registries -> scenario lookup -> target resolution -> run_async).

hannahwestra25 · 2026-06-04T19:16:16Z

+
+`RapidResponse` is the broadest of the AIRT scenarios — a comprehensive sweep across the most common techniques and the full AIRT harm-category catalog. It's a natural jumping-off point: you run it to get a wide, shallow read on which categories the target handles well and which ones come back concerning, then pivot to the more focused AIRT scenarios (or to `TextAdaptive`) to dig into whatever came back interesting.
+
+The technique side pulls from seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`). The dataset side covers seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.


i think the "technique side pulls" wording is confusing here

Agreed, the technique side / dataset side framing was clunky. Collapsed it into one sentence: It runs seven core techniques (...) across seven AIRT datasets (...). That gets both lists across without the awkward two-axis personification.

hannahwestra25 · 2026-06-04T19:18:28Z

+
+The technique side pulls from seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`). The dataset side covers seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.
+
+Like other scenarios it can run a baseline pass first (`include_baseline=True`, the default) — the raw prompts with no converters or wrapping techniques — so the report has a control group sitting next to the attacked numbers. Then it runs every selected technique against every selected dataset, in parallel, through the execution engine. The thing that's specific to `RapidResponse` is the *grouping*: results roll up by harm category rather than by technique, because when leadership asks "are we exposed to hate speech?", you want the answer organized that way.


explain what grouping by harm category vs technique means. are other scenarios grouped by technique? I think we can omit the part about baseline here.

Both done. (a) Removed the baseline mention from this paragraph — baseline is a generic Scenario feature and is already covered in the What's in a scenario section a few paragraphs up, so re-explaining it here was redundant. (b) Yes, the default base Scenario._build_display_group returns the technique name, so most scenarios print one block per technique; RapidResponse overrides _build_display_group to return the dataset (seed_group_name) instead. Rewrote the paragraph to spell that out concretely with two example formats (crescendo_simulated: 14/50, role_play: 7/50 vs airt_hate: 21/100, airt_violence: 4/100) and a one-liner noting it's the same AttackResult rows in memory, just a different rollup axis on the way out.

hannahwestra25 · 2026-06-04T19:19:34Z

+
+The full path from a single technique to a scenario run looks like:
+
+```mermaid


diagram sucks. it should lay out how the attack technique relates to attack strategies relates to a scenario relates to an atomic attack etc.

Fair. Rewrote the diagram to cover the full chain instead of just the registry-to-strategy slice. It now shows AttackStrategy + seed configuration -> AttackTechnique -> AttackTechniqueRegistry (with the ScenarioTechniqueInitializer arrow making it explicit who populates the registry) -> ScenarioStrategy -> Scenario (+ its datasets feeding in) -> AtomicAttack -> ScenarioResult. Switched to flowchart TB since there are too many levels for an LR layout to stay readable. Also added a short follow-on paragraph that defines AttackStrategy, AttackTechnique, and AtomicAttack in plain English since the diagram now uses all three terms.

hannahwestra25 · 2026-06-04T19:20:33Z

@@ -0,0 +1,176 @@
+# Scenarios: Where We Started, and Where We're Going
+


should we incorporate initializers somewhere ? also i think i want more on the why of scenarios and rapid response and adaptive scenarios

Two things in this one: (a) Initializers. Added them in two places. They get a sentence in the A catalog those techniques live in paragraph (Population of the registry itself is handled by initializers — the canonical one is ScenarioTechniqueInitializer...), show up explicitly on the registry arrow in the rebuilt mermaid diagram, and the new pyrit_scan paragraph after line 23 walks through how ScenarioTechniqueInitializer, TargetInitializer, and LoadDefaultDatasets populate their registries before the scenario itself runs. (b) More on the why. Expanded the opening paragraph (the Lego bricks paragraph) to call out the concrete pains scenarios solved — rerun after a model update, hand a config to a teammate, diff against last quarter — instead of the vague no clean unit of work. Expanded the RapidResponse opener to name the trigger moments (new model onboarded, new model release, new vulnerability in the news) and what the question being asked actually is at those moments. Expanded the adaptive opener to make the cost of brute force concrete (latency, API rate-limit budget, adversarial-chat tokens) so the motivation isn't just wasted attempts in the abstract.

- Replace 'operators' with 'red teamers' to match repo terminology. - Reword 'role-play scenario' bullet to 'role-play prompt template' so the word scenario isn't overloaded mid-sentence. - Add a fourth screenshot placeholder + explanatory paragraph showing what 'pyrit_scan airt.rapid_response --target my_target' actually does at the CLI; this is also where the post introduces initializers (ScenarioTechnique- Initializer, TargetInitializer, LoadDefaultDatasets) as the things that populate the registries before run_async fires. - Reword the RapidResponse 'technique side pulls / dataset side covers' sentence as a single 'It runs N techniques across N datasets' line. - Drop the baseline mention from the RapidResponse paragraph (already covered in 'What's in a scenario') and replace it with a concrete explanation of what 'group by harm category vs by technique' means in the printer. - Rebuild the mermaid diagram. It now spans the full chain: AttackStrategy + seed config -> AttackTechnique -> AttackTechniqueRegistry (via Scenario- TechniqueInitializer) -> ScenarioStrategy -> Scenario + datasets -> AtomicAttack -> ScenarioResult, with a follow-on paragraph that defines AttackStrategy, AttackTechnique, and AtomicAttack in plain English. - Expand the opening paragraph and adaptive intro slightly with more 'why' (reproducibility, comparing runs, real cost of brute-force scans). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ng/TAP (microsoft#1377) Co-authored-by: Robert Fitzpatrick <fitzpatrickr@microsoft.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com> Co-authored-by: Robert Fitzpatrick <robertfitzpatrick@Roberts-MacBook-Pro.local> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…cript for release (microsoft#2028)

…t#2128)

…ft#2130) Co-authored-by: Behnam Ousat <behnamousat@microsoft.com>

Co-authored-by: Behnam Ousat <behnamousat@microsoft.com>

…rom memory (microsoft#2120) Co-authored-by: Behnam Ousat <behnamousat@microsoft.com>

)

…rosoft#2137) Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com> Co-authored-by: Richard Lundeen <rlundeen@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…icrosoft#2134) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…rosoft#2111) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…icrosoft#2132) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…lper (1/3) (microsoft#2125) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tch group (microsoft#2149) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

… and attacks (microsoft#2140) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…atch group across 1 directory (microsoft#2148) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…icrosoft#2151) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ft#2145)

…date Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

hannahwestra25 and others added 3 commits June 4, 2026 11:28