draft: text adaptive scenario

[hannahwestra25]

Add Adaptive Scenario Framework with TextAdaptive

Summary

Introduces an adaptive scenario framework that picks attack techniques per-objective using an epsilon-greedy bandit informed by observed success rates, rather than running every selected technique against every objective. Concentrates spend on techniques that actually work against the target, and stops early on first success.

Adds:

• AdaptiveScenario — modality-agnostic base class
• TextAdaptive — concrete text-attack subclass
• AdaptiveTechniqueSelector — epsilon-greedy selector with Laplace-smoothed estimates and pooled cross-context backoff
• AdaptiveDispatchAttack — per-objective dispatch strategy
• Walkthrough notebook + .py doc
• Unit tests (63 tests across selector, dispatcher, and scenario)

Motivation

Static scenarios are O(techniques × objectives): every technique runs against every objective regardless of whether earlier attempts already succeeded or whether the technique is known to be ineffective against the target. For evaluation runs with many techniques and many objectives, this wastes spend on combinations that aren't informative.

Adaptive scenarios reduce this to O(max_attempts × objectives) by:

• learning from observed outcomes,
• exploiting techniques that work on the target,
• still exploring (with probability epsilon) so the table doesn't collapse onto a single technique prematurely,
• stopping per-objective on first success.

How it works

For each objective the dispatcher loops up to max_attempts_per_objective times:

1. Select — with probability epsilon pick a random technique, otherwise pick the one with the highest Laplace-smoothed success estimate (s + 1) / (n + 1). Cells with fewer than pool_threshold local observations fall back to the technique's pooled rate across all contexts (cold-start handling).
2. Execute — run the chosen technique against the bound SeedAttackGroup, merging the technique's seed_technique if it declares one.
3. Record — update the selector's (context, technique) → (successes, attempts) table and stop early on success.

The selector is shared by reference across all per-objective dispatchers in a scenario run, so learning accumulates globally. The per-objective context key is derived by a ContextExtractor; global_context (default) shares one table across all objectives, harm_category_context partitions by harm category.

Public API

from pyrit.scenario.scenarios.adaptive import TextAdaptive, harm_category_context

scenario = TextAdaptive(
    epsilon=0.2,
    pool_threshold=3,
    max_attempts_per_objective=3,
    seed=42,                                  # reproducible selection
    context_extractor=harm_category_context,  # optional per-category learning
)
await scenario.initialize_async(objective_target=target)
result = await scenario.run_async()

Adaptive scenarios are also resumable — pass scenario_result_id="..." to the constructor and prior dispatch trails are replayed into the selector before the remaining objectives run.

notes

• BASELINE_POLICY = Forbidden — prompt_sending participates as one of the selector's techniques rather than being prepended as a guaranteed baseline. Mixing a forced baseline with adaptive selection would bias the table.
• Per-objective compatibility filtering — techniques whose seed_technique is incompatible with a given seed group are dropped per-objective rather than globally, so a technique that's incompatible with one objective still participates for others. Objectives with no compatible techniques are skipped with a warning.
• Resume rehydration — each attempt is persisted as a step in metadata["adaptive_attempts"] on the outer AttackResult; on resume these trails are replayed via record_outcome so the new selector starts with the prior run's learned state. Already-completed atomics are skipped by the base Scenario resume path (by atomic_attack_name), so there is no double-counting.
• Two-row persistence per success — the inner technique persists its raw AttackResult via its own post-execute hook; the dispatcher returns a replace-based copy with a fresh attack_result_id/timestamp and the adaptive trail stamped onto metadata. Both rows share conversation_id. Documented on the dispatcher's class docstring.
• Thread safety — AdaptiveTechniqueSelector guards its counts table with a threading.Lock so individual select / record_outcome operations are atomic. The overall select → execute → record sequence is not serialized.

Files

Area	File
Base scenario	pyrit/scenario/scenarios/adaptive/adaptive_scenario.py
Text subclass	pyrit/scenario/scenarios/adaptive/text_adaptive.py
Selector + context extractors	pyrit/scenario/scenarios/adaptive/selector.py
Per-objective dispatcher	pyrit/scenario/scenarios/adaptive/dispatcher.py
Package wiring	pyrit/scenario/scenarios/adaptive/__init__.py, pyrit/scenario/__init__.py
Walkthrough	doc/code/scenarios/3_adaptive_scenarios.py, doc/code/scenarios/3_adaptive_scenarios.ipynb
Tests	tests/unit/scenario/scenarios/adaptive/test_selector.py, tests/unit/scenario/scenarios/adaptive/test_dispatcher.py, tests/unit/scenario/scenarios/adaptive/test_text_adaptive.py

Testing

pytest tests/unit/scenario/scenarios/adaptive/ — 63 tests pass. Coverage includes:

• selector exploration / exploitation / cold-start / pooled backoff / concurrent record_outcome
• dispatcher early-stop, max-attempts retry, label propagation, context-label routing, fresh-result invariant
• scenario atomic-attack emission, harm-category partitioning, per-objective seed-technique filtering, resume rehydration, baseline-policy enforcement

[Copilot]

Pull request overview

This PR adds a new adaptive text scenario that chooses attack techniques per objective using an epsilon-greedy selector, plus unit tests for the selector, dispatcher, and scenario construction.

Changes:

• Adds TextAdaptive, AdaptiveDispatchAttack, and AdaptiveTechniqueSelector.
• Registers the adaptive scenario virtual package under pyrit.scenario.adaptive.
• Adds unit coverage for adaptive selection, dispatch behavior, and atomic attack generation.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyrit/scenario/scenarios/adaptive/text_adaptive.py	Implements the adaptive scenario and per-objective atomic attack construction.
pyrit/scenario/scenarios/adaptive/selector.py	Adds epsilon-greedy technique selection and context helpers.
pyrit/scenario/scenarios/adaptive/dispatcher.py	Adds dispatcher attack that selects and runs inner attack arms.
pyrit/scenario/scenarios/adaptive/__init__.py	Exports adaptive scenario components.
pyrit/scenario/__init__.py	Registers adaptive as a virtual scenario subpackage.
tests/unit/scenario/scenarios/adaptive/test_text_adaptive.py	Adds scenario construction and baseline policy tests.
tests/unit/scenario/scenarios/adaptive/test_selector.py	Adds selector behavior tests.
tests/unit/scenario/scenarios/adaptive/test_dispatcher.py	Adds dispatcher behavior tests.