FEAT: Refactoring score model (phase 2) by rlundeen2 · Pull Request #2491 · microsoft/PyRIT

Richard Lundeen (rlundeen2) · 2026-08-25T19:37:52Z

This PR separates the generic scorer contract from message-specific scoring behavior and gives every score an honest, durable anchor. It also makes “could not determine” a first-class result instead of silently treating it as failure.

This implements Phase 2: The Score model, in one migration from the PyRIT design proposal: Generalizing the Scorer contract (Message / OpenTelemetry / Surface).

Summary

Add Score.scorable, make message_piece_id optional and derived for message scorables, and persist loose content so scores remain auditable and re-scorable.
Add ScoreStatus.COMPLETE / UNDETERMINED, nullable score values, strict get_value() behavior, three-valued aggregation, and an attack-side AttackOutcome.UNDETERMINED contract.
Separate generic true/false and float scorer families from MessageScorer, while preserving a narrow bridge for pre-2.0 message and piece-level subclass implementations.
Forward (scorable, expectation) through composites, inverters, and thresholds without erasing the new distinctions.
Add the Alembic migration, database mappings, output/backend/frontend projections, batched content lookup and backfill paths, and transactional loose-content anchor rewrites.
Exclude undetermined scorer-evaluation samples instead of counting them as negative labels.

Compatibility

Deprecated message-shaped score_async(message=..., objective=..., ...) calls remain on MessageScorer until 2.0.
Direct pre-2.0 scorer subclasses that implement the old protected message or piece hooks retain a narrow bridge to the canonical scorable= entry point.
Deprecated response helpers on Scorer remain forwarding shims until 2.0.
Intentional removal: generic scorers and wrappers expose only score_async(*, scorable, expectation). They do not borrow message-only public APIs or policy from message-capable children. Preserving that incidental behavior would couple evidence-agnostic wrappers to the message pipeline.
Complete scores retain their existing get_value() behavior.
message_piece_id remains populated for message-backed scores while scorable becomes the canonical anchor.

- Materialize aggregator input before validating so a generator is not drained. - Read undetermined-safe values in Crescendo/TAP so an undetermined score cannot crash an attack. - Thread the caller's scorable through the message pipeline so re-scoring stored content keeps its durable anchor. - Tag every storable scorable with an explicit scorable_type discriminator and refuse unsupported anchors. - Move message error/refusal policy off the generic Scorer base into the message family.

…cked errors

…8_20_score2 # Conflicts: # frontend/src/types/index.ts # pyrit/cli/_results.py # pyrit/executor/attack/__init__.py # pyrit/executor/attack/core/__init__.py # pyrit/models/__init__.py # pyrit/models/score/__init__.py # pyrit/score/__init__.py # pyrit/score/float_scale/float_scale_score_aggregator.py # pyrit/score/score_utils.py # pyrit/score/true_false/float_scale_threshold_scorer.py # pyrit/score/true_false/true_false_score_aggregator.py

Roman Lutz (romanlutz) · 2026-08-26T17:57:29Z

Are you planning to rerun all the notebooks? Lots of potential for breaking things in here.

The generic Scorer owned message-family orchestration, and the generic score-family bases owned message fallback policy. Both are responsibility bleed against doc/code/framework.md. Move score_prompts_batch_async, score_response_async, score_response_multiple_scorers_async, and _score_response_with_scorer_async from Scorer to MessageScorer. This removes the isinstance guard and the circular-import workaround. Add an evidence-agnostic Scorer.score_batch_async that does concurrency and rate limiting only. MessageScorer.score_prompts_batch_async builds the scorables and the message options, then delegates to it. This also fixes a regression: the previous batch method raised a TypeError for scorers that are not MessageScorers. FloatScaleThresholdScorer, TrueFalseCompositeScorer, and TrueFalseInverterScorer extend TrueFalseScorer directly, so the Fuzzer default scorer failed. Delete _build_fallback_score from TrueFalseScorer and FloatScaleScorer. Add one shared MessageScorer._build_neutral_fallback_score that holds the blocked, error, and filtered branching once. MessageTrueFalseScorer and MessageFloatScaleScorer each delegate to it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc221fb6-d5ee-4dbd-a91f-3aaed63a97a7

The contract test asserted that TrueFalseScorer extends MessageScorer. That stopped being true when TrueFalseScorer became the score-value axis only, so the test failed. State the real contract instead: TrueFalseScorer carries no message hooks, and MessageTrueFalseScorer combines both axes. This is what RAIServiceScorer must extend now. The ty hook passed filenames, so pre-commit split them into parallel batches and each batch ran its own uv run. The concurrent syncs fought over the same venv and failed to copy locked files on Windows. Run the hook once instead. The hook also let uv run relock, which rewrites every index URL in uv.lock when a contributor has a private package index configured. Pass --frozen. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bc221fb6-d5ee-4dbd-a91f-3aaed63a97a7

Adrian Gavrila (adrian-gavrila)

Overall I think this is a great refactor, there are a few places where context can get lost or changed to make undetermined out out more definitive than is probably intended, which I thought might be worth attention.

hannahwestra25 · 2026-08-27T16:39:20Z

Could we update the scoring docs for this new hierarchy? 0_scoring still omits the new MessageScorer classes, and 3_combining_scorers says conversation scoring accepts any true/false or float scorer, while it now accepts only message scorers. framework.md also still describes generic scoring as future work.

…een2-score-model-review

Preserve scorer compatibility, durable evidence, and undetermined outcomes across storage, attacks, output, and documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3e111a6-b6a9-441c-8edf-9146d369b136

Richard Lundeen (rlundeen2) · 2026-08-27T21:55:35Z

(GHCP Generated): Audited all 65 paired documentation notebooks and executed 56 end to end with the worktree uv environment, including every scoring notebook. The remaining examples are externally blocked: Azure ML workspace configuration, Playwright/browser prerequisites, a redacted HTTP request, an unsupported legacy completions endpoint, and one adaptive scenario that exceeded the execution limit. I retained outputs and verified that changed notebooks contain no configured secrets.

Richard Lundeen (rlundeen2) · 2026-08-27T21:55:36Z

(GHCP Generated): Addressed. The framework and scoring guides now show MessageScorer, MessageTrueFalseScorer, and MessageFloatScaleScorer; distinguish generic scorable families from message-capable helpers; document wrapper constraints and blocked defaults; and describe managed file evidence plus the separate Phase 3 message-media follow-up.

Exclude private scorer adapters from the buildable registry, clarify conversation scorer documentation, and remove output-only notebook churn. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3e111a6-b6a9-441c-8edf-9146d369b136

Restores behavior that worked before the scorer model split: - Media archiving is best effort again. score_image_async no longer raises FileNotFoundError and discards a verdict the scorer already reached; the score persists without a loose-content anchor. - Pre-2.0 subclasses of TrueFalseScorer and FloatScaleScorer keep score_async(message=...), score_message_async and score_prompts_batch_async, plus the default validator their family base used to supply. - create_conversation_scorer accepts TrueFalseCompositeScorer, TrueFalseInverterScorer and FloatScaleThresholdScorer again. - Scorer.score_response_async and Scorer.score_response_multiple_scorers_async return as deprecating forwarders to MessageScorer instead of disappearing. Also corrects comments that still promised a False / 0.0 fallback where a blocked scorer response now yields an undetermined score. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3e111a6-b6a9-441c-8edf-9146d369b136

…-review

Register pre-2.0 family subclasses as virtual subclasses of LegacyMessageScorerCompatibility. isinstance now recognizes them, so a legacy scorer works again inside TrueFalseCompositeScorer, TrueFalseInverterScorer, FloatScaleThresholdScorer, and create_conversation_scorer. That registration removes the need for a separate prepared-message dispatcher, so create_conversation_scorer goes back to a MessageScorer and normalizes a non-message scorer through the existing compatibility adapter. Restore the raising behavior of add_scores_to_memory_async, so an unreadable source file fails loudly instead of silently. Keep the Scorers section of framework.md concise. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3e111a6-b6a9-441c-8edf-9146d369b136

Rename score_blocked_content to should_score_blocked_content across the message scorer API, compatibility wrappers, attack guidance, and tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3e111a6-b6a9-441c-8edf-9146d369b136

Keep generic wrappers on the canonical scorable API and preserve only narrow legacy subclass bridges. Fix scorer edge cases and align the scoring documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c3e111a6-b6a9-441c-8edf-9146d369b136

Richard Lundeen (rlundeen2) added 5 commits August 25, 2026 12:37

FEAT: Refactoring score model

8278b7e

FIX: Score content behind a block by default; undetermined on non-blo…

86f5039

…cked errors

MAINT: Remove storable_scorable in favor of an inline anchor check

3b882e5

Richard Lundeen (rlundeen2) changed the title ~~FEAT: Refactoring score model~~ FEAT: Refactoring score model (phase 2) Aug 26, 2026

Richard Lundeen (rlundeen2) and others added 2 commits August 26, 2026 15:54

Roman Lutz (romanlutz) reviewed Aug 27, 2026

View reviewed changes