Skip to content

FIX Stop JSON retries for adversarial refusals - #2443

Merged
hannahwestra25 merged 9 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-fix-refusal-retries
Aug 25, 2026
Merged

FIX Stop JSON retries for adversarial refusals#2443
hannahwestra25 merged 9 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-fix-refusal-retries

Conversation

@hannahwestra25

@hannahwestra25 hannahwestra25 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Description

OpenAI structured refusals arrive as blocked/error Messages, but the shared adversarial conversation manager parsed them as malformed JSON and exhausted the JSON retry budget.

This change detects error responses before parsing, preserving their semantics: blocked responses raise BadRequestException with the provider status/message, empty responses raise EmptyResponseException, and processing/unknown failures raise PyritException. These terminal errors do not enter the JSON retry loop, while malformed normal responses still retry.

Tests and Documentation

  • Added regression tests for blocked, empty, processing, unknown, mixed-piece, and malformed-JSON responses.
  • Passed 82 manager tests plus focused TAP, Red Teaming, and Crescendo paths.
  • Passed Ruff and ty for changed files.
  • No documentation changes; public APIs are unchanged.

Detect blocked and errored adversarial chat responses before schema parsing so terminal target failures do not consume the malformed-JSON retry budget.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@hannahwestra25
hannahwestra25 marked this pull request as ready for review August 20, 2026 21:59
@hannahwestra25 hannahwestra25 changed the title [DRAFT] FIX Stop JSON retries for adversarial refusals FIX Stop JSON retries for adversarial refusals Aug 20, 2026
@romanlutz

Copy link
Copy Markdown
Contributor

Red Teaming and Crescendo retain their existing error handling.

Aren't they all using the same adv conv manager?

@hannahwestra25

Copy link
Copy Markdown
Contributor Author

Red Teaming and Crescendo retain their existing error handling.

Aren't they all using the same adv conv manager?

yes updated the description!

Comment thread pyrit/executor/attack/component/adversarial_conversation_manager.py Outdated
Keep blocked, empty, processing, and unknown adversarial response semantics while treating each as terminal for JSON retry.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5a7e5481-94f6-4cfe-97b4-441ad9b3939c
@hannahwestra25
hannahwestra25 added this pull request to the merge queue Aug 25, 2026
Merged via the queue into microsoft:main with commit da5cfd9 Aug 25, 2026
53 of 54 checks passed
@hannahwestra25
hannahwestra25 deleted the hannahwestra25-fix-refusal-retries branch August 25, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants