Facing error while running the AI Red Team scan through azure ai evaluation #44106
Description
- Package Name: azure_ai_evaluation
- Package Version: 1.13.7
- Operating System: Windows
- Python Version: 3.13
Describe the bug
Facing error while running the AI Red Team scan . While using the default attack_strategies and risk_categories , the objectives selection is been skipped and hence the whole scan has been skipped , eventually returning null results.
To Reproduce
Steps to reproduce the behavior:
- from azure.identity import DefaultAzureCredential
from azure.ai.evaluation.red_team import RedTeam, RiskCategory
azure_ai_project = {
"subscription_id": "your subscription",
"resource_group_name": "resource group",
"project_name": "project name ",
}
red_team_agent = RedTeam(
azure_ai_project=azure_ai_project, # required
credential=DefaultAzureCredential(), # required
risk_categories=[ # optional, defaults to all four risk categories
RiskCategory.Violence,
RiskCategory.HateUnfairness,
RiskCategory.Sexual,
RiskCategory.SelfHarm,
],
num_objectives=5, # optional, defaults to 10
)
from azure.ai.evaluation.red_team import AttackStrategy
def simple_callback(query: str) -> str:
# Your implementation to call your application (e.g., RAG system, chatbot)
return "I'm an AI assistant that follows ethical guidelines. I cannot provide harmful content."
red_team_result = await red_team_agent.scan(
target=simple_callback,
scan_name="Scan with many strategies",
attack_strategies=[
AttackStrategy.EASY,
AttackStrategy.MODERATE,
AttackStrategy.DIFFICULT,
],
)
Expected behavior
The scan should run and the detailed evaluation report should be generated like below
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
I have fixed this issue in my local , here is the resolution :
under : azure/ai/evaluation/simulator/_model_tools/_generated_rai_client.py ( line 160) , removing the target_type parameter in "get_attack_objectives" , as it not used in the function definition anywhere.