Pipeline analysis

[ReilleyMilne (ReilleyMilne)]

• Separated the azsdk-common-pipeline-troublshooting skill into two separate skills one for analysis and one for fixing.
• Added a new skill azsdk-common-pipeline-fixer, which automatically applies and verifies fixes for Azure SDK CI/CD pipeline failures.
• Refined and expanded evaluation suites for both skills to ensure correct routing of user prompts: fixer skill triggers on "fix/apply" requests, while analysis skill triggers only on diagnosis requests.
• Added more explicit failure messages to the PipelineIdentiferHelper to fail early when a valid project name is not given.

Fixes: #16200
Fixes: #16204

[github-actions]

📊 GEPA Skill Quality Scores

Skill	Quality	Triggers	Tests
azsdk-common-pipeline-analysis	⚠️ 0.57	N/A	---
azsdk-common-pipeline-fixer	⚠️ 0.57	N/A	---
sensei	⚠️ 0.57	N/A	---
azsdk-common-prepare-release-plan	✅ 0.86	N/A	---
azsdk-common-apiview-feedback-resolution	✅ 1.00	N/A	---
azsdk-common-generate-sdk-locally	✅ 1.00	N/A	---
azsdk-common-generate-sdk-pipeline	✅ 1.00	N/A	---
azsdk-common-sdk-release	✅ 1.00	N/A	---
azure-typespec-author	✅ 1.00	N/A	---
markdown-token-optimizer	✅ 1.00	N/A	---
sdk-ai-bot-eval-dataset	✅ 1.00	N/A	---
sdk-ai-bot-run-evaluation	✅ 1.00	N/A	---
skill-authoring	✅ 1.00	N/A	---

10/13 skills at quality ≥ 0.80

How to improve

# Score a specific skill
python .github/skills/sensei/scripts/gepa/auto_evaluator.py score --skill <name> --skills-dir .github/skills --tests-dir tests

# Optimize a skill with GEPA
python .github/skills/sensei/scripts/gepa/auto_evaluator.py optimize --skill <name> --skills-dir .github/skills --tests-dir tests

[Copilot]

Pull request overview

This PR expands the Azure.Sdk.Tools.Vally evaluation coverage for pipeline troubleshooting/fixing by adding a new “quality” tier, updating the canonical mock pipeline failure fixture to a real-world storage version-parsing bug, and splitting pipeline capabilities into analysis-only vs fix-and-verify skills.

Changes:

• Add output-quality pipeline evals (analyze + fix) and include them in the Vally PR-gate suite.
• Update the mock azsdk_analyze_pipeline handler and related Vally scenarios/fixtures to use the QueueClientOptions service-version parsing failure.
• Introduce azsdk-common-pipeline-fixer and adjust the existing troubleshooting skill to be analysis-only with a structured output format.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tools/azsdk-cli/Azure.Sdk.Tools.Vally/README.md	Documents the new evals/quality tier and updated scenario inventory/layout.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptionsTests.cs	Adds the canonical failing test fixture for the pipeline “fix” eval.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptions.cs	Adds the canonical failing source fixture (missing switch cases) used by pipeline evals.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/fixtures/.gitkeep	Updates fixture path guidance for environment.files overlays.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/workflow-scenarios/mock/analyze-failed-pipeline.eval.yaml	Aligns the mock workflow-scenario grader substring with the new canonical failure.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/tools/prompt-to-tool-pipeline.eval.yaml	Adds pipeline-failure routing stimuli (required/disallowed tool routing).
tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/quality/pipeline/fix-pipeline.eval.yaml	New output-quality eval that requires applying the code fix and verifying via package MCP tools.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/quality/pipeline/analyze-pipeline.eval.yaml	New output-quality eval that requires a correct root-cause diagnosis from build ref.
tools/azsdk-cli/Azure.Sdk.Tools.Vally/.vally.yaml	Adds a quality suite and includes it in pr-gate.
tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Pipeline/AnalyzeAndArtifactsHandlers.cs	Updates the mock azsdk_analyze_pipeline output to the canonical fixture failure.
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Helpers/PipelineIdentifierHelper.cs	Adjusts project-name resolution logic for DevOps build references.
.github/skills/README.md	Documents the new pipeline fixer skill and refines troubleshooting description.
.github/skills/azsdk-common-pipeline-troubleshooting/SKILL.md	Repositions the skill as analysis-only (“Pipeline Analysis”) and updates tools/steps.
.github/skills/azsdk-common-pipeline-troubleshooting/references/output-format.md	Adds a required markdown output structure for pipeline diagnoses.
.github/skills/azsdk-common-pipeline-troubleshooting/references/failure-patterns.md	Reworks failure pattern guidance into categorized tables (tests/build/validation/infra).
.github/skills/azsdk-common-pipeline-troubleshooting/evals/trigger.eval.yaml	Updates trigger prompts and adds an anti-trigger for auto-fix requests.
.github/skills/azsdk-common-pipeline-troubleshooting/evals/eval.yaml	Removes the old capability suite file.
.github/skills/azsdk-common-pipeline-fixer/SKILL.md	Introduces a new skill for applying and verifying pipeline fixes.
.github/skills/azsdk-common-pipeline-fixer/references/language-notes.md	Adds language-specific guidance for what build/check cover and what’s out-of-scope.
.github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml	Adds trigger/anti-trigger evals for the new fixer skill.
.github/skills/.vally.yaml	Registers the new fixer skill’s eval path for skill-eval discovery.

[azure-sdk-automation]

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

[ReilleyMilne (ReilleyMilne)]

/check-enforcer override