Rename top-level azsdk-evals folder to evals - #16402
Conversation
|
Azure Pipelines: Successfully started running 2 pipeline(s). 63 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
This pull request renames the repository-root Vally evaluation suite directory from azsdk-evals/ to evals/, treating evals as a first-class, repo-wide test category. It updates documentation and engineering-system wiring so CI and link-checking continue to discover and run the suite from the new location.
Changes:
- Renamed the top-level eval suite directory to
evals/and updated references in documentation. - Updated CI/live pipeline path triggers and
vallyRootto point atevals. - Updated relative-link allowlist and the fixture invariant test to use the new root.
Reviewed changes
Copilot reviewed 5 out of 31 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| evals/README.md | Updates suite documentation to reflect the new evals/ root. |
| evals/fixtures/README.md | Adds/updates fixture usage documentation under the renamed root. |
| evals/fixtures/Microsoft.Widget/Widget/tspconfig.yaml | Fixture file moved under evals/fixtures as part of the rename. |
| evals/fixtures/Microsoft.Widget/Widget/main.tsp | Fixture file moved under evals/fixtures as part of the rename. |
| evals/fixtures/Microsoft.Widget/Widget/employee.tsp | Fixture file moved under evals/fixtures as part of the rename. |
| evals/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptionsTests.cs | Pipeline-analysis fixture moved under evals/fixtures as part of the rename. |
| evals/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptions.cs | Pipeline-analysis fixture moved under evals/fixtures as part of the rename. |
| evals/evals/workflow-scenarios/mock/typespec-generation-step02.eval.yaml | Mock workflow scenario eval moved/updated under the new root. |
| evals/evals/workflow-scenarios/mock/rename-client-property.eval.yaml | Mock workflow scenario eval moved/updated under the new root. |
| evals/evals/workflow-scenarios/mock/release-planner-workflows.eval.yaml | Mock release-planner scenario suite moved/updated under the new root. |
| evals/evals/workflow-scenarios/mock/multi-turn-release-workflows.eval.yaml | Mock multi-turn scenario suite moved/updated under the new root. |
| evals/evals/workflow-scenarios/mock/fix-pipeline.eval.yaml | Mock pipeline-fix scenario moved/updated under the new root. |
| evals/evals/workflow-scenarios/mock/check-public-repo-then-validate.eval.yaml | Mock TypeSpec workflow scenario moved/updated under the new root. |
| evals/evals/workflow-scenarios/mock/analyze-failed-pipeline.eval.yaml | Mock pipeline-analysis scenario moved/updated under the new root. |
| evals/evals/workflow-scenarios/live/release-planner.eval.yaml | Live release-planner scenario moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-verify.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-typespec.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-releaseplan.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-pipeline.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-package.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-github.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-engsys.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-config.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/prompt-to-tool-apiview.eval.yaml | Tool-trigger eval spec moved/updated under the new root. |
| evals/evals/tools/add-arm-resource.eval.yaml | Scenario eval spec moved/updated under the new root. |
| evals/.vally.yaml | Vally configuration located under the renamed suite root. |
| evals/.gitignore | Ignores eval outputs under the renamed root. |
| eng/common/scripts/eval/test/init-eval-git-fixtures.test.ts | Updates invariant test discovery to use evals as the root. |
| eng/common/scripts/allow-relative-links.txt | Updates allowlist entry from azsdk-evals/** to evals/**. |
| eng/common/pipelines/workflow-eval.yml | Updates PR/trigger path filters and vallyRoot to evals. |
| eng/common/pipelines/live-eval.yml | Updates vallyRoot to evals for live eval execution. |
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16402 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: helen229 <gaoh@microsoft.com>
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16402 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: helen229 <gaoh@microsoft.com>
Resolves conflict in evals/.vally.yaml: PR #16402 (merged to main) renamed azsdk-evals/ to evals/ but did not include the nightly/tag-filtered suite glob tightening from this branch's review-feedback fix. Kept this branch's tightened globs (tools/**/*.eval.yaml + workflows/**/*.eval.yaml).
Summary
Renames the top-level
azsdk-evalsdirectory introduced by #16367 to the simplerevalsname, following review discussion that evals are a first-class repository-wide test category.azsdk-evals/toevals/.evals/evals/...toevals/tools/andevals/workflows/{mock,live}/.Validation
npm testineng/common/scripts/eval: 42/42 passed.azure-rest-api-specsfixture at the canonical cache location.evals/evals, orworkflow-scenariospaths remain.git diff --checkpassed.Follow-up to #16367.