Skip to content

Rename top-level azsdk-evals folder to evals - #16402

Merged
helen229 merged 2 commits into
mainfrom
feat/rename-azsdk-evals-to-evals
Jul 21, 2026
Merged

Rename top-level azsdk-evals folder to evals#16402
helen229 merged 2 commits into
mainfrom
feat/rename-azsdk-evals-to-evals

Conversation

@helen229

@helen229 helen229 commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

Renames the top-level azsdk-evals directory introduced by #16367 to the simpler evals name, following review discussion that evals are a first-class repository-wide test category.

  • Renames the suite root from azsdk-evals/ to evals/.
  • Flattens the inner layout from evals/evals/... to evals/tools/ and evals/workflows/{mock,live}/.
  • Updates Vally configuration, workflow/live pipeline globs, relative fixture and skill paths, tests, documentation, and source references.
  • Updates the relative-link allowlist and fixture invariant test for the new root.

Validation

  • npm test in eng/common/scripts/eval: 42/42 passed.
  • CI matrix discovery against the flattened root found the expected 7 shards.
  • Git-fixture discovery found the expected azure-rest-api-specs fixture at the canonical cache location.
  • No tracked references to the former Vally project, evals/evals, or workflow-scenarios paths remain.
  • git diff --check passed.

Follow-up to #16367.

Copilot AI review requested due to automatic review settings July 17, 2026 22:09
@helen229
helen229 requested a review from a team as a code owner July 17, 2026 22:09
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).
63 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request renames the repository-root Vally evaluation suite directory from azsdk-evals/ to evals/, treating evals as a first-class, repo-wide test category. It updates documentation and engineering-system wiring so CI and link-checking continue to discover and run the suite from the new location.

Changes:

  • Renamed the top-level eval suite directory to evals/ and updated references in documentation.
  • Updated CI/live pipeline path triggers and vallyRoot to point at evals.
  • Updated relative-link allowlist and the fixture invariant test to use the new root.

Reviewed changes

Copilot reviewed 5 out of 31 changed files in this pull request and generated no comments.

Show a summary per file
File Description
evals/README.md Updates suite documentation to reflect the new evals/ root.
evals/fixtures/README.md Adds/updates fixture usage documentation under the renamed root.
evals/fixtures/Microsoft.Widget/Widget/tspconfig.yaml Fixture file moved under evals/fixtures as part of the rename.
evals/fixtures/Microsoft.Widget/Widget/main.tsp Fixture file moved under evals/fixtures as part of the rename.
evals/fixtures/Microsoft.Widget/Widget/employee.tsp Fixture file moved under evals/fixtures as part of the rename.
evals/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptionsTests.cs Pipeline-analysis fixture moved under evals/fixtures as part of the rename.
evals/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptions.cs Pipeline-analysis fixture moved under evals/fixtures as part of the rename.
evals/evals/workflow-scenarios/mock/typespec-generation-step02.eval.yaml Mock workflow scenario eval moved/updated under the new root.
evals/evals/workflow-scenarios/mock/rename-client-property.eval.yaml Mock workflow scenario eval moved/updated under the new root.
evals/evals/workflow-scenarios/mock/release-planner-workflows.eval.yaml Mock release-planner scenario suite moved/updated under the new root.
evals/evals/workflow-scenarios/mock/multi-turn-release-workflows.eval.yaml Mock multi-turn scenario suite moved/updated under the new root.
evals/evals/workflow-scenarios/mock/fix-pipeline.eval.yaml Mock pipeline-fix scenario moved/updated under the new root.
evals/evals/workflow-scenarios/mock/check-public-repo-then-validate.eval.yaml Mock TypeSpec workflow scenario moved/updated under the new root.
evals/evals/workflow-scenarios/mock/analyze-failed-pipeline.eval.yaml Mock pipeline-analysis scenario moved/updated under the new root.
evals/evals/workflow-scenarios/live/release-planner.eval.yaml Live release-planner scenario moved/updated under the new root.
evals/evals/tools/prompt-to-tool-verify.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-typespec.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-releaseplan.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-pipeline.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-package.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-github.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-engsys.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-config.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/prompt-to-tool-apiview.eval.yaml Tool-trigger eval spec moved/updated under the new root.
evals/evals/tools/add-arm-resource.eval.yaml Scenario eval spec moved/updated under the new root.
evals/.vally.yaml Vally configuration located under the renamed suite root.
evals/.gitignore Ignores eval outputs under the renamed root.
eng/common/scripts/eval/test/init-eval-git-fixtures.test.ts Updates invariant test discovery to use evals as the root.
eng/common/scripts/allow-relative-links.txt Updates allowlist entry from azsdk-evals/** to evals/**.
eng/common/pipelines/workflow-eval.yml Updates PR/trigger path filters and vallyRoot to evals.
eng/common/pipelines/live-eval.yml Updates vallyRoot to evals for live eval execution.

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@helen229
helen229 requested a review from a team as a code owner July 20, 2026 16:42
@azure-sdk-automation

Copy link
Copy Markdown
Contributor

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

helen229 added a commit to Azure/azure-sdk-for-rust that referenced this pull request Jul 21, 2026
Sync eng/common directory with azure-sdk-tools for PR
Azure/azure-sdk-tools#16402 See [eng/common
workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow)

---------

Co-authored-by: helen229 <gaoh@microsoft.com>
helen229 added a commit to Azure/azure-sdk-for-js that referenced this pull request Jul 21, 2026
Sync eng/common directory with azure-sdk-tools for PR
Azure/azure-sdk-tools#16402 See [eng/common
workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow)

---------

Co-authored-by: helen229 <gaoh@microsoft.com>
@helen229
helen229 merged commit 3f79d42 into main Jul 21, 2026
21 checks passed
@helen229
helen229 deleted the feat/rename-azsdk-evals-to-evals branch July 21, 2026 21:17
helen229 added a commit that referenced this pull request Jul 21, 2026
Resolves conflict in evals/.vally.yaml: PR #16402 (merged to main) renamed
azsdk-evals/ to evals/ but did not include the nightly/tag-filtered suite
glob tightening from this branch's review-feedback fix. Kept this branch's
tightened globs (tools/**/*.eval.yaml + workflows/**/*.eval.yaml).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants