Move Vally eval suite to repo-root azsdk-evals#16333
Conversation
There was a problem hiding this comment.
Pull request overview
Moves the Vally evaluation suite out of tools/azsdk-cli/Azure.Sdk.Tools.Vally/ into a repo-root azsdk-evals/ folder, updating CI/pipeline wiring and relative paths so eval discovery and execution continue to work from the new location.
Changes:
- Relocates Vally config, eval specs, and fixtures into
azsdk-evals/and rewrites depth-sensitive relative paths. - Updates pipeline entrypoints to run evals from the new
vallyRootand adjusts path-based triggers. - Updates eval discovery tests and refreshes
azsdk-evals/README.mdto reflect the new layout and commands.
Reviewed changes
Copilot reviewed 11 out of 27 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| eng/common/scripts/eval/test/init-eval-git-fixtures.test.js | Updates the folder invariant to point eval discovery at azsdk-evals/evals. |
| eng/common/pipelines/workflow-eval.yml | Repoints CI triggers and vallyRoot to azsdk-evals for mock-tier workflow evals. |
| eng/common/pipelines/live-eval.yml | Repoints vallyRoot to azsdk-evals for the live-tier workflow evals. |
| azsdk-evals/README.md | Updates documentation, links, and command snippets for the new repo-root eval location. |
| azsdk-evals/.vally.yaml | Updates MCP server DLL/build paths to be correct relative to azsdk-evals/. |
| azsdk-evals/.gitignore | Ignores local eval output folders under the new root. |
| azsdk-evals/fixtures/.gitkeep | Documents fixture usage and intended referencing from eval specs. |
| azsdk-evals/fixtures/Microsoft.Widget/Widget/tspconfig.yaml | Adds TypeSpec fixture config used by scenario evals. |
| azsdk-evals/fixtures/Microsoft.Widget/Widget/main.tsp | Adds TypeSpec fixture entrypoint for Microsoft.Widget. |
| azsdk-evals/fixtures/Microsoft.Widget/Widget/employee.tsp | Adds TypeSpec fixture models/ops used by authoring scenarios. |
| azsdk-evals/evals/workflow-scenarios/mock/typespec-generation-step02.eval.yaml | Updates skill and git fixture paths for the new directory depth. |
| azsdk-evals/evals/workflow-scenarios/mock/rename-client-property.eval.yaml | Updates skill and git fixture paths for the new directory depth. |
| azsdk-evals/evals/workflow-scenarios/mock/release-planner-workflows.eval.yaml | Updates skill mounts and git worktree source paths after the move. |
| azsdk-evals/evals/workflow-scenarios/mock/multi-turn-release-workflows.eval.yaml | Updates multi-skill mount paths and git worktree source paths after the move. |
| azsdk-evals/evals/workflow-scenarios/mock/check-public-repo-then-validate.eval.yaml | Updates skill and git fixture paths for the new directory depth. |
| azsdk-evals/evals/workflow-scenarios/mock/analyze-failed-pipeline.eval.yaml | Adds/relocates a mock workflow scenario eval under the new root. |
| azsdk-evals/evals/workflow-scenarios/live/release-planner.eval.yaml | Updates live scenario skill mount and git worktree source paths after the move. |
| azsdk-evals/evals/tools/prompt-to-tool-verify.eval.yaml | Adds/relocates unit-tier tool-invocation prompts for setup verification. |
| azsdk-evals/evals/tools/prompt-to-tool-typespec.eval.yaml | Adds/relocates TypeSpec tool-invocation prompts and full-context checks. |
| azsdk-evals/evals/tools/prompt-to-tool-releaseplan.eval.yaml | Adds/relocates release-plan tool-invocation prompts and full-context checks. |
| azsdk-evals/evals/tools/prompt-to-tool-pipeline.eval.yaml | Adds/relocates pipeline tool-invocation prompts and full-context checks. |
| azsdk-evals/evals/tools/prompt-to-tool-package.eval.yaml | Adds/relocates package tool-invocation prompts. |
| azsdk-evals/evals/tools/prompt-to-tool-github.eval.yaml | Adds/relocates GitHub tool-invocation prompts. |
| azsdk-evals/evals/tools/prompt-to-tool-engsys.eval.yaml | Adds/relocates engineering-system tool-invocation prompts. |
| azsdk-evals/evals/tools/prompt-to-tool-config.eval.yaml | Adds/relocates config/service-label tool-invocation prompts. |
| azsdk-evals/evals/tools/prompt-to-tool-apiview.eval.yaml | Adds/relocates APIView tool-invocation prompts. |
| azsdk-evals/evals/tools/add-arm-resource.eval.yaml | Adds/relocates an ARM authoring scenario seeded by fixtures. |
Comments suppressed due to low confidence (1)
azsdk-evals/README.md:20
- This README link points to
eng/pipelines/vally-eval.yml, but that file doesn't exist in the repo (so the link is dead). It looks like the intended entrypoint for the mock tier iseng/common/pipelines/workflow-eval.yml(updated in this PR).
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
|
The following pipelines have been queued for testing: |
|
@copilot resolve the merge conflicts in this pull request |
Resolved. The branch was behind
All 42 eval tests pass after the resolution (commit |
|
@copilot Pipelines - Run 20260714.17 logs can u fix the pipeline build failure |
Fixed in commit
All 42 eval tests still pass. |
|
@copilot Skip to main contentAzure DevOpsazure-sdk/internal/Pipelines/tools - sync-eng-common/20260714.17Account manager for Helen GaointernalOverviewBoardsReposPipelinesPipelinesEnvironmentsReleasesLibraryTest PlansArtifactsComplianceAgency HubDevOps Assistant ✨NEWProject settingsJobs in run #20260714.17Apply Patch for eng/common from azure-sdk-tools to azure-sdkView raw log - tools/azsdk-cli/Azure.Sdk.Tools.Cli/**
@@ -48,7 +48,7 @@ extends:
diff --git a/eng/common/scripts/eval/test/init-eval-git-fixtures.test.js b/eng/common/scripts/eval/test/init-eval-git-fixtures.test.js
-- ##[warning]Failed to properly apply patch files to [https://github.com/azure/azure-sdk] ##[error]PowerShell exited with code '1'. |
I've diagnosed the root cause. The The fix requires force-pushing a clean, squashed branch. The squashed commit (which I created locally) generates a single clean patch that only touches To fix this manually, please run: git checkout feat/move-evals-to-root
git fetch origin
git reset --soft origin/main
git commit -m "Move Vally eval suite to repo-root azsdk-evals"
git push --force-with-lease origin feat/move-evals-to-rootThis will produce a single patch that correctly modifies |
I've cleaned up the branch content but the What I fixed (commit
Why To fix git fetch origin
git checkout feat/move-evals-to-root
git pull origin feat/move-evals-to-root
git reset --soft origin/main
git commit -m "Move Vally eval suite to repo-root azsdk-evals"
git push --force-with-lease origin feat/move-evals-to-rootThis will squash everything into a single clean commit that only touches |
PR Summary: Move Vally eval suite to Goal: Move Completed:
Still failing:
Action needed (must be done manually — automated push tool can't force-push): git fetch origin
git checkout feat/move-evals-to-root
git pull origin feat/move-evals-to-root
git reset --soft origin/main
git commit -m "Move Vally eval suite to repo-root azsdk-evals"
git push --force-with-lease origin feat/move-evals-to-rootAfter the force-push, |
9784fc4 to
f776017
Compare
|
The following pipelines have been queued for testing: |
What
Moves the Vally eval suite from
tools/azsdk-cli/Azure.Sdk.Tools.Vally/to a top-levelazsdk-evals/folder, so the evals live at the repo root alongside other cross-cutting assets rather than nested under the CLI tool.Changes
git mvof the tracked eval config, eval specs, and fixtures:.vally.yaml,.gitignore,README.mdevals/**(13 eval specs)fixtures/**(Microsoft.Widget TypeSpec fixtures).vally.yamlMCP build/run paths (../artifacts/...,../tools/azsdk-cli/...)*.eval.yamlgitsource:and skill-mount paths (6-up -> 4-up)eng/common/pipelines/workflow-eval.ymlandlive-eval.yml(vallyRoot: azsdk-evals+ trigger/PR path filters).eng/common/scripts/eval/test/init-eval-git-fixtures.test.js(vallyRoot->azsdk-evals).README.mdtitle, layout tree, and root-relative links.Validation
node --testineng/common/scripts/eval)..vally.yamlparses cleanly; no remainingAzure.Sdk.Tools.Vallyreferences or wrong-depth (../../../../../../) paths in tracked files..slnreferences to the old path exist.Notes
{project-root}/../../../../common-typestoken infixtures/.../tspconfig.yamlis a TypeSpec project-root path (not filesystem-repo-relative) and is intentionally left unchanged.scratch/output remains physically under the old folder (gitignored dev artifacts); not part of this PR.