azure-typespec-author skill, switch to the new KB endpoint and update evals

[haolingdong-msft]

Fix #16015
Fix #16425

Switch the azure-typespec-author code-quality benchmark from the locally-built qa-bot backend (localhost:8088) to the new remote KB agent test endpoint.

• .vally.yaml: repoint the azsdk-mcp environment's AZURE_SDK_KB_ENDPOINT to the new test endpoint and add AZURE_SDK_KB_CLIENT_ID (the endpoint is MSAL-protected; azsdk-cli requests a token for /.default).
• benchmark pipeline: set StartQaBotBackend=false for the code-quality job (no local Go backend), run each Vally suite inside an authenticated AzureCLI@2 context, and drop the now-unused qa-bot backend log artifact.
• run-suite template: add optional AzureSubscription param; when set the eval runs under AzureCLI@2 so azsdk-cli can obtain a bearer token for the remote KB. Trigger/mock runs (no subscription) are unchanged.

[haolingdong-msft]

benchmark pipeline runs:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6600498&view=results
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6605947&view=results

Run	Pass	Rate
Build 6600498	31 / 38	81.6%
Build 6605947	29 / 38	76.3%
Combined (pass if either)	32 / 38	84.2%

Case	Failure Reason	Grounding Used	Authoring-Plan Issue?
001005 add-preview-after-preview	New examples/2025-05-04-preview/*.json not created; old examples/2024-10-01-preview/ not deleted; 2024-10-01-preview / v2024_10_01_preview still in main.tsp & employee.tsp — unreleased preview not rebased	run1: MCP plan · run2: agentic search	✅ Yes in run 1, hit vally issue #401 in run 2
001007 add-stable-after-preview	Old preview examples not retired; old version refs remain; expected @added(v2025_01_01) age?: int32 = 21; remove/rename/add not produced	run1: MCP plan · run2: agentic search	✅ Yes in run1, prompt loss in run 2
002004 ARM extension-resource	Grader wants Extension.Read< / Extension.ListByTarget< templates in badgeAssignment.tsp; ListByTarget not emitted	plan (both runs)	✅ Yes
003001 arm-action-lro	Grader wants 4-arg ArmResourceActionAsync<Employee, MoveRequest, MoveResponse, …> + ArmCombinedLroHeaders<…>; plan recommended 3-arg form	plan (both runs)	✅ Yes
004003 operationId-decorator	run1: over-applied @operationId( to VpnServerConfigurationPolicyGroup.tsp · run2: missing #suppress …operationId in BastionHost.tsp (flaky)	no plan, no agentic search	❌ No (skill execution)
005001 suppress-warning	run1: missing #suppress + FIXME…conversion-fix in models.tsp · run2: required azsdk_run_typespec_validation not called	no plan, no agentic search	❌ No (skill execution)

It aligns with our previous tests. And the pass rate looks acceptable.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
64 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Updates the azure-typespec-author skill evaluation/benchmarking flow to use the new remote Knowledge Base (KB) agent test endpoint (MSAL-protected), adjusting pipelines and eval cases to run in an authenticated Azure CLI context and tightening eval graders.

Changes:

• Repoints KB defaults/overrides to the new agent endpoints and configures required auth environment variables for the remote KB.
• Updates the TypeSpec author benchmark pipeline and run-suite template to run Vally under AzureCLI@2 and removes the local QA bot backend dependency for these eval jobs.
• Expands eval coverage/robustness by adding a Node dependency check script and updating graders/patterns across multiple eval cases.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tools/azsdk-cli/Azure.Sdk.Tools.Cli/Services/AzureSdkKnowledgeBaseService.cs	Updates the default KB service endpoint used by azsdk-cli.
eng/pipelines/templates/steps/typespec-author-eval-run-suite.yml	Runs each Vally suite inside AzureCLI@2 to enable token acquisition for remote KB.
eng/pipelines/azure-typespec-author-benchmark.yml	Disables local backend startup for benchmark jobs, adds Azure subscription context, removes backend log artifact.
.github/skills/azure-typespec-author/evaluate/scripts/check-node-dependencies.js	Adds a script to validate node_modules and required TypeSpec packages exist before compiling.
.github/skills/azure-typespec-author/evaluate/evals/005001.eval.yaml	Adds dependency checks; loosens required tool calls to allow bash; keeps validation/tool-call requirements.
.github/skills/azure-typespec-author/evaluate/evals/004003.eval.yaml	Adds dependency checks; loosens required tool calls to allow bash; keeps validation/tool-call requirements.
.github/skills/azure-typespec-author/evaluate/evals/004002.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/004001.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/003002.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/003001.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002011.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002010.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002009.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002008.eval.yaml	Adds dependency checks and tightens regex expectations for ARM top/skip parameter patterns.
.github/skills/azure-typespec-author/evaluate/evals/002007.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002006.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002005.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002004.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002003.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002002.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/002001.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/001013.eval.yaml	Adds dependency checks and relaxes versioning decorator matchers to allow optional namespace qualifiers.
.github/skills/azure-typespec-author/evaluate/evals/001011.eval.yaml	Adds dependency checks and relaxes versioning decorator matchers to allow optional namespace qualifiers.
.github/skills/azure-typespec-author/evaluate/evals/001010.eval.yaml	Adds dependency checks and relaxes versioning decorator matchers to allow optional namespace qualifiers.
.github/skills/azure-typespec-author/evaluate/evals/001009.eval.yaml	Adds dependency checks and relaxes versioning decorator matchers to allow optional namespace qualifiers.
.github/skills/azure-typespec-author/evaluate/evals/001008.eval.yaml	Adds dependency checks and allows either web_fetch or the MCP authoring-plan tool for version info.
.github/skills/azure-typespec-author/evaluate/evals/001007.eval.yaml	Adds dependency checks, allows either web_fetch or MCP authoring-plan tool, relaxes regex matchers.
.github/skills/azure-typespec-author/evaluate/evals/001006.eval.yaml	Adds dependency checks and allows either web_fetch or the MCP authoring-plan tool for version info.
.github/skills/azure-typespec-author/evaluate/evals/001005.eval.yaml	Adds dependency checks and allows either web_fetch or the MCP authoring-plan tool; relaxes regex matchers.
.github/skills/azure-typespec-author/evaluate/evals/001004.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/001003.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/evals/001002.eval.yaml	Adds dependency checks and relaxes versioning decorator matchers to allow optional namespace qualifiers.
.github/skills/azure-typespec-author/evaluate/evals/001001.eval.yaml	Adds dependency checks to ensure fixture node deps are present before compilation.
.github/skills/azure-typespec-author/evaluate/.vally.yaml	Switches the azsdk-mcp environment to the remote KB endpoint and supplies client ID/scope env vars.
.github/skills/.gitignore	Ignores result/ output directory used by local Vally runs.

[haolingdong-msft]

Fix #16425