[TypeSpec Authoring][Benchmark] eval multi trial - #16341
Closed
JoyerJin wants to merge 13 commits into
Closed
Conversation
…_generate_authoring_plan' tool grader
JoyerJin
requested review from
a team,
catalinaperalta and
haolingdong-msft
as code owners
July 14, 2026 09:52
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the azure-typespec-author benchmark/eval pipeline to support multi-trial (multi-run) evaluation, improves skill overlay checkout by using full git refs, and adds additional fixture dependency validation to reduce flaky/invalid eval runs.
Changes:
- Allow benchmark pipelines to run Vally suites with configurable
--runs(and--workers) and increase job timeouts to accommodate longer runs. - Make skill overlay more robust by accepting full branch refs (e.g.,
refs/heads/...) and checking out fromFETCH_HEAD. - Add a Node script + new eval graders to verify fixture
node_modulespresence and required TypeSpec packages before compiling.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/pipelines/templates/steps/typespec-author-eval-setup.yml | Switch SkillBranch default to full ref and normalize/fetch/checkout via FETCH_HEAD for robust skill overlay. |
| eng/pipelines/templates/steps/typespec-author-eval-run-suite.yml | Add Runs/Workers parameters and pass them to vally eval for multi-trial execution. |
| eng/pipelines/azure-typespec-author-benchmark.yml | Add EvalRuns pipeline parameter, pass Runs into suite steps, and increase timeouts. |
| eng/pipelines/azure-typespec-author-benchmark-no-skill.yml | Add EvalRuns pipeline parameter, pass Runs into suite steps, and increase timeouts. |
| .github/skills/azure-typespec-author/evaluate/scripts/report-eval-results.sh | Update reporting to group multi-trial results and summarize pass@k/flakiness. |
| .github/skills/azure-typespec-author/evaluate/scripts/check-node-dependencies.js | New script to validate fixture node_modules and required TypeSpec-related packages. |
| .github/skills/azure-typespec-author/evaluate/evals/005001.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders; loosen tool-call requirement pattern. |
| .github/skills/azure-typespec-author/evaluate/evals/004003.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders; loosen tool-call requirement pattern. |
| .github/skills/azure-typespec-author/evaluate/evals/004002.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/004001.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/003002.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/003001.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002011.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002010.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002009.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002008.eval.yaml | Copy dependency-check script; add run-command graders; tighten accepted ARM list-parameter patterns. |
| .github/skills/azure-typespec-author/evaluate/evals/002007.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002006.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002005.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002004.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002003.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002002.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/002001.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/001013.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/001011.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders; relax @renamedFrom match to allow namespace prefix. |
| .github/skills/azure-typespec-author/evaluate/evals/001010.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders; relax @removed match to allow namespace prefix. |
| .github/skills/azure-typespec-author/evaluate/evals/001009.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders; relax @madeRequired match to allow namespace prefix. |
| .github/skills/azure-typespec-author/evaluate/evals/001008.eval.yaml | Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch. |
| .github/skills/azure-typespec-author/evaluate/evals/001007.eval.yaml | Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch; relax decorator matches. |
| .github/skills/azure-typespec-author/evaluate/evals/001006.eval.yaml | Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch. |
| .github/skills/azure-typespec-author/evaluate/evals/001005.eval.yaml | Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch; relax decorator matches. |
| .github/skills/azure-typespec-author/evaluate/evals/001004.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/001003.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders. |
| .github/skills/azure-typespec-author/evaluate/evals/001002.eval.yaml | Copy dependency-check script; add run-command graders; relax decorator matches to allow namespace prefix. |
| .github/skills/azure-typespec-author/evaluate/evals/001001.eval.yaml | Copy dependency-check script into the sandbox; add run-command graders to forced stimulus. |
Comment on lines
29
to
+36
| SKILL_ARG="--skill-dir .." | ||
| RUNS=${{ parameters.Runs }} | ||
| WORKERS=${{ parameters.Workers }} | ||
| RUNS_ARG="" | ||
| if [ "$RUNS" -gt 1 ]; then | ||
| RUNS_ARG="--runs $RUNS" | ||
| fi | ||
| WORKERS_ARG="--workers $WORKERS" |
Comment on lines
+27
to
29
| RESULTS_FILE = r"""$LATEST""" | ||
| REPORT_GRADERS = {"tool-calls", "skill-invocation"} | ||
|
|
JoyerJin
marked this pull request as draft
July 15, 2026 08:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.