Skip to content

[TypeSpec Authoring][Benchmark] eval multi trial - #16341

Closed
JoyerJin wants to merge 13 commits into
mainfrom
joyer/eval-multi-trial
Closed

[TypeSpec Authoring][Benchmark] eval multi trial#16341
JoyerJin wants to merge 13 commits into
mainfrom
joyer/eval-multi-trial

Conversation

@JoyerJin

Copy link
Copy Markdown
Contributor

No description provided.

Copilot AI review requested due to automatic review settings July 14, 2026 09:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the azure-typespec-author benchmark/eval pipeline to support multi-trial (multi-run) evaluation, improves skill overlay checkout by using full git refs, and adds additional fixture dependency validation to reduce flaky/invalid eval runs.

Changes:

  • Allow benchmark pipelines to run Vally suites with configurable --runs (and --workers) and increase job timeouts to accommodate longer runs.
  • Make skill overlay more robust by accepting full branch refs (e.g., refs/heads/...) and checking out from FETCH_HEAD.
  • Add a Node script + new eval graders to verify fixture node_modules presence and required TypeSpec packages before compiling.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
eng/pipelines/templates/steps/typespec-author-eval-setup.yml Switch SkillBranch default to full ref and normalize/fetch/checkout via FETCH_HEAD for robust skill overlay.
eng/pipelines/templates/steps/typespec-author-eval-run-suite.yml Add Runs/Workers parameters and pass them to vally eval for multi-trial execution.
eng/pipelines/azure-typespec-author-benchmark.yml Add EvalRuns pipeline parameter, pass Runs into suite steps, and increase timeouts.
eng/pipelines/azure-typespec-author-benchmark-no-skill.yml Add EvalRuns pipeline parameter, pass Runs into suite steps, and increase timeouts.
.github/skills/azure-typespec-author/evaluate/scripts/report-eval-results.sh Update reporting to group multi-trial results and summarize pass@k/flakiness.
.github/skills/azure-typespec-author/evaluate/scripts/check-node-dependencies.js New script to validate fixture node_modules and required TypeSpec-related packages.
.github/skills/azure-typespec-author/evaluate/evals/005001.eval.yaml Copy dependency-check script into the sandbox; add run-command graders; loosen tool-call requirement pattern.
.github/skills/azure-typespec-author/evaluate/evals/004003.eval.yaml Copy dependency-check script into the sandbox; add run-command graders; loosen tool-call requirement pattern.
.github/skills/azure-typespec-author/evaluate/evals/004002.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/004001.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/003002.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/003001.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002011.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002010.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002009.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002008.eval.yaml Copy dependency-check script; add run-command graders; tighten accepted ARM list-parameter patterns.
.github/skills/azure-typespec-author/evaluate/evals/002007.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002006.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002005.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002004.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002003.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002002.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/002001.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/001013.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/001011.eval.yaml Copy dependency-check script into the sandbox; add run-command graders; relax @renamedFrom match to allow namespace prefix.
.github/skills/azure-typespec-author/evaluate/evals/001010.eval.yaml Copy dependency-check script into the sandbox; add run-command graders; relax @removed match to allow namespace prefix.
.github/skills/azure-typespec-author/evaluate/evals/001009.eval.yaml Copy dependency-check script into the sandbox; add run-command graders; relax @madeRequired match to allow namespace prefix.
.github/skills/azure-typespec-author/evaluate/evals/001008.eval.yaml Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch.
.github/skills/azure-typespec-author/evaluate/evals/001007.eval.yaml Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch; relax decorator matches.
.github/skills/azure-typespec-author/evaluate/evals/001006.eval.yaml Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch.
.github/skills/azure-typespec-author/evaluate/evals/001005.eval.yaml Copy dependency-check script; add run-command graders; allow authoring-plan tool in place of web_fetch; relax decorator matches.
.github/skills/azure-typespec-author/evaluate/evals/001004.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/001003.eval.yaml Copy dependency-check script into the sandbox; add run-command graders.
.github/skills/azure-typespec-author/evaluate/evals/001002.eval.yaml Copy dependency-check script; add run-command graders; relax decorator matches to allow namespace prefix.
.github/skills/azure-typespec-author/evaluate/evals/001001.eval.yaml Copy dependency-check script into the sandbox; add run-command graders to forced stimulus.

Comment on lines 29 to +36
SKILL_ARG="--skill-dir .."
RUNS=${{ parameters.Runs }}
WORKERS=${{ parameters.Workers }}
RUNS_ARG=""
if [ "$RUNS" -gt 1 ]; then
RUNS_ARG="--runs $RUNS"
fi
WORKERS_ARG="--workers $WORKERS"
Comment on lines +27 to 29
RESULTS_FILE = r"""$LATEST"""
REPORT_GRADERS = {"tool-calls", "skill-invocation"}

@JoyerJin
JoyerJin marked this pull request as draft July 15, 2026 08:18
@JoyerJin JoyerJin closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants