[TypeSpec Authoring][Benchmark] typespec authoring eval archetype integration - #16479
[TypeSpec Authoring][Benchmark] typespec authoring eval archetype integration#16479JoyerJin wants to merge 66 commits into
Conversation
…_generate_authoring_plan' tool grader
edit or bash tool call
| - name: PythonVersion | ||
| type: string | ||
| default: '3.10' | ||
| - name: McpKind |
There was a problem hiding this comment.
this is separated, like we discussed this pipeline run is either Live-only or Mock-only. for PR description and track naming, it should not claim that Forced is always Live and Trigger is always Mock.
There was a problem hiding this comment.
this pipeline cannot test MCP whatever the name use
There was a problem hiding this comment.
As we discussed offline, currently code quality (live MCP) and skill invocation (mock MCP) tests are in the same pipeline because both are expected to run from branch builds rather than PR validation.
In our case, the mock MCP tests are closer to integration tests than traditional unit tests. My assumption is that unit tests typically complete within seconds or a couple of minutes and can run as part of PR validation. However, for our mock tests:
- The execution time is comparable to the live tests. The mock tests take over 20 minutes to complete, while the live tests take around 30 minutes.
- They also cannot be triggered through PR validation because they rely on Vally and require the Copilot SDK.
Based on our discussion, my understanding is that there are two possible approaches:
- Create separate pipelines for the mock and live tests now.
- Wait for the SDK evaluation framework to support running both mock and live tests within a single pipeline.
Given that, do you think we should create separate pipelines in this PR, or merge the current PR first and wait for the framework support before consolidating them into a single pipeline?
Would love to hear your thoughts.
There was a problem hiding this comment.
According to result of 20260806.8 with mock, the forced and trigger did not get worst. Based on the summary, the mock without auth can work on benchmark.
|
|
||
| # This runner keeps the pipeline threshold out of `vally eval`: TypeSpec | ||
| # gates the completed results.jsonl without overriding eval scoring. | ||
| - task: AzureCLI@2 |
There was a problem hiding this comment.
for Mock one, we don't need AzureCLI@2 auth
There was a problem hiding this comment.
we need AzureCLI@2 auth because we need to query the SDK Knowledge base backend, this backend needs auth. We currently shared the environment set up codes between mock test and live test.
similar as this comment
| # publish the results directory rather than the common junit-only path. | ||
| - output: pipelineArtifact | ||
| displayName: 'Publish $(shardName) eval results' | ||
| artifactName: '${{ parameters.matrixStageName }}-eval-result-$(shardName)-$(System.JobAttempt)' |
There was a problem hiding this comment.
The common summary code recognizes retry attempts only when an artifact directory starts with eval-result-. but here the '${{ parameters.matrixStageName }}-eval-result-$(shardName)-$(System.JobAttempt)',
the shared parser does not recognize 2 as System.JobAttempt; it treats that whole string as a new shard name with attempt 1.
Fix #16269
Summary
This PR implements the main parts of the Azure TypeSpec Author benchmark integration plan for moving the benchmark onto the shared eval infrastructure.
It reuses the common Prepare → Eval → Summary flow for the no-skill benchmark, preserves the intended forced/live and trigger/mock behavior for the with-skill benchmark, fixes MCP runtime staging, and updates the Vally eval specs so they are compatible with
@microsoft/vally-cli@0.7.0.Implementation Status
Completed from the integration plan:
archetype-eval.yml.forced/liveuses the live Azure SDK CLI MCP server with Azure authentication.trigger/mockuses the mock MCP server without Azure authentication.no-skill/liveon the live Azure SDK CLI MCP server without Azure authentication.Validation
Validated locally that scoring weights are normalized, grader types are covered, and the adjusted thresholds preserve the expected pass/fail behavior.