feat: enable evals on smoke-copilot by Copilot · Pull Request #45324 · github/gh-aw

Copilot · 2026-07-13T20:40:29Z

The evals feature (ADR-43641) was fully implemented but never enabled on any workflow. A stale compiler warning claimed job compilation was pending despite buildEvalsJob being complete.

Changes

Remove stale warning — compiler_orchestrator_workflow.go was warning "job compilation is pending" and incrementing warning count. Removed it along with now-unused os/console imports.
Remove stale TODO — compiler_jobs.go comment incorrectly described buildEvalsJob as a no-op stub.

Enable evals on smoke-copilot — Added 3 BinEval questions to frontmatter:

evals:
  - id: smoke-passed
    question: Did all or most smoke tests pass?
  - id: build-succeeded
    question: Did the gh-aw binary build succeed?
  - id: issue-created
    question: Was a smoke test issue created with test results?

Recompile — smoke-copilot.lock.yml gains the evals job positioned at safe_outputs → evals → conclusion.
ADR-43641 — Status promoted Draft → Accepted; Negative section corrected (had duplicated the Positive bullets).

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-13T21:13:26Z

/smoke-copilot

github-actions · 2026-07-13T21:13:55Z

📰 BREAKING: Smoke Copilot is now investigating this issue comment. Sources say the story is developing...

github-actions · 2026-07-13T21:18:54Z

Hey @copilot-swe-agent 👋 — nice work activating the evals feature on smoke-copilot and cleaning up the stale compiler warning and TODO that had been lingering since ADR-43641 was written. The changes are well-scoped and the description is thorough.

One thing that would strengthen this PR:

Add tests — the compiler changes in compiler_orchestrator_workflow.go (removing the stale warning branch) and compiler_jobs.go (removing the stub comment) touch compilation logic. A unit test asserting that buildEvalsJob no longer produces a spurious warning, and that the compiled smoke-copilot.lock.yml includes the expected evals job, would give reviewers confidence the removal is safe.

If you would like a hand, here is a prompt to assign to your coding agent:

Add unit tests covering the evals compiler changes in PR #45324:

1. In pkg/workflow, add a test asserting that compiling a workflow with a valid `evals` frontmatter block does NOT produce any "job compilation is pending" warning for the evals job.
2. Add a test asserting that the compiled output for a workflow with `evals` entries contains an `evals` job node positioned after `safe_outputs` and before `conclusion`.
3. Verify that `buildEvalsJob` is no longer treated as a no-op stub (i.e., it returns a non-nil job definition).

Generated by ✅ Contribution Check · 95.3 AIC · ⌖ 22.3 AIC · ⊞ 6.2K · ◷

github-actions · 2026-07-13T21:27:18Z

Comment Memory

Circuits hum at dawn
Bots test branches without blinking
Green lights teach the wind

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

📰 BREAKING: Report filed by Smoke Copilot · 124.2 AIC · ⌖ 7.89 AIC · ⊞ 19K · ◷
_{Comment /smoke-copilot to run again}
_{Add label smoke to run again}

github-actions

Smoke test review comments added.

📰 BREAKING: Report filed by Smoke Copilot · 124.2 AIC · ⌖ 7.89 AIC · ⊞ 19K
_{Comment /smoke-copilot to run again}
_{Add label smoke to run again}

pelikhan · 2026-07-14T02:38:52Z

@copilot enable evals on a few more agentic workflows in this repo

…ntribution-check Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-14T03:02:45Z

@copilot enable evals on a few more agentic workflows in this repo

Done. Added evals: blocks to three more agentic workflows and recompiled all of them:

auto-triage-issues — evaluates whether labels were applied (or noop correctly called) and whether a summary discussion was created
breaking-change-checker — evaluates whether the analysis completed and whether an issue was created or noop called appropriately
contribution-check — evaluates whether PRs were dispatched to the contribution-checker subagent and a report issue was created

Commit: feat: enable evals on auto-triage-issues, breaking-change-checker, contribution-check

Copilot

Pull request overview

Activates BinEval evaluation jobs, removes stale compiler messaging, and updates ADR status. The diff enables evals on four workflows, broader than the stated smoke-copilot scope.

Changes:

Removes obsolete eval warning and TODO.
Enables and recompiles BinEval jobs across four workflows.
Marks ADR-43641 Accepted and updates consequences.

Show a summary per file

File	Description
`pkg/workflow/compiler_orchestrator_workflow.go`	Removes stale warning.
`pkg/workflow/compiler_jobs.go`	Removes obsolete TODO.
`docs/adr/43641-add-bineval-evals-support.md`	Accepts and revises ADR.
`.github/workflows/smoke-copilot.md`	Adds smoke evaluations.
`.github/workflows/smoke-copilot.lock.yml`	Compiles smoke eval job.
`.github/workflows/smoke-call-workflow.lock.yml`	Includes unrelated recompilation changes.
`.github/workflows/contribution-check.md`	Adds contribution evaluations.
`.github/workflows/contribution-check.lock.yml`	Compiles contribution eval job.
`.github/workflows/breaking-change-checker.md`	Adds breaking-change evaluations.
`.github/workflows/breaking-change-checker.lock.yml`	Compiles breaking-change eval job.
`.github/workflows/auto-triage-issues.md`	Adds triage evaluations.
`.github/workflows/auto-triage-issues.lock.yml`	Compiles triage eval job.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 12/12 changed files
Comments generated: 6
Review effort level: Medium

+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        env:
+          GH_AW_EVALS_QUESTIONS: '[{"id":"smoke-passed","question":"Did all or most smoke tests pass? Look for an overall PASS status or the majority of tests showing ✅ in the agent output."},{"id":"build-succeeded","question":"Did the gh-aw binary build succeed? Look for a successful make build step or explicit mention that compilation succeeded."},{"id":"issue-created","question":"Was a smoke test issue created with test results? Look for a create_issue output containing ''Smoke Test'' in the title."}]'
+          GH_AW_EVALS_MODEL: "small"


+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        env:
+          GH_AW_EVALS_QUESTIONS: '[{"id":"labels-applied","question":"Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?"},{"id":"report-created","question":"Was a summary discussion created listing the issues processed and the labels applied?"}]'
+          GH_AW_EVALS_MODEL: "small"


+          fi
+          # shellcheck disable=SC1003,SC2016,SC2086
+          sudo -E awf --config "${RUNNER_TEMP}/gh-aw/awf-config.json" --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" ${GH_AW_TOOL_CACHE_MOUNT:+--mount "$GH_AW_TOOL_CACHE_MOUNT"} ${GH_AW_DOCKER_HOST:+--docker-host "$GH_AW_DOCKER_HOST"} --env-all --exclude-env COPILOT_GITHUB_TOKEN --log-level info --enable-host-access --allow-host-ports 80,443,8080 --skip-pull \
+            -- /bin/bash -c 'set +o histexpand; : "${RUNNER_TOOL_CACHE:?RUNNER_TOOL_CACHE must be set}"; GH_AW_TOOL_CACHE="$RUNNER_TOOL_CACHE"; export PATH="$(find "$GH_AW_TOOL_CACHE" -maxdepth 5 -type d -name bin 2>/dev/null | tr '\''\n'\'' '\'':'\'')$PATH"; [ -n "$GOROOT" ] && export PATH="$GOROOT/bin:$PATH" || true; [ -n "$ERLANG_HOME" ] && export PATH="$ERLANG_HOME/bin:$PATH" || true && GH_AW_NODE_EXEC="${GH_AW_NODE_BIN:-}"; if [ -z "$GH_AW_NODE_EXEC" ] || [ ! -x "$GH_AW_NODE_EXEC" ]; then GH_AW_NODE_EXEC="$(command -v node 2>/dev/null || true)"; fi; if [ -z "$GH_AW_NODE_EXEC" ]; then echo "node runtime missing on this runner — check runtimes.node in workflow YAML" >&2; exit 127; fi; GH_AW_NPM_GLOBAL_ROOT="$(npm root -g 2>/dev/null || true)"; if [ -n "$GH_AW_NPM_GLOBAL_ROOT" ]; then export NODE_PATH="${GH_AW_NPM_GLOBAL_ROOT}${NODE_PATH:+:${NODE_PATH}}"; fi; "$GH_AW_NODE_EXEC" ${RUNNER_TEMP}/gh-aw/actions/copilot_harness.cjs /usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all --log-dir /tmp/gh-aw/sandbox/agent/logs/ --disable-builtin-mcps --no-ask-user --agent contribution-checker --allow-all-tools --add-dir "${GITHUB_WORKSPACE}" --prompt-file /tmp/gh-aw/aw-prompts/prompt.txt' 2>&1 | tee -a /tmp/gh-aw/evals/evals.log



 **Date**: 2026-07-06
-**Status**: Draft
+**Status**: Accepted


+evals:
+  - id: prs-evaluated
+    question: Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?
+  - id: report-created
+    question: Was a report issue created summarizing PR compliance with the contributing guidelines?


+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        env:
+          GH_AW_EVALS_QUESTIONS: '[{"id":"analysis-completed","question":"Did the agent complete an analysis of recent commits and merged PRs for breaking CLI changes?"},{"id":"issue-created-or-noop","question":"Was a breaking change issue created when breaking changes were found, or was noop correctly called when no breaking changes were detected?"}]'
+          GH_AW_EVALS_MODEL: "small"


Copilot AI and others added 3 commits July 13, 2026 20:27

Start: enable evals on smoke-copilot

c7b0db9

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

feat: enable evals on smoke-copilot, remove stale pending warning

c384310

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

fix: correct ADR negative section, fix duplicate content

b729019

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 13, 2026

Copilot created this pull request from a session on behalf of pelikhan July 13, 2026 20:41 View session

Copilot AI requested a review from pelikhan July 13, 2026 20:41

Copilot finished work on behalf of pelikhan July 13, 2026 20:41

github-actions Bot mentioned this pull request Jul 13, 2026

[Contribution Check Report] Contribution Check — 2026-07-13 #45156

Closed

github-actions Bot reviewed Jul 13, 2026

View reviewed changes

github-actions Bot added pr-action:defer pr-agent:copilot pr-priority:medium pr-risk:medium pr-type:feature labels Jul 14, 2026

Copilot started work on behalf of pelikhan July 14, 2026 02:39 View session

feat: enable evals on auto-triage-issues, breaking-change-checker, co…

48b872e

…ntribution-check Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 14, 2026 03:03

pelikhan marked this pull request as ready for review July 14, 2026 03:04

Copilot AI review requested due to automatic review settings July 14, 2026 03:04

pelikhan merged commit 0e64b08 into main Jul 14, 2026

pelikhan deleted the copilot/enable-evals-on-smoke-copilot branch July 14, 2026 03:04

Copilot started reviewing on behalf of pelikhan July 14, 2026 03:04 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 14, 2026

[aw] No-Op Runs #43499

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: enable evals on smoke-copilot#45324

feat: enable evals on smoke-copilot#45324
pelikhan merged 4 commits into
mainfrom
copilot/enable-evals-on-smoke-copilot

Copilot AI commented Jul 13, 2026 •

edited

Loading

Uh oh!

pelikhan commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

pelikhan commented Jul 14, 2026

Uh oh!

Copilot AI commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Copilot AI commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

pelikhan commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Jul 14, 2026

Uh oh!

Copilot AI commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading