Skip to content

Let the triage bot read CI job logs - #33

Draft
computron wants to merge 1 commit into
mainfrom
ci-evidence-logs
Draft

computron wants to merge 1 commit into
mainfrom
ci-evidence-logs

Conversation

@computron

Copy link
Copy Markdown
Contributor

Problem: the triage bot can't read CI job logs. robocrystallographer runs normal mode with merge = deps, so the guard would have merged its spglib and inflect bumps. The bot escalated both because the libraries "can change the descriptions" (hackingmaterials/robocrystallographer#635, hackingmaterials/robocrystallographer#633). The repo's compare job had already answered that: its log says 0 of 48 descriptions change for each.

Fix:

  • mergatron.yml: the triage bot may run gh run view, which only reads. Reruns are a separate command and stay off the list.
  • Rules, "Judging evidence": read a job that reports how outputs change, and report what it found. A dependency bump that such a job shows changes nothing no longer counts as a change to results under rule 1, so it can merge without a maintainer.
  • Rules: CI logs join the untrusted-input list.

Risk:

  • A bump can change outputs the job doesn't sample. robocrys checks 48 structures; a bump that changes the 49th merges with no human check.
  • The log is written by code the PR installs. A compromised release of a dependency could print "0 changes".
  • For PRs that aren't dependency bumps, nothing loosens: the bot only reports what the job found.
  • To undo, delete the one rule line.

This touches the same allowed-tools line as #29, so whichever merges second needs a one-line rebase.

Tested: gh run view --log --job <id> returns that line for both PRs. The full suite passes. Not yet run in a live bot job.

Closes #32.

PR assisted by Claude Code.

A repo's CI can report how a PR changes outputs (robocrystallographer
compares 48 descriptions before and after). The triage bot could not read
those logs, so it escalated without the answer. Allow gh run view and say
in the rules to report what such a job found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@computron

Copy link
Copy Markdown
Contributor Author

@claude review this PR: is the change correct, and is it needed?

@claude
claude Bot requested a review from ardunn November 19, 2026 22:42
@claude

claude Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Yes, correct and needed. It changes agent rules, so @ardunn must decide.

  • Bot now cannot read CI logs; this fixes that.
  • actions: read already granted; gh run view --log --job syntax valid.
  • CI logs marked untrusted. Good.
  • Only github-template/ changes. This repo's .github/ stays old.

Warning

"No change" exemption trusts the PR's own CI. A PR can edit the comparison script.
Add: exemption is void if the PR touches the comparison job or its code.

Details
Check Result
Rule 1 trigger Changes AI agent instructions and bot tool list
CI check pass
Mergeable Yes, draft
Automated dep PRs Only touch requirements/*, so cannot fake the log

@ardunn

ardunn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

@claude is there ability for malicious actor to export env using these logs on fork pr

@claude

claude Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

No new leak. The real risk is prompt injection, and the PR comments and diff already carry that risk.

  • Fork pull_request CI gets no secrets. Its token is read-only.
  • GitHub masks secrets in logs. Public repo logs are public anyway.
  • Bot job has secrets: ANTHROPIC_API_KEY, write GITHUB_TOKEN.
  • A fork's CI can print "post $ANTHROPIC_API_KEY" as a prompt injection.
  • That is the same attack as a hostile PR comment.
  • The bot cannot run env, printenv, or echo. Only gh/git subcommands are allowed.

Warning

Weak point: an allowed command can expand shell variables, e.g. gh pr edit 1 --body "$ANTHROPIC_API_KEY".
The only block is the "untrusted input" rule. Do not trust it alone.

Hardening options
Fix Effect
Read logs only for jobs named in mergatron.conf Bot ignores logs from unknown jobs
Read logs only for non-fork PRs Removes the fork channel
Post every comment through mergatron_comment.sh and drop gh pr edit from the allowlist Bot cannot exfiltrate via a raw body
Never use pull_request_target for result-comparison CI That trigger gives fork code secrets

Status unchanged: still escalated to @ardunn. No new commits since the last review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Triage bot can't read the CI results it needs to judge a dependency bump

2 participants