fix: show LLM provenance in automation output

automations/catalog/github-pr-reviewer/manifest.json

@@ -1,6 +1,6 @@
 {
   "id": "github-pr-reviewer",
-  "version": "1.0.0",
+  "version": "1.0.2",
   "name": "GitHub code review",
   "category": "Code review",
   "description": "Watch for a configurable label on GitHub pull requests, inspect full PR and repository context, and post an AI review comment once per label event.",
@@ -88,7 +88,7 @@
       }
     },
     "bundle": {
-      "version": "1.0.0",
+      "version": "1.0.2",
       "entrypoint": "python3 main.py",
       "timeout": 600,
       "files": {

skills/github-pr-reviewer/README.md

@@ -20,6 +20,10 @@ This skill is activated by:
   checkout when the review ends, so nothing accumulates between runs
 - Publishes a real pull request review, with inline comments where a finding
   maps to a changed line, and verifies on GitHub that it landed
+- Shows the active LLM profile and model in every published review or fallback
+  result
+- Verifies and repairs the provenance footer on submitted reviews; publication
+  failures remain pending for retry
 - Posts acknowledgement comments with AI disclosure
 - Configurable review tone and polling schedule
 

skills/github-pr-reviewer/SKILL.md

@@ -261,7 +261,8 @@ For each repository:
      paths, and symlinks skipped rather than materialised.
    - Starts an OpenHands conversation **whose working directory is that
      checkout**, with a review prompt carrying PR metadata, the exact head SHA,
-     and label event details.
+     label event details, and the LLM profile/model footer required in the
+     published review.
    - Posts an acknowledgement comment with the label event, head SHA, and
      conversation link.
    - Records the review in state with `status: "active"` and the checkout path.
@@ -272,9 +273,12 @@ For each repository:
    - Suppresses stale results if the PR head SHA changed after the review was
      queued.
    - When the conversation reaches `idle`, `finished`, `error`, or `stuck`,
-     asks GitHub whether a review by the token's own user exists for that head
-     SHA. If it does, the review is complete. If it does not, the agent's final
-     response is posted as a comment so the work is not lost.
+     verifies a submitted review by the token's own user at that head SHA,
+     submitted since this conversation started, and repairs a missing or
+     incorrect LLM provenance footer before marking the review complete.
+     If no matching review exists, the agent's final response is posted as a
+     comment with provenance. Failed verification or publication is retried
+     on the next poll.
    - Abandons a conversation that has not reached a terminal status within two
      hours, so its checkout can be reclaimed.
 6. Removes the checkout of every finished review, but only after confirming the

skills/github-pr-reviewer/references/state-schema.md

@@ -81,6 +81,9 @@ Value: **ReviewRecord**
   "status": "active",
   "conversation_id": "550e8400-e29b-41d4-a716-446655440000",
   "workspace_dir": "/workspace/repositories/owner__repo/pr-42-0123456789ab",
+  "review_started_at": "2026-06-12T00:01:00Z",
+  "llm_profile": "review-profile",
+  "llm_model": "openai/review-model",
   "last_activity": 1717200000.0
 }
 ```
@@ -97,6 +100,12 @@ Value: **ReviewRecord**
 
 When a review becomes stale, `stale_reason` records the old and new head SHAs.
 When a review closes after posting, `completed_at` records the completion time.
+`llm_profile` and `llm_model` record the configuration used to start the
+conversation, including any fallback. `review_started_at` prevents an older
+review on the same commit from being attributed to that conversation when its
+provenance footer is verified or repaired. Legacy records without this timestamp
+use `trigger_label_event_created_at`; without either timestamp, collection posts
+a new fallback comment instead of modifying an existing review.
 When a review expires, `expired_after` records how many seconds it had been
 waiting.
 

skills/github-pr-reviewer/scripts/main.py

@@ -443,7 +443,7 @@ def _latest_trigger_label_event(token: str, repo: str, pr_number: int) -> dict |
     return max(matching, key=lambda event: (event.get("created_at") or "", int(event.get("id") or 0)))
 
 
-def _post_github_comment(token: str, repo: str, pr_number: int, body: str) -> None:
+def _post_github_comment(token: str, repo: str, pr_number: int, body: str) -> bool:
     try:
         _github_request(
             token,
@@ -453,9 +453,20 @@ def _post_github_comment(token: str, repo: str, pr_number: int, body: str) -> No
         )
     except Exception as exc:
         print(f"  Warning: failed to post comment on PR #{pr_number}: {exc}")
+        return False
+    return True
 
 
-def _matching_review_exists(token: str, repo: str, pr_number: int, head_sha: str) -> bool:
+def _matching_review_exists(
+    token: str,
+    repo: str,
+    pr_number: int,
+    head_sha: str,
+    *,
+    llm_profile: str | None = None,
+    llm_model: str | None = None,
+    submitted_after: str | None = None,
+) -> bool:
     """Has this token's user already published a review for this exact commit?
 
     The agent is asked to report success, but a report is not evidence: reviews
@@ -464,16 +475,29 @@ def _matching_review_exists(token: str, repo: str, pr_number: int, head_sha: str
     """
     if not head_sha or not _AUTH_LOGIN:
         return False
-    try:
-        reviews = _github_paginate(token, f"/repos/{repo}/pulls/{pr_number}/reviews")
-    except Exception as exc:
-        print(f"  Warning: could not list reviews for PR #{pr_number}: {exc}")
+    if llm_profile is not None and not submitted_after:
+        # Older state without a start time cannot attribute an existing review.
         return False
-    for review in reviews:
+    reviews = _github_paginate(token, f"/repos/{repo}/pulls/{pr_number}/reviews")
+    for review in reversed(reviews):
         if (review.get("user") or {}).get("login", "").lower() != _AUTH_LOGIN.lower():
             continue
-        if review.get("commit_id") == head_sha:
-            return True
+        if review.get("commit_id") != head_sha:
+            continue
+        if review.get("state") not in {"COMMENTED", "APPROVED", "CHANGES_REQUESTED"}:
+            continue
+        if submitted_after and (review.get("submitted_at") or "") < submitted_after:
+            continue
+        if llm_profile is not None and llm_model is not None:
+            body = _with_llm_provenance(review.get("body", ""), llm_profile, llm_model)
+            if body != review.get("body"):
+                _github_request(
+                    token,
+                    "PUT",
+                    f"/repos/{repo}/pulls/{pr_number}/reviews/{review['id']}",
+                    body={"body": body},
+                )
+        return True
     return False
 
 
@@ -620,6 +644,11 @@ def _oh_request(agent_url: str, api_key: str, method: str, path: str, body: dict
 
 
 def _fetch_settings(agent_url: str, api_key: str) -> dict:
+    """Fetch the concrete LLM config used to serialize the child agent.
+
+    Plaintext is returned only to this trusted script and sent straight back to
+    the same authenticated Agent Server in the conversation creation request.
+    """
     req = urllib.request.Request(
         f"{agent_url}/api/settings",
         headers={"X-Session-API-Key": api_key, "X-Expose-Secrets": "plaintext"},
@@ -628,14 +657,22 @@ def _fetch_settings(agent_url: str, api_key: str) -> dict:
         return json.loads(r.read())
 
 
-def _get_agent_dict(agent_url: str, api_key: str) -> dict:
+def _get_agent_and_llm_provenance(
+    agent_url: str, api_key: str
+) -> tuple[dict, str, str]:
     data = _fetch_settings(agent_url, api_key)
     llm = data.get("agent_settings", {}).get("llm", {})
-    return {
-        "kind": "Agent",
-        "llm": llm,
-        "tools": [{"name": "terminal"}, {"name": "file_editor"}],
-    }
+    profile_name = data.get("active_profile") or "default"
+    model = llm.get("model") or "unknown"
+    return (
+        {
+            "kind": "Agent",
+            "llm": llm,
+            "tools": [{"name": "terminal"}, {"name": "file_editor"}],
+        },
+        profile_name,
+        model,
+    )
 
 
 def _get_mcp_config(agent_url: str, api_key: str) -> dict | None:
@@ -682,10 +719,11 @@ def create_conversation(
     api_key: str,
     initial_message: str,
     workspace_dir: Path,
+    agent: dict | None = None,
 ) -> str:
     payload: dict = {
         "workspace": {"working_dir": str(workspace_dir)},
-        "agent": _get_agent_dict(agent_url, api_key),
+        "agent": agent or _get_agent_and_llm_provenance(agent_url, api_key)[0],
         "initial_message": {"content": [{"text": initial_message}]},
     }
     secrets = _build_secrets_payload(agent_url, api_key)
@@ -749,6 +787,20 @@ def _with_ai_disclosure(body: str) -> str:
     return f"{body}\n\n{disclosure}" if body else disclosure
 
 
+def _llm_provenance(profile: str, model: str) -> str:
+    return f"LLM profile: `{profile}` · Model: `{model}`"
+
+
+def _with_llm_provenance(body: str, profile: str, model: str) -> str:
+    provenance = _llm_provenance(profile, model)
+    body = "\n".join(
+        line
+        for line in (body or "").splitlines()
+        if not re.fullmatch(r"LLM profile: .* · Model: .*", line.strip())
+    ).strip()
+    return f"{body}\n\n{provenance}" if body else provenance
+
+
 def _load_repo_review_guide(workspace_dir: Path) -> str | None:
     """Read the repo-specific review guide from the checked-out repository.
 
@@ -770,7 +822,15 @@ def _load_repo_review_guide(workspace_dir: Path) -> str | None:
     return None
 
 
-def _build_review_prompt(repo: str, pr: dict, head_sha: str, label_event: dict, repo_review_guide: str | None = None) -> str:
+def _build_review_prompt(
+    repo: str,
+    pr: dict,
+    head_sha: str,
+    label_event: dict,
+    repo_review_guide: str | None = None,
+    llm_profile: str = "default",
+    llm_model: str = "unknown",
+) -> str:
     number = pr.get("number", "?")
     title = pr.get("title", "(no title)")
     body = (pr.get("body") or "").strip() or "(no description)"
@@ -827,10 +887,12 @@ def _build_review_prompt(repo: str, pr: dict, head_sha: str, label_event: dict,
         "If the API rejects the inline positions, retry with every finding in the body and no `comments` array.\n"
         "6. Begin the review body with this disclosure: "
         "`_This review was posted by an AI agent (OpenHands)._`\n"
-        "7. End the review body with a verdict on its own line: either `✅ APPROVED` "
+        "7. End the assessment with a verdict on its own line: either `✅ APPROVED` "
         "or `🔄 CHANGES REQUESTED`.\n"
-        "8. If there are no material issues, still publish a review saying so, with the "
-        "disclosure and the verdict.\n"
+        "8. After the verdict, append this exact provenance footer on its own line:\n"
+        f"{_llm_provenance(llm_profile, llm_model)}\n"
+        "9. If there are no material issues, still publish a review saying so, with the "
+        "disclosure, verdict, and provenance footer.\n"
         f"\nReview instructions:\n{tone}{extra}{guide_section}\n\n"
         "After GitHub accepts the review, output exactly `GITHUB_REVIEW_POSTED`. "
         "If publishing still fails after the fallback in step 5, output the complete review text "
@@ -882,8 +944,22 @@ def _process_review_request(
         repo_review_guide = _load_repo_review_guide(workspace_dir)
         if repo_review_guide:
             print(f"  Injected repo review guide for PR #{number}")
-        prompt = _build_review_prompt(repo, pr, head_sha, label_event, repo_review_guide)
-        conv_id = create_conversation(agent_url, api_key, prompt, workspace_dir)
+        agent, llm_profile, llm_model = _get_agent_and_llm_provenance(
+            agent_url, api_key
+        )
+        prompt = _build_review_prompt(
+            repo,
+            pr,
+            head_sha,
+            label_event,
+            repo_review_guide,
+            llm_profile,
+            llm_model,
+        )
+        review_started_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+        conv_id = create_conversation(
+            agent_url, api_key, prompt, workspace_dir, agent=agent
+        )
     except Exception as exc:
         # The claim is dropped so the next poll retries this label event. The
         # checkout goes with it rather than being left behind.
@@ -899,6 +975,9 @@ def _process_review_request(
             "status": "active",
             "conversation_id": conv_id,
             "workspace_dir": str(workspace_dir),
+            "llm_profile": llm_profile,
+            "llm_model": llm_model,
+            "review_started_at": review_started_at,
             "last_activity": time.time(),
         }
     )
@@ -972,31 +1051,62 @@ def _check_conversation_completion(
     except Exception:
         final = ""
 
+    llm_profile = rec.get("llm_profile", "default")
+    llm_model = rec.get("llm_model", "unknown")
     if status in {"error", "stuck"}:
-        _post_github_comment(
+        posted = _post_github_comment(
             github_token,
             repo,
             pr_number,
-            _with_ai_disclosure(
-                f"⚠️ **OpenHands PR Reviewer encountered a problem** at commit `{reviewed_sha[:12]}` "
-                f"(status: `{status}`).\n\n{final}".strip()
+            _with_llm_provenance(
+                _with_ai_disclosure(
+                    f"⚠️ **OpenHands PR Reviewer encountered a problem** at commit `{reviewed_sha[:12]}` "
+                    f"(status: `{status}`).\n\n{final}".strip()
+                ),
+                llm_profile,
+                llm_model,
             ),
         )
-    elif _matching_review_exists(github_token, repo, pr_number, reviewed_sha):
-        print(f"  PR #{pr_number}: review confirmed on GitHub at {reviewed_sha[:12]}")
+        if not posted:
+            return
     else:
-        # The agent was asked to publish the review itself; it did not, so the
-        # work is not lost - post whatever it produced as a comment.
-        _post_github_comment(
-            github_token,
-            repo,
-            pr_number,
-            _with_ai_disclosure(
-                final
-                or f"✅ **OpenHands completed the review for commit `{reviewed_sha[:12]}`.** No review text was produced."
-            ),
-        )
-        print(f"  PR #{pr_number}: no review found on GitHub; posted the result as a comment")
+        try:
+            found = _matching_review_exists(
+                github_token,
+                repo,
+                pr_number,
+                reviewed_sha,
+                llm_profile=llm_profile,
+                llm_model=llm_model,
+                submitted_after=(
+                    rec.get("review_started_at")
+                    or rec.get("trigger_label_event_created_at")
+                ),
+            )
+        except Exception as exc:
+            print(f"  Warning: could not verify or complete review provenance for PR #{pr_number}: {exc}")
+            return
+        if found:
+            print(f"  PR #{pr_number}: review and provenance confirmed on GitHub at {reviewed_sha[:12]}")
+        else:
+            if final.strip() == "GITHUB_REVIEW_POSTED":
+                final = ""
+            posted = _post_github_comment(
+                github_token,
+                repo,
+                pr_number,
+                _with_llm_provenance(
+                    _with_ai_disclosure(
+                        final
+                        or f"✅ **OpenHands completed the review for commit `{reviewed_sha[:12]}`.** No review text was produced."
+                    ),
+                    llm_profile,
+                    llm_model,
+                ),
+            )
+            if not posted:
+                return
+            print(f"  PR #{pr_number}: no review found on GitHub; posted the result as a comment")
 
     rec["status"] = "closed"
     rec["completed_at"] = time.time()