Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419)

[Sushant2504]

Summary

• Adds CodeRabbit AI review as a complementary finding source for FullSend reviews
• Implements dual-mode ingestion: sandbox-safe GitHub comment scraping (S3) and runner-side CLI pre-script (S2)
• CodeRabbit findings are prefixed with coderabbit-* categories for proper deduplication alongside built-in review dimensions

Changes

New Files:

• .fullsend/harness/review.yaml - Extends upstream review harness (agents v0.38.0) with coderabbit-review skill
• .fullsend/skills/coderabbit-review/SKILL.md - Skill documentation with CI setup instructions
• .fullsend/skills/coderabbit-review/scripts/run-coderabbit.sh - Dual-mode finding collector (comment/CLI)

Key Features:

• S3 Mode (default): Sandbox-compatible, reads existing CodeRabbit PR comments via gh CLI
• S2 Mode (production): Runner-side pre-script using CodeRabbit CLI with API key (never enters sandbox)
• Filters stale comments by commit SHA, respects path exclusions from .coderabbit.yaml
• Non-fatal: emits empty findings if CodeRabbit unavailable

Security:

• CODERABBIT_API_KEY isolated to runner-side pre-script only
• Findings injected via host_files mapping
• No credentials or network access to api.coderabbit.ai from sandbox

Test Plan

• Verify S3 mode reads existing CodeRabbit comments from a PR
• Verify S2 mode produces coderabbit-findings.json when API key is configured
• Confirm path exclusions work (boilerplate, vendor, generated files filtered)
• Test graceful degradation when CodeRabbit is unavailable (emits [])
• Verify coderabbit-* category prefixes in findings output
• Run /fs-review on a PR with CodeRabbit comments and verify synthesis

Related

• JIRA: ROSAENG-62419
• Upstream: Extends FullSend agents v0.38.0 review harness
• Docs: See .fullsend/skills/coderabbit-review/SKILL.md for CI setup

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Team

Run ID: 38f2ac2b-792f-4ac6-97fc-9d1ab8909c12

---

_{Comment @coderabbitai help to get the list of available commands.}

[samanthajayasinghe]

/lgtm
/approve

[openshift-ci]

@Sushant2504: Closed this PR.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.39%. Comparing base (1c770c2) to head (9628bda).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #364   +/-   ##
=======================================
  Coverage   65.39%   65.39%           
=======================================
  Files          23       23           
  Lines        1598     1598           
=======================================
  Hits         1045     1045           
  Misses        473      473           
  Partials       80       80           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 9:44 AM UTC · Completed 9:56 AM UTC

Commit: 9628bda · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.60

[Sushant2504]

/reopen

[openshift-ci]

@Sushant2504: Reopened this PR.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[fullsend-ai-retro]

Retro: PR #364 — CodeRabbit review skill for FullSend

Outcome: Closed without merge. No agent code, no agent review on this PR. The workflow guardrails functioned correctly.

Timeline

PR #364 is the fourth attempt at JIRA ROSAENG-62419 (adding CodeRabbit as a complementary review source for FullSend). All four PRs were closed on 2026-09-07:

PR	Author	Files	What happened
#340	Sushant2504	21	Original spike, too broad, CI failures. Closed by author.
#362	Sushant2504	6	Focused retry, CI passed. Closed by samanthajayasinghe as duplicate of #364.
#363	samanthajayasinghe	3	Minimal scope. Review agent ran (claude-opus-4-6, high effort) and requested changes: 1 high (harness not registered in config.yaml), 2 medium (doc inaccuracies, secrets exposure via CLI flag), 12 low. Closed, superseded by #364.
#364	Sushant2504	3	Same 3 files and 295 additions as #363. Review agent skipped (external fork contributor lacks triage permission). /lgtm /approve from samanthajayasinghe within 15 min. OWNERS gate blocked merge (needed bergmannf). Closed via /close.

Why the review agent was skipped on PR #364

The routing logic requires the PR author to either be a [bot] or have at least triage-level collaborator permission. Sushant2504 is an external fork contributor without collaborator permissions on openshift/ocm-agent-operator, so the permission check correctly failed and no review stage was dispatched. This is the intended security behavior for external contributors.

What worked well

1. Review agent quality on (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363: Found real issues — configuration gap, documentation inaccuracies, secrets exposure via --api-key CLI flag. These are actionable findings that should be addressed before this code merges.
2. OWNERS approval gate: Correctly prevented merge of Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 despite /lgtm — required approval from bergmannf which was never granted.
3. Permission-gated routing: Correctly prevented automatic review dispatch for an external fork contributor.

Evidence supporting existing open issues

• Review agent: detect close-and-recreate PRs and reuse prior review fullsend-ai/fullsend#2175 (detect close-and-recreate PRs and reuse prior review): PR (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 received a thorough review that found real issues. PR Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 contains identical code but received no review because the author changed from a collaborator to an external contributor. The review findings from (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 were not surfaced on Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364, and a human approved without addressing them. This is a concrete instance of the close-and-recreate pattern where prior findings are lost.
• Overlay condition eval failures should use structured fallback instead of logging opaque errors fullsend-ai/fullsend#6934 (overlay condition eval failures): The warning overlay[2].when eval failed: no such key: source appeared in harness dispatch logs on multiple runs (34095126076, 34096540276, 34107053660). Non-fatal but recurring.
• Deduplicate retro analysis across cross-branch sibling PRs with identical diffs fullsend-ai/fullsend#4571 (deduplicate retro across sibling PRs): Both PR (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 (run 34106979073) and PR Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 (run 34107584562) triggered retro agents for essentially identical code diffs.
• Skip retro dispatch for closed-without-merge PRs with immediate successor fullsend-ai/fullsend#2176 (skip retro for closed-without-merge PRs with immediate successor): PR Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 was closed without merge and is the latest in a chain of 4 superseding PRs, all closed. The retro dispatch could have been skipped.

[Sushant2504]

/test osd-gcp-e2e

[openshift-ci]

@Sushant2504: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[samanthajayasinghe]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: samanthajayasinghe, Sushant2504
Once this PR has been reviewed and has the lgtm label, please assign bergmannf for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 1:09 AM UTC · Completed 1:21 AM UTC

Commit: 9628bda · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.97

[fullsend-ai-retro]

Retro: PR #364 — CodeRabbit review skill merged with unresolved review findings

Outcome: PR #364 merged 3 .fullsend/ configuration files (coderabbit-review skill) that are functionally identical to PR #363. PR #363 received thorough agent review (1 HIGH, 2 MEDIUM, 10 LOW) and CodeRabbit review (4 findings). PR #364 was correctly skipped for review (author Sushant2504 is an external fork contributor without triage permissions), and none of the #363 review findings were addressed before merge.

Timeline

Time	Event
Aug 27	PR #340 opened (spike, 21 files). Closed by author — too broad.
Sep 7, 06:13	PR #362 opened (6 files). Closed as duplicate of #364.
Sep 7, 07:16	PR #363 opened by samanthajayasinghe (collaborator, 3 files). Review agent ran — requested changes (1 HIGH, 2 MEDIUM, 10 LOW). CodeRabbit posted 4 inline findings.
Sep 7, 09:22	PR #364 opened by Sushant2504 (external contributor, same 3 files). Review agent correctly skipped.
Sep 7, 09:37	samanthajayasinghe posted /lgtm /approve on #364 (15 min after open). Closed #363 as superseded.
Sep 7, 09:43	PR #364 closed via /close, then reopened at 09:53.
Sep 9, 01:07	samanthajayasinghe posted /approve. OWNERS gate flagged NOT APPROVED (needs bergmannf).
Sep 9, 01:08	PR #364 merged by samanthajayasinghe.

Key findings

1. Unresolved review findings merged into master. The review agent's HIGH finding — .fullsend/harness/review.yaml not registered in .fullsend/config.yaml — was confirmed: only triage and code have agent entries; review is listed under roles: but has no source entry. CodeRabbit's findings about pagination breaking JSON.parse and --plain fallback silently discarding findings are also real bugs. No follow-up issue exists.

2. File content is identical between PRs (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 and Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 — the only difference is that Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 is missing POSIX trailing newlines on all 3 files.

Evidence supporting existing issues

• Enable review dispatch for fork PRs from trusted contributors fullsend-ai/fullsend#2967 (review dispatch for fork contributors): PR Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 is a direct instance — review was correctly skipped for external contributor, but this meant 13 agent findings and 4 CodeRabbit findings from sibling PR (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 were never surfaced on the merged PR.
• Review agent: detect close-and-recreate PRs and reuse prior review fullsend-ai/fullsend#2175 (close-and-recreate review carryover): PR (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 was closed and Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 opened with identical code. All review findings were lost in the transition.
• Extend retro skip logic to merged PRs with zero fullsend agent involvement fullsend-ai/fullsend#6297 (retro skip for zero-agent-involvement merged PRs): This retro is itself an instance — no agent review, triage, or code work occurred on PR Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364, yet two retro runs were dispatched (costing ~$4.60+ each).
• Deduplicate retro analysis across cross-branch sibling PRs with identical diffs fullsend-ai/fullsend#4571 (sibling PR dedup): PRs (ROSAENG-62419) Add coderabbit-review skill for FullSend reviews #363 and Feat: add coderabbit-review skill for fullsend reviews (ROSAENG-62419) #364 both triggered retros for functionally identical code.

Proposals filed

• Address unresolved review findings in merged coderabbit-review skill (in openshift/ocm-agent-operator)