Skip to content

feat(security): add Endor evidence to plugin ClawScan profiles - #3742

Closed
jesse-merhi wants to merge 21 commits into
stagingfrom
jesse/endor-plugin-scan-pipeline
Closed

jesse-merhi wants to merge 21 commits into
stagingfrom
jesse/endor-plugin-scan-pipeline

Conversation

@jesse-merhi

@jesse-merhi jesse-merhi commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Plugin publication and rescans queue work immediately. The existing worker prepares Endor reachability and SkillSpector concurrently, then sends their evidence through a native custom profile in released ClawScan 0.2.0 to one AI judge. ClawHub saves and reports the result for the exact release.

Behavior

  • Keep only FINDING_TAGS_REACHABLE_FUNCTION findings, with a bounded list and full count.
  • Reuse queue, leases, retry handling and atomic completion. Endor failure is explicit; primary moderation still completes.
  • Isolate scanner credentials from dependency resolvers and the judge; disable dependency scripts, reject submitted .npmrc, reset Git metadata and clean up the owned container.
  • Endor uses dry-run. Reports appear in the ClawHub audit page and ZIP rather than persistent Endor dashboard projects.

Isolated Staging acceptance

This PR targets staging and reuses Patrick's deployment to stg.clawhub.openclaw.org, backed by cheery-civet-733. The obsolete feature-specific Test deployment changes were removed.

The manual security-scan-codex.yml Staging trial requires the exact deployed revision and 1–3 ready native plugin bulk-rescan jobs assigned only to shared shard 0. It serializes with Staging deployment, builds an immutable local scanner image and preserves the backend worker credential. It succeeds only when every assigned release has completed Endor and judge results from this worker with its lease cleared. Production Endor remains disabled.

Live trial pending: Staging currently lacks SECURITY_SCAN_WORKER_TOKEN. Native credential setup was rejected with deployment:env:write; a Staging admin must set it on the isolated backend. No current hosted Endor + real-AI acceptance result is claimed. Earlier live Slack/Discord Endor scans used previous pipeline revisions.

Review and validation

  • Original feature reviews, cleanup audit and released-ClawScan Docker proof remain applicable to unchanged runtime code. The current Staging adaptation and API/proxy integration received a scoped simplification pass and two independent findings-only reviews.
  • Fixed an incorrect workflow name and a trial that could pass while claiming zero jobs; the actual shell regression accepted an unclaimable publication job before the repair. The trial now checks both admission and stored completion.
  • Focused worker/workflow tests: 47 passed. Deployment tests: 16 passed. Integration tests: 766 passed across 12 files; root/schema/CLI typechecks passed.
  • Final local full coverage: 7,417 passed, 3 skipped. Full static checks, actionlint and type/build gate passed. Package verification passed (608 tests); Hosted CI passed on the updated PR, including API and browser smoke, authenticated browser flows, CodeQL and verified-secret checks.
  • The first full run exposed a one-second subprocess-fixture wait. Its wait now matches the existing five-second fixture budget; focused and full suites pass afterward.

The Staging adjustment, including removal of obsolete Test guards and tests, is +175 / −132 lines (net +43), excluding Patrick's existing platform changes. Retain the additive backend schema when disabling Endor so saved results remain readable.

@clawsweeper

clawsweeper Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vercel

vercel Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
clawhub Ready Ready Preview Oct 1, 2026 4:50am UTC

Request Review

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper

clawsweeper Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge. Reviewed October 1, 2026, 1:00 AM ET / 05:00 UTC (Revision 16).

ClawSweeper review

What this changes

The branch adds Endor dependency-reachability evidence to plugin security scans and exposes release-bound summaries through audit pages, APIs, and report downloads.

Merge readiness

⛔ Blocked before merge - 4 items remain

The Endor integration remains distinct work, and this member-authored PR stays open. Both previous findings remain unresolved; targeting staging makes the pending hosted trial a production-promotion requirement rather than a prerequisite to staging deployment.

Priority: P2
Reviewed head: 9dbba844140e434ac3668ed3a103b933846a8908

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The integration is coherent and substantially covered, but the two unresolved compatibility and report-scope findings prevent a clean patch verdict.
Proof confidence 🌊 off-meta tidepool Not applicable: The author is a MEMBER, so the ordinary external-contributor proof gate does not apply. The captured body preserves earlier Docker-proof claims for unchanged runtime code but explicitly leaves the new hosted Staging worker-to-Endor-to-judge-to-storage trial pending; production promotion should await that result. Stored-data compatibility is supported by the optional additive field, unchanged LLM shape, and explicit schema-preserving rollback contract.
Patch quality 🦐 gold shrimp (3/6) 2 actionable review findings remain.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The author is a MEMBER, so the ordinary external-contributor proof gate does not apply. The captured body preserves earlier Docker-proof claims for unchanged runtime code but explicitly leaves the new hosted Staging worker-to-Endor-to-judge-to-storage trial pending; production promotion should await that result. Stored-data compatibility is supported by the optional additive field, unchanged LLM shape, and explicit schema-preserving rollback contract.
Evidence reviewed 11 items Pinned change ownership: Reviewed the introduced delta from f12c6db to 9dbba84. Raw head records agree with the supplied original-head parents; current main and the verified test merge were treated as separate context.
Production upgrade remains introduced: The workflow defaults every environment to ClawScan 0.2.0 independently of the Endor enable flag. Its install step replaces the previous 0.1.8 pin.
Current-main baseline: Current main explicitly installs ClawScan 0.1.8, and its tracked scanner paths do not contain this Endor integration. No merged replacement was established from the supplied related context or bounded REST inspection.
Findings 2 actionable findings [P1] Keep ordinary Production scans on the validated ClawScan pin
[P2] Leave Endor out of skill-only report archives
Security None None.

How this fits together

ClawHub’s security worker consumes queued artifact scans, gathers scanner evidence, and asks ClawScan for the moderation verdict. Saved results then control publication visibility and feed security audit pages and downloads.

flowchart TD
  A[Plugin publication or rescan] --> B[Scan queue and lease]
  B --> C[Verified release artifact]
  C --> D[Endor and SkillSpector evidence]
  D --> E[ClawScan security judge]
  E --> F[Atomic release completion]
  F --> G[Moderation and audit reports]
Loading

Before merge

  • Keep ordinary Production scans on the validated ClawScan pin (P1) - This default selects 0.2.0 for ordinary Production jobs even when Endor is disabled, replacing current main’s 0.1.8 baseline. The captured body still provides no current ordinary skill-and-plugin upgrade acceptance, and the isolated Staging plugin trial cannot establish that compatibility. Preserve 0.1.8 for ordinary Production runs and select 0.2.0 explicitly for Staging until the wider upgrade is validated. This previous finding remains unresolved.
  • Leave Endor out of skill-only report archives (P2) - This shared ZIP builder also handles skill reports, but it unconditionally writes endor.json and describes Endor as an available scanner. Skill jobs never run Endor and their completion endpoint rejects Endor results, so those downloads contain a misleading null report. Gate the entry and related README text on an applicable Endor result, matching the conditional frontend export. This previous finding remains unresolved.
  • Resolve merge risk (P1) - The ordinary Production path upgrades to ClawScan 0.2.0 even with Endor disabled, without established real skill-and-plugin upgrade acceptance.
  • Complete next step (P2) - Preserve the ordinary Production scanner pin, explicitly select the Staging version, and remove Endor entries and instructions from skill-only HTTP report archives.

Findings

  • [P1] Keep ordinary Production scans on the validated ClawScan pin — .github/workflows/security-scan-codex.yml:73
  • [P2] Leave Endor out of skill-only report archives — convex/httpApiV1/skillsV1.ts:506
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Introduced scope 41 files; +3,836/-333 lines The change spans scanner execution, stored results, API contracts, and audit presentation.
Production versus test growth Production +1,359/-310; tests and proof fixtures +2,262/-22; generated +89/-1; spec +126 Production growth is justified by the scanner wrapper, isolation, profile integration, and result delivery described in the spec.

Merge-risk options

Maintainer options:

  1. Keep the Production baseline (recommended)
    Retain ClawScan 0.1.8 as the ordinary Production default and select 0.2.0 explicitly for the isolated Staging trial.
  2. Validate the wider upgrade
    Retain the broader default change only after recording real ordinary skill and plugin scans that demonstrate compatible completion and moderation behavior.
Copy recommended automerge instruction
@clawsweeper automerge

Special instructions:
Preserve ClawScan 0.1.8 as the ordinary Production default, explicitly select 0.2.0 for Staging, and add workflow regression coverage for both environments without enabling Production Endor.

Technical review

Best possible solution:

Use the existing worker with a staged Endor rollout, preserve the validated Production scanner default, and include Endor reports only for applicable plugin results.

Do we have a high-confidence way to reproduce the issue?

Yes, source establishes both review triggers: an unset scanner-version variable selects 0.2.0 in Production, and a skill-only HTTP report receives a null Endor archive entry. No runtime scan failure was reproduced.

Is this the best way to solve the issue?

The existing queue and atomic completion path are an appropriate implementation layer; a generic scanner-storage redesign is unnecessary for this feature. The unconditional Production upgrade and skill-only report additions should be narrowed.

Full review comments:

  • [P1] Keep ordinary Production scans on the validated ClawScan pin — .github/workflows/security-scan-codex.yml:73
    This default selects 0.2.0 for ordinary Production jobs even when Endor is disabled, replacing current main’s 0.1.8 baseline. The captured body still provides no current ordinary skill-and-plugin upgrade acceptance, and the isolated Staging plugin trial cannot establish that compatibility. Preserve 0.1.8 for ordinary Production runs and select 0.2.0 explicitly for Staging until the wider upgrade is validated. This previous finding remains unresolved.
    Confidence: 0.96
  • [P2] Leave Endor out of skill-only report archives — convex/httpApiV1/skillsV1.ts:506
    This shared ZIP builder also handles skill reports, but it unconditionally writes endor.json and describes Endor as an available scanner. Skill jobs never run Endor and their completion endpoint rejects Endor results, so those downloads contain a misleading null report. Gate the entry and related README text on an applicable Endor result, matching the conditional frontend export. This previous finding remains unresolved.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.94

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 72a32c262670.

Labels

Label changes:

  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.
  • remove merge-risk: 🚨 security-boundary: Current PR review merge-risk labels are merge-risk: 🚨 compatibility.

Label justifications:

  • P2: This is a bounded security-evidence enhancement with compatibility repairs remaining, rather than an observed urgent production incident.
  • merge-risk: 🚨 compatibility: The introduced default upgrades ordinary Production scans independently of Endor’s opt-in flag and lacks established upgrade acceptance.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: The author is a MEMBER, so the ordinary external-contributor proof gate does not apply. The captured body preserves earlier Docker-proof claims for unchanged runtime code but explicitly leaves the new hosted Staging worker-to-Endor-to-judge-to-storage trial pending; production promotion should await that result. Stored-data compatibility is supported by the optional additive field, unchanged LLM shape, and explicit schema-preserving rollback contract.

Evidence

Acceptance criteria:

  • [P1] bun run test -- scripts/security/security-scan-worker-workflow.test.ts convex/httpApiV1.handlers.test.ts.
  • [P1] bun run ci:static.
  • [P1] bun run ci:unit.
  • [P1] bun run ci:types-build.
  • [P1] bunx tsc -p packages/schema/tsconfig.json --noEmit.

What I checked:

  • Pinned change ownership: Reviewed the introduced delta from f12c6db to 9dbba84. Raw head records agree with the supplied original-head parents; current main and the verified test merge were treated as separate context. (9dbba844140e)
  • Production upgrade remains introduced: The workflow defaults every environment to ClawScan 0.2.0 independently of the Endor enable flag. Its install step replaces the previous 0.1.8 pin. (.github/workflows/security-scan-codex.yml:73, 9dbba844140e)
  • Current-main baseline: Current main explicitly installs ClawScan 0.1.8, and its tracked scanner paths do not contain this Endor integration. No merged replacement was established from the supplied related context or bounded REST inspection. (.github/workflows/security-scan-codex.yml:92, 72a32c262670)
  • Previous findings checked: The prior completed review named the Production pin and skill-only archive findings. Reading the exact earlier revision through GitHub contents confirmed both offending behaviors already existed there and remain at this head. The frontend export is conditional, but the shared HTTP ZIP builder still emits Endor unconditionally. (convex/httpApiV1/skillsV1.ts:506, 9dbba844140e)
  • Staging acceptance scope: The captured PR body explicitly reports a pending hosted trial because Staging lacks its backend worker credential; it claims no current hosted Endor-plus-real-judge result. The spec requires merging into staging and deploying the exact revision before running the bounded trial, so that trial cannot reasonably precede staging landing. (specs/endor-plugin-scans.md:91, 9dbba844140e)
  • Additive stored-data compatibility: Endor is an optional package-release field, and the shared LLM validator preserves the previous stored shape. Existing releases without Endor remain valid. The rollback instructions explicitly retain the additive schema after results have been stored; a rollback to the old schema is not supported. (convex/schema.ts:1937, 9dbba844140e)

Likely related people:

  • Patrick-Erichsen: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Separate the ordinary Production scanner baseline from the explicit Staging version selection.
  • Add regression coverage proving skill-only HTTP downloads omit Endor entries and instructions.
  • After staging deployment, record the bounded hosted acceptance result before production promotion.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (15 earlier review cycles; latest 8 shown)
  • reviewed 2026-09-22T04:36:06.556Z sha 092e082 :: needs changes before merge. :: none
  • reviewed 2026-09-23T05:46:59.299Z sha 6abbf61 :: needs changes before merge. :: none
  • reviewed 2026-09-23T05:53:54.045Z sha 6abbf61 :: needs changes before merge. :: none
  • reviewed 2026-09-23T06:00:51.518Z sha 6abbf61 :: needs maintainer review before merge. :: none
  • reviewed 2026-09-30T08:46:34.011Z sha 6abbf61 :: blocked before merge. :: none
  • reviewed 2026-10-01T00:00:54.448Z sha bcecd57 :: blocked before merge. :: none
  • reviewed 2026-10-01T01:57:24.569Z sha bcecd57 :: blocked before merge. :: none
  • reviewed 2026-10-01T02:15:42.916Z sha cd50758 :: blocked before merge. :: [P1] Keep ordinary Production scans on the validated ClawScan pin | [P2] Leave Endor out of skill-only report archives

@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. proof: sufficient Contributor real behavior proof is sufficient. labels Sep 16, 2026
@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi
jesse-merhi marked this pull request as ready for review September 23, 2026 05:55
@clawsweeper clawsweeper Bot added the proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. label Sep 23, 2026
@jesse-merhi jesse-merhi changed the title feat(security): queue Endor reachability scans for plugins feat(security): add Endor evidence to plugin ClawScan profiles Sep 30, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. labels Oct 1, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Oct 1, 2026
@jesse-merhi
jesse-merhi changed the base branch from main to staging October 1, 2026 04:48
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. labels Oct 1, 2026
@Patrick-Erichsen

Copy link
Copy Markdown
Collaborator

Superseded by #3896, which carries the Endor plugin scan work against main. I preserved the feature commits and added the Test/Production routing and worker hardening in the replacement.

This branch had an error being deployed

1 failed (outdated) and 1 active deployments
Preview – clawhub — 9dbba844 Deployed Oct 1, 2026 by vercel[bot]
Test — bcecd578 Deployed Sep 30, 2026 by jesse-merhi via deploy-test #3991
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal backlog priority with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants