Skip to content

build(deps): refresh bundled scanner and harness tools - #65

Merged
steipete merged 1 commit into
mainfrom
build/round8-runtime-pins
Oct 7, 2026
Merged

steipete merged 1 commit into
mainfrom
build/round8-runtime-pins

Conversation

@steipete

@steipete steipete commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Refresh five bundled runtime tools to releases that satisfy the two-day dependency cooldown, retaining Node 24 and Python 3.12.

Tool Before After
Cisco AI Skill Scanner 2.1.0 2.2.0
Snyk Agent Scan 0.6.4 0.6.8
Claude Code 2.1.278 2.1.289
Codex 0.155.1 0.160.0
Socket CLI 1.1.176 1.4.2

Newer releases inside the cooldown are held. Go modules, pinned GitHub Actions, A.I.G, and AgentVerus are already current. The latest SkillSpector source revision is also held for cooldown.

Validation on a disposable Linux host:

  • docker build -t clawscan-runtime:candidate docker/clawscan-runtime passed.
  • All eight bundled entrypoints passed --help: Codex, Claude, A.I.G, SkillSpector, Snyk, Socket, AgentVerus, and Cisco.
  • Ran clawscan benchmark SkillTrustBench --limit 10 --scanner cisco --sandbox-image <image> --output <artifact> against the current published image and the candidate. Both completed 10/10 scans with zero failures or skips. The sample contains 2 clean, 4 suspicious, and 4 malicious ground-truth cases.
  • Every case retained the same is_safe and maximum severity. Seven cases retained identical finding identities; three changed lower-severity findings, matching the changes in the checksum-verified published Cisco sources:
    • case_03510: dropped a medium pipeline finding because a Python script consumes stdin as data rather than executing it as code.
    • case_03004: dropped a medium brand finding because the updated rule requires an affiliation claim rather than a vendor mention.
    • case_02130: added a low undeclared-network-destination finding for an undocumented API host.
  • Aggregate scanner time was 63.3s → 64.4s in this single comparison; this is a smoke sample, not a performance or accuracy evaluation. No judge was configured, so all verdict evaluations abstained. Credentialed API scanners and model-backed judges were not exercised.
  • Independent Codex autoreview found no actionable P0–P3 findings.

@clawsweeper

clawsweeper Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Oct 7, 2026
@clawsweeper

clawsweeper Bot commented Oct 7, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed October 7, 2026, 6:45 AM ET / 10:45 UTC.

ClawSweeper review

What this changes

Updates five scanner and judge tools bundled in ClawScan’s Docker image and records the refresh in the changelog.

Merge readiness

✅ Ready for maintainer review

The refresh remains useful: current main retains the older pins. No actionable defect was found in the introduced changes, and the supplied runtime validation supports this bounded dependency update.

Priority: P2
Reviewed head: 8d4dd86bfd8696e3eb12e2995851c0e88a3ab488

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused dependency refresh with useful production-path smoke evidence and no actionable findings.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The captured Linux validation reports a built candidate image and a real ClawScan Cisco benchmark through the Docker execution boundary, with 10/10 completed scans and stable safety classifications. Help checks additionally establish executable startup for the bundled tools, without claiming credentialed scanner or judge coverage. No stored-data contract changes require migration proof.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The captured Linux validation reports a built candidate image and a real ClawScan Cisco benchmark through the Docker execution boundary, with 10/10 completed scans and stable safety classifications. Help checks additionally establish executable startup for the bundled tools, without claiming credentialed scanner or judge coverage. No stored-data contract changes require migration proof.
Evidence reviewed 7 items Pinned introduced changes: The verified base-to-head delta changes five Dockerfile version arguments and adds one changelog entry. Installation sources, Node 24, Python 3.12, permissions, and application code remain unchanged.
Still necessary on main: The pinned main Dockerfile still contains Cisco 2.1.0, Snyk 0.6.4, Claude Code 2.1.278, Codex 0.155.1, and Socket 1.1.176. This PR is open and unmerged; the refresh is not already implemented.
Real runtime validation: The complete captured PR body reports a successful candidate image build, all eight bundled entrypoints passing help checks, and a before/after ten-case Cisco benchmark through ClawScan’s Docker sandbox. Both images completed 10/10 scans without failures or skips, preserving safety classification and maximum severity. Credentialed scanners and model-backed judges were explicitly outside this smoke sample. Captured source identity: 34a6a9245b3f070e8d61dd5f590d840492cdf3565a191388718f3b7972692a10.
Findings None None.
Security None None.

How this fits together

ClawScan runs command-backed scanners and external judge commands in a bundled Docker image by default. The image supplies their executable dependencies; scanner results return to ClawScan as raw JSON evidence.

flowchart TD
  A[Skill files] --> B[ClawScan CLI]
  B --> C[Docker sandbox]
  D[Pinned tool packages] --> C
  C --> E[Scanner or judge command]
  E --> F[JSON evidence and verdict artifacts]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Bundled dependency refresh 5 tool pins updated; 2 files changed The patch stays within existing runtime packaging and its changelog.

Technical review

Best possible solution:

Keep the established Docker packaging and raw-evidence adapters while refreshing the pinned tools with the documented smoke coverage.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR refreshes existing dependencies rather than reporting a bug; the supplied comparison exercises the Cisco production scan path.

Is this the best way to solve the issue?

Yes: updating the existing image pins is the narrow packaging change, and no competing implementation or concrete compatibility defect was found.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against ff17e1d76e98.

Labels

Label changes:

  • add P2: This is a bounded runtime dependency improvement with no demonstrated urgent regression.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The captured Linux validation reports a built candidate image and a real ClawScan Cisco benchmark through the Docker execution boundary, with 10/10 completed scans and stable safety classifications. Help checks additionally establish executable startup for the bundled tools, without claiming credentialed scanner or judge coverage. No stored-data contract changes require migration proof.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The captured Linux validation reports a built candidate image and a real ClawScan Cisco benchmark through the Docker execution boundary, with 10/10 completed scans and stable safety classifications. Help checks additionally establish executable startup for the bundled tools, without claiming credentialed scanner or judge coverage. No stored-data contract changes require migration proof.

Label justifications:

  • P2: This is a bounded runtime dependency improvement with no demonstrated urgent regression.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The captured Linux validation reports a built candidate image and a real ClawScan Cisco benchmark through the Docker execution boundary, with 10/10 completed scans and stable safety classifications. Help checks additionally establish executable startup for the bundled tools, without claiming credentialed scanner or judge coverage. No stored-data contract changes require migration proof.
  • proof: sufficient: Contributor real behavior proof is sufficient. The captured Linux validation reports a built candidate image and a real ClawScan Cisco benchmark through the Docker execution boundary, with 10/10 completed scans and stable safety classifications. Help checks additionally establish executable startup for the bundled tools, without claiming credentialed scanner or judge coverage. No stored-data contract changes require migration proof.

Evidence

What I checked:

  • Pinned introduced changes: The verified base-to-head delta changes five Dockerfile version arguments and adds one changelog entry. Installation sources, Node 24, Python 3.12, permissions, and application code remain unchanged. (docker/clawscan-runtime/Dockerfile:5, 8d4dd86bfd86)
  • Still necessary on main: The pinned main Dockerfile still contains Cisco 2.1.0, Snyk 0.6.4, Claude Code 2.1.278, Codex 0.155.1, and Socket 1.1.176. This PR is open and unmerged; the refresh is not already implemented. (docker/clawscan-runtime/Dockerfile:5, ff17e1d76e98)
  • Real runtime validation: The complete captured PR body reports a successful candidate image build, all eight bundled entrypoints passing help checks, and a before/after ten-case Cisco benchmark through ClawScan’s Docker sandbox. Both images completed 10/10 scans without failures or skips, preserving safety classification and maximum severity. Credentialed scanners and model-backed judges were explicitly outside this smoke sample. Captured source identity: 34a6a9245b3f070e8d61dd5f590d840492cdf3565a191388718f3b7972692a10. (8d4dd86bfd86)
  • Production integration boundary: The Cisco adapter invokes skill-scanner scan with JSON file output, while the Socket and Snyk Docker paths invoke installed image executables. The built-in ClawHub profile invokes Codex as an external judge. These are affirmative dependency compatibility signals; this patch does not introduce a new provider framework or change stored artifact contracts. (internal/runner/cisco_scanner.go:22, 8d4dd86bfd86)
  • Socket upgrade compatibility: The upstream comparison from v1.1.176 to v1.4.2 adds opt-in uv workspace scanning to scan create. Its uvMembers flag defaults to false, retaining the existing path used by ClawScan. npm metadata identifies this repository, confirms Node compatibility, and dates version 1.4.2 to 2026-10-05T08:25:59.750Z, beyond the stated two-day cooldown when this PR opened. (src/commands/scan/cmd-scan-create.mts)
  • Runtime ownership history: Prior merged runtime refresh history identifies steipete as the author of the previous Docker toolchain update. Local historical blob inspection encountered unavailable promisor objects; GitHub’s read-only commit endpoint supplied the prior patch and account attribution instead. (docker/clawscan-runtime/Dockerfile, 37f74167237e)

Likely related people:

  • Patrick-Erichsen: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete
steipete merged commit 128e696 into main Oct 7, 2026
10 checks passed
@steipete
steipete deleted the build/round8-runtime-pins branch October 7, 2026 10:55
@steipete

steipete commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Merged as 128e696.

The candidate runtime image built successfully, and all eight bundled CLI entrypoints passed their help checks. The baseline and candidate each completed 10/10 Cisco SkillTrustBench scans without failures or skips, retaining the same safety flags and maximum severities. Three lower-severity finding changes were checked against the published scanner sources and are explained in the PR body. This was a scanner smoke comparison without model-backed judging or credentialed API validation.

Independent Codex autoreview was clean through P3. Exact-head CI and the two-architecture PR image build passed. After merge, main CI, CodeQL, and the automatic runtime image job all passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant