Skip to content

Refresh README and benchmark snapshots #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
benvinegar merged 1 commit into from
Apr 27, 2026
Merged

Refresh README and benchmark snapshots #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 27 additions & 27 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ slop-scan delta --base ../main --fail-on added,worsened

## What it catches

Current checks focus on patterns that often show up in unreviewed generated code:
Current default-pack checks focus on patterns that often show up in unreviewed generated code:

- [log-and-continue catch blocks](src/rules/error-swallowing/README.md)
- [error-obscuring catch blocks](src/rules/error-obscuring/README.md) (default-return or generic replacement error)
Expand Down Expand Up @@ -166,38 +166,38 @@ _Why before Jan 1, 2025?_ Because this cutoff aims to catch mature OSS before AI

| Metric | AI median | Mature OSS median | Ratio |
| ------------------- | --------: | ----------------: | --------: |
| Blended score | **3.02** | **1.00** | **3.02x** |
| Score / file | **0.99** | **0.24** | **4.11x** |
| Score / KLOC | **9.51** | **4.04** | **2.35x** |
| Score / function | **0.22** | **0.10** | **2.28x** |
| Findings / file | **0.31** | **0.08** | **3.74x** |
| Findings / KLOC | **2.96** | **1.38** | **2.14x** |
| Findings / function | **0.08** | **0.04** | **2.21x** |
| Blended score | **6.91** | **1.00** | **6.91x** |
| Score / file | **1.07** | **0.12** | **8.82x** |
| Score / KLOC | **10.90** | **1.48** | **7.38x** |
| Score / function | **0.28** | **0.05** | **5.51x** |
| Findings / file | **0.29** | **0.04** | **6.97x** |
| Findings / KLOC | **3.04** | **0.56** | **5.39x** |
| Findings / function | **0.08** | **0.02** | **3.51x** |

### Rolling benchmark snapshot

Latest default-branch history, still normalized against the frozen pinned baseline. Ordered by latest pinned score.

| Repository | Cohort | Latest ref | Current blended | Latest pinned | Highest pinned | Δ prev | Δ peak |
| --------------------------------------------------------------------- | ---------- | ---------------- | --------------: | ------------: | -------------: | -----: | -----: |
| [`garrytan/gstack`](https://github.com/garrytan/gstack) | ai | `main@c6e6a21` | **4.59** | **4.77** | **6.37** | -0.64 | -1.60 |
| [`redwoodjs/agent-ci`](https://github.com/redwoodjs/agent-ci) | ai | `main@c61f27d` | **3.76** | **3.91** | **3.91** | +0.51 | 0.00 |
| [`jiayun/DevWorkbench`](https://github.com/jiayun/DevWorkbench) | ai | `main@ea50862` | **3.26** | **3.39** | **3.40** | 0.00 | -0.02 |
| [`robinebers/openusage`](https://github.com/robinebers/openusage) | ai | `main@06113d6` | **2.91** | **3.03** | **3.06** | +0.01 | -0.03 |
| [`openclaw/openclaw`](https://github.com/openclaw/openclaw) | ai | `main@1de5610` | **2.81** | **2.92** | **3.15** | -0.23 | -0.23 |
| [`FullAgent/fulling`](https://github.com/FullAgent/fulling) | ai | `main@d95060f` | **2.07** | **2.16** | **2.16** | 0.00 | 0.00 |
| [`emdash-cms/emdash`](https://github.com/emdash-cms/emdash) | ai | `main@a1dac00` | **1.94** | **2.01** | **2.17** | -0.16 | -0.16 |
| [`cloudflare/vinext`](https://github.com/cloudflare/vinext) | ai | `main@e81a621` | **1.85** | **1.93** | **1.99** | -0.06 | -0.07 |
| [`vitejs/vite`](https://github.com/vitejs/vite) | mature-oss | `main@bc5c6a7` | **1.46** | **1.52** | **1.52** | +0.02 | 0.00 |
| [`modem-dev/hunk`](https://github.com/modem-dev/hunk) | ai | `main@53242b4` | **1.46** | **1.51** | **1.51** | +0.44 | 0.00 |
| [`withastro/astro`](https://github.com/withastro/astro) | mature-oss | `main@7fe40bc` | **1.40** | **1.46** | **1.55** | 0.00 | -0.09 |
| [`pmndrs/zustand`](https://github.com/pmndrs/zustand) | mature-oss | `main@00f96a3` | **1.33** | **1.38** | **1.38** | 0.00 | -0.01 |
| [`payloadcms/payload`](https://github.com/payloadcms/payload) | mature-oss | `main@5afcef5` | **1.29** | **1.34** | **1.34** | +0.02 | 0.00 |
| [`umami-software/umami`](https://github.com/umami-software/umami) | mature-oss | `master@3a31ad3` | **1.00** | **1.04** | **1.04** | +0.00 | 0.00 |
| [`egoist/tsup`](https://github.com/egoist/tsup) | mature-oss | `main@b906f86` | **0.89** | **0.92** | **0.92** | 0.00 | 0.00 |
| [`sindresorhus/execa`](https://github.com/sindresorhus/execa) | mature-oss | `main@f3a2e84` | **0.85** | **0.89** | **0.89** | 0.00 | 0.00 |
| [`mikaelbr/node-notifier`](https://github.com/mikaelbr/node-notifier) | mature-oss | `master@b36c237` | **0.40** | **0.41** | **0.41** | 0.00 | 0.00 |
| [`vercel/hyper`](https://github.com/vercel/hyper) | mature-oss | `canary@2a7bb18` | **0.40** | **0.41** | **0.41** | 0.00 | 0.00 |
| [`garrytan/gstack`](https://github.com/garrytan/gstack) | ai | `main@ed1e4be` | **9.15** | **11.14** | **11.14** | +6.37 | 0.00 |
| [`FullAgent/fulling`](https://github.com/FullAgent/fulling) | ai | `main@d95060f` | **8.42** | **10.24** | **10.24** | +8.08 | 0.00 |
| [`redwoodjs/agent-ci`](https://github.com/redwoodjs/agent-ci) | ai | `main@76b46f9` | **7.83** | **9.53** | **9.53** | +5.61 | 0.00 |
| [`jiayun/DevWorkbench`](https://github.com/jiayun/DevWorkbench) | ai | `main@ea50862` | **7.39** | **8.99** | **8.99** | +5.60 | 0.00 |
| [`robinebers/openusage`](https://github.com/robinebers/openusage) | ai | `main@584d44d` | **5.27** | **6.41** | **6.41** | +3.39 | 0.00 |
| [`openclaw/openclaw`](https://github.com/openclaw/openclaw) | ai | `main@6b6dcaf` | **5.26** | **6.40** | **6.40** | +3.48 | 0.00 |
| [`emdash-cms/emdash`](https://github.com/emdash-cms/emdash) | ai | `main@3dd1a1f` | **4.15** | **5.06** | **5.06** | +3.04 | 0.00 |
| [`cloudflare/vinext`](https://github.com/cloudflare/vinext) | ai | `main@67a929b` | **3.06** | **3.73** | **3.73** | +1.80 | 0.00 |
| [`modem-dev/hunk`](https://github.com/modem-dev/hunk) | ai | `main@a6aa1cb` | **2.95** | **3.59** | **3.59** | +2.08 | 0.00 |
| [`vitejs/vite`](https://github.com/vitejs/vite) | mature-oss | `main@640202a` | **1.71** | **2.08** | **2.08** | +0.56 | 0.00 |
| [`withastro/astro`](https://github.com/withastro/astro) | mature-oss | `main@1058428` | **1.69** | **2.05** | **2.05** | +0.59 | 0.00 |
| [`egoist/tsup`](https://github.com/egoist/tsup) | mature-oss | `main@b906f86` | **1.25** | **1.52** | **1.52** | +0.60 | 0.00 |
| [`pmndrs/zustand`](https://github.com/pmndrs/zustand) | mature-oss | `main@95d3f33` | **1.12** | **1.36** | **1.38** | -0.02 | -0.02 |
| [`payloadcms/payload`](https://github.com/payloadcms/payload) | mature-oss | `main@0ceba02` | **1.00** | **1.22** | **1.34** | -0.13 | -0.13 |
| [`sindresorhus/execa`](https://github.com/sindresorhus/execa) | mature-oss | `main@f3a2e84` | **0.82** | **0.99** | **0.99** | +0.11 | 0.00 |
| [`mikaelbr/node-notifier`](https://github.com/mikaelbr/node-notifier) | mature-oss | `master@b36c237` | **0.78** | **0.95** | **0.95** | +0.53 | 0.00 |
| [`vercel/hyper`](https://github.com/vercel/hyper) | mature-oss | `canary@2a7bb18` | **0.74** | **0.90** | **0.90** | +0.49 | 0.00 |
| [`umami-software/umami`](https://github.com/umami-software/umami) | mature-oss | `master@c78ff36` | **0.70** | **0.85** | **1.04** | -0.19 | -0.19 |

Legend:

Expand All @@ -207,7 +207,7 @@ Legend:
- `Δ prev` = latest pinned - previous week's pinned score
- `Δ peak` = latest pinned - highest pinned score, so more negative means the repo is below its own historical high

For exact pinned SHAs and the full per-metric breakdowns, see the saved snapshot and pinned benchmark report.
For exact pinned SHAs, the current default-pack medians, and the full per-metric breakdowns, see the saved snapshot and pinned benchmark report.

Full benchmark assets:

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/history/known-ai-vs-solid-oss/agent-ci.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
{"schemaVersion":1,"benchmarkSetId":"known-ai-vs-solid-oss","benchmarkSetName":"Known AI repos vs older solid OSS repos","recordedAt":"2026-03-30T12:00:00Z","periodStart":"2026-03-30","repoId":"agent-ci","repo":"redwoodjs/agent-ci","cohort":"explicit-ai","scanMode":"default-branch-as-of-recorded-at","defaultBranch":"main","ref":"4326a2b517cf90155e92bea515a04d1dde0b4e8d","analyzerVersion":"0.3.0","analyzerCommit":"c64455e21d379cfda17f99b37ad7d1160763bef7","summary":{"fileCount":85,"directoryCount":22,"findingCount":25,"repoScore":79.35714285714288,"physicalLineCount":15268,"logicalLineCount":7373,"functionCount":199,"normalized":{"scorePerFile":0.9336134453781515,"scorePerKloc":10.763209393346381,"scorePerFunction":0.39877961234745163,"findingsPerFile":0.29411764705882354,"findingsPerKloc":3.3907500339075,"findingsPerFunction":0.12562814070351758}},"blended":{"vsCurrentCohort":3.304551337754753,"vsPinnedBaseline":3.4336696742651553},"ruleCounts":{"defensive.empty-catch":17,"defensive.error-obscuring":3,"defensive.error-swallowing":3,"structure.directory-fanout-hotspot":2}}
{"schemaVersion":1,"benchmarkSetId":"known-ai-vs-solid-oss","benchmarkSetName":"Known AI repos vs older solid OSS repos","recordedAt":"2026-04-06T12:00:00Z","periodStart":"2026-04-06","repoId":"agent-ci","repo":"redwoodjs/agent-ci","cohort":"explicit-ai","scanMode":"default-branch-as-of-recorded-at","defaultBranch":"main","ref":"be06e223d1f419565c7e5589bab0f789f1779f60","analyzerVersion":"0.3.0","analyzerCommit":"c64455e21d379cfda17f99b37ad7d1160763bef7","summary":{"fileCount":90,"directoryCount":23,"findingCount":26,"repoScore":81.35714285714288,"physicalLineCount":15656,"logicalLineCount":7673,"functionCount":204,"normalized":{"scorePerFile":0.9039682539682542,"scorePerKloc":10.60304220736907,"scorePerFunction":0.3988095238095239,"findingsPerFile":0.28888888888888886,"findingsPerKloc":3.388505147921282,"findingsPerFunction":0.12745098039215685}},"blended":{"vsCurrentCohort":3.277166214819975,"vsPinnedBaseline":3.4043912743913416},"ruleCounts":{"defensive.empty-catch":17,"defensive.error-obscuring":3,"defensive.error-swallowing":3,"structure.directory-fanout-hotspot":2,"structure.barrel-density":1}}
{"schemaVersion":1,"benchmarkSetId":"known-ai-vs-solid-oss","benchmarkSetName":"Known AI repos vs older solid OSS repos","recordedAt":"2026-04-13T12:00:00Z","periodStart":"2026-04-13","repoId":"agent-ci","repo":"redwoodjs/agent-ci","cohort":"explicit-ai","scanMode":"default-branch-as-of-recorded-at","defaultBranch":"main","ref":"c61f27d7734c74a9c0b9efa7bb833ff451c8b61c","analyzerVersion":"0.3.0","analyzerCommit":"c64455e21d379cfda17f99b37ad7d1160763bef7","summary":{"fileCount":114,"directoryCount":28,"findingCount":38,"repoScore":140.10000000000005,"physicalLineCount":22952,"logicalLineCount":11687,"functionCount":278,"normalized":{"scorePerFile":1.228947368421053,"scorePerKloc":11.987678617267054,"scorePerFunction":0.5039568345323743,"findingsPerFile":0.3333333333333333,"findingsPerKloc":3.2514759989732185,"findingsPerFunction":0.1366906474820144}},"blended":{"vsCurrentCohort":3.763892950592061,"vsPinnedBaseline":3.913159449141177},"ruleCounts":{"defensive.empty-catch":22,"tests.duplicate-mock-setup":6,"defensive.error-obscuring":5,"defensive.error-swallowing":3,"structure.barrel-density":1,"structure.pass-through-wrappers":1}}
{"schemaVersion":1,"benchmarkSetId":"known-ai-vs-solid-oss","benchmarkSetName":"Known AI repos vs older solid OSS repos","recordedAt":"2026-04-20T12:22:33.984Z","periodStart":"2026-04-20","repoId":"agent-ci","repo":"redwoodjs/agent-ci","cohort":"explicit-ai","scanMode":"default-branch-as-of-recorded-at","defaultBranch":"main","ref":"37f98a6c372bf909aeb5b8c7b1194708febccdce","analyzerVersion":"0.3.0","analyzerCommit":"df9e2d480c43ab6a1556caff16ab313302926a2a","summary":{"fileCount":137,"directoryCount":33,"findingCount":57,"repoScore":202.50000000000003,"physicalLineCount":28299,"logicalLineCount":13697,"functionCount":399,"normalized":{"scorePerFile":1.478102189781022,"scorePerKloc":14.7842593268599,"scorePerFunction":0.5075187969924813,"findingsPerFile":0.41605839416058393,"findingsPerKloc":4.161495217930934,"findingsPerFunction":0.14285714285714285}},"blended":{"vsCurrentCohort":4.321646110996529,"vsPinnedBaseline":4.556987923331088},"ruleCounts":{"defensive.empty-catch":23,"defensive.error-obscuring":9,"defensive.promise-default-fallbacks":8,"defensive.error-swallowing":6,"tests.duplicate-mock-setup":6,"structure.pass-through-wrappers":2,"api.generic-status-envelopes":1,"structure.barrel-density":1,"structure.directory-fanout-hotspot":1}}
{"schemaVersion":1,"benchmarkSetId":"known-ai-vs-solid-oss","benchmarkSetName":"Known AI repos vs older solid OSS repos","recordedAt":"2026-04-26T18:00:48.765Z","periodStart":"2026-04-20","repoId":"agent-ci","repo":"redwoodjs/agent-ci","cohort":"explicit-ai","scanMode":"default-branch-as-of-recorded-at","defaultBranch":"main","ref":"76b46f9929509a90355f34c0550dc20d29924aec","analyzerVersion":"0.3.0","analyzerCommit":"326869c100e1e999f1676c47f6a9e8a2327cca97","summary":{"fileCount":156,"directoryCount":42,"findingCount":60,"repoScore":207.50000000000003,"physicalLineCount":30109,"logicalLineCount":14645,"functionCount":483,"normalized":{"scorePerFile":1.3301282051282053,"scorePerKloc":14.16865824513486,"scorePerFunction":0.4296066252587992,"findingsPerFile":0.38461538461538464,"findingsPerKloc":4.09696142027996,"findingsPerFunction":0.12422360248447205}},"blended":{"vsCurrentCohort":7.828395917596652,"vsPinnedBaseline":9.52677143526095},"ruleCounts":{"defensive.empty-catch":25,"defensive.error-obscuring":11,"defensive.promise-default-fallbacks":9,"defensive.error-swallowing":6,"tests.duplicate-mock-setup":6,"structure.pass-through-wrappers":2,"api.generic-status-envelopes":1}}
Loading
Loading