🔒 feat: test what the presets decide, not just how they are written - #15
Merged
Conversation
The validator checks shape. Every fault this preset has shipped passed it:
- `^jest` unanchored matched every jest-* package on npm, a prefix nobody
owns, and those automerged with the trust meant for Jest
- per-tool rules sat after the anchored trust list and re-granted automerge
through looser patterns
- the GitHub Actions rule had no matchUpdateTypes, so it overrode
"major: automerge false" and merged bare digest moves
- matchManagers named pnpm and yarn, which are not managers, so a rule that
read as covering three ecosystems covered one
tooling/test_policy.py freezes those as 22 cases over the automerge decision,
fails if lockFileMaintenance gets automerge back, and rejects manager names
Renovate does not have. Each guard was verified by reintroducing the fault it
covers and watching it fail.
It reimplements Renovate's matching rather than calling Renovate, so it can
drift. That limitation is written at the top of the file and in SECURITY.md
rather than left for someone to discover.
SECURITY.md was missing on a repo about supply-chain security. It says where to
report, what this preset does not do -- it cannot help a repo that already
installed a hostile version -- and the limitations a reader would otherwise
find the hard way.
MIT badge is green; yellow read as a warning.
Biome was configured and run by nothing. Now part of validate.sh, pinned to the version biome.json declares -- biome rejects a schema from a different release, so the two move together or not at all. The custom manager in renovate.json was widened to keep both pinned tools updated, not just renovate. Formatting applied. Verified it changed no meaning: every preset parses to the same object as before. The adoption section contradicted itself -- "copy this repo" followed by "do not fork it". Those were two different situations written as one. Split into three, with the cost of each stated: - extend: fixes reach you - extend and override: same, for a second brand, instead of forking - take a copy: you own it, and it stops receiving fixes That last cost is not hypothetical. Four faults were found here that no amount of reading the documentation would have surfaced, and a copy taken before each one kept it. Anyone choosing that path should choose it knowing. The template flag is back on. Turning it off during the transfer cleanup was my error: I read it as contradicting "inherit, do not fork", but that advice is about a second brand inside one org, not about someone else adopting this as a starting point. Those are different, and the README now says so.
MIT already disclaims warranty and liability, but nobody reads LICENSE. A preset that decides what merges unattended should say where it stops, in the file people actually open. Five limits, all of them true and none obvious: it gates updates rather than what is already installed, it is not a scanner, a release-age delay is not a guarantee, the watch list describes one attack in one month, and the policy test reimplements Renovate's matching rather than calling it. Documents the Setup Owner workflow for the copy path, including that it runs with contents: write and pushes to the default branch. Anyone extending the preset instead of copying it will never run it, and should know that without having to read the YAML.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap this closes
renovate-config-validatorchecks that the rules are well formed. It does not check what they decide. Every fault this preset has shipped passed it:^jestunanchored matched everyjest-*package on npm — a prefix nobody owns — and those automerged with the trust meant for JestmatchUpdateTypes, so it overrode "major: automerge false" and merged bare digest movesmatchManagersnamedpnpmandyarn, which are not managerstooling/test_policy.pyfreezes those as 22 cases over the automerge decision, fails iflockFileMaintenancegets automerge back, and rejects manager names Renovate does not have.Each guard was verified by reintroducing the fault and watching it fail — a test that passes without being able to catch the bug is worse than none:
/^jest$/→/^jest/jest-environment-foo: expected automerge=False, got TruelockFileMaintenance.automerge: truematchManagers: [npm, pnpm, yarn]The limitation, stated up front
It reimplements Renovate's matching rather than calling Renovate, so it can drift from the real engine. A failure is a reason to look; a pass is weaker evidence than a dry run. That is written at the top of the file and in SECURITY.md, not left to be discovered.
SECURITY.md
Missing on a repo about supply-chain security. Where to report, what this preset does not do — it cannot help a repo that already installed a hostile version — and the known limitations.
Also
MIT badge green. Yellow read as a warning.