fix(ci): make the OpenAPI lint capable of failing, with a vendored ruleset - #219
Conversation
…leset `spectral lint openapi.yaml --ruleset https://unpkg.com/@stoplight/spectral-openapi/dist/ruleset.js || true` never linted anything. Three stacked failures: 1. The URL returns HTTP 404. The package it names, @stoplight/spectral-openapi, does not exist on the npm registry at all (the real one is @stoplight/spectral-rulesets), so the ruleset was never loadable. 2. Spectral therefore exits 2 with "Could not load ...: Not Found" on every run. 3. `|| true` converts that to 0, so the step has always reported success. The adjacent python check at :151 only asserts the document parses and has openapi/info/paths keys, so openapi.yaml has been checked for parsing and never for conformance. The ruleset is now the local .spectral.yaml, and no URL is fetched at lint time. `spectral:oas` is built into the CLI, resolved from the @stoplight/spectral-rulesets copy in the CLI's own dependency tree, and that tree is pinned exactly and sha512-verified by tools/spectral/package-lock.json (248 packages, every one with an integrity hash) installed via `npm ci`. This also replaces the floating `npm install -g @stoplight/spectral-cli`. One rule is disabled with a recorded reason: duplicated-entry-in-enum aborts the entire run in CLI 6.15.0 with "Cannot read properties of null (reading 'enum')" whenever openapi.yaml's external ./schemas/*.json refs resolve. Isolated to that rule alone -- the crash persists with every referenced schema replaced by {"type":"object"}, so the schema contents are not the trigger. openapi.yaml itself is clean: 0 errors, 1 warning (info-contact). Proof that the guard now has teeth, exit codes: state NEW lint OLD step py parse (:151) clean 0 0 0 broken $ref -> schemas/DoesNotExist.json 1 0 0 duplicate operationId 1 0 0 invalid response key notAStatusCode 1 0 0 restored 0 0 0
There was a problem hiding this comment.
🟡 Not ready to approve
The workflow should be adjusted to avoid Node engine mismatch risk and to reduce supply-chain exposure from dependency install scripts during CI.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR fixes the OpenAPI lint in CI so it can actually fail on conformance errors, by replacing a broken remote Spectral ruleset URL (and the || true swallow) with a local .spectral.yaml ruleset and a pinned, lockfile-driven Spectral CLI install under tools/spectral/.
Changes:
- Add a local
.spectral.yamlextendingspectral:oas, disabling the one crashing rule (duplicated-entry-in-enum). - Vendor a pinned Spectral CLI install (via
tools/spectral/package.json+package-lock.json) and switch CI tonpm ci+ local.bin/spectral. - Ignore
node_modules/now that CI installs Node dependencies.
File summaries
| File | Description |
|---|---|
tools/spectral/package.json |
Introduces a dedicated, non-root Node package to pin Spectral CLI. |
tools/spectral/package-lock.json |
Locks and integrity-pins the Spectral dependency tree used in CI. |
.spectral.yaml |
Defines the local Spectral ruleset (extends spectral:oas) and disables the crashing rule. |
.gitignore |
Ignores node_modules/ for the new Spectral install. |
.github/workflows/validate.yml |
Makes the OpenAPI lint step real: pinned install + fail on error severity. |
Review details
- Files reviewed: 3/5 changed files
- Comments generated: 2
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| @@ -142,11 +142,19 @@ jobs: | |||
| with: | |||
| node-version: "20" | |||
There was a problem hiding this comment.
Evaluated — accepted as a real but low-severity risk, not blocking.
The lockfile does declare node: ^16.20 || ^18.18 || >= 20.17 for the pinned CLI. setup-node with node-version: "20" resolves to the newest cached 20.x, which on GitHub-hosted runners has been >= 20.17 since 2024-09, and npm ci does not enforce engines unless engine-strict=true. So the failure mode is "possible on an unusual toolcache state", not "possible today" — and it would fail loudly at install, not silently mislint.
Recorded as a follow-up (pin node-version: "20.17" or add .nvmrc) rather than a change to this PR, because the PR is already establishing the pin discipline and a version bump here would not be covered by the CI evidence attached to this head SHA.
| # than `npm install -g @stoplight/spectral-cli`, which floats to whatever | ||
| # the registry serves that day. | ||
| - name: Install Spectral (pinned, integrity-verified) | ||
| run: npm ci --prefix tools/spectral |
There was a problem hiding this comment.
Evaluated — this is the sharpest finding on the PR, and it is correct.
npm ci runs transitive install scripts by default, and @scarf/scarf in this tree is install-time analytics that phones home. That sits badly next to the estate rule this PR is otherwise serving (vendor it, do not reach out at build time): the PR removes a CDN fetch at lint time and leaves a network side effect at install time.
It is not a regression — the status quo it replaces (npm install -g @stoplight/spectral-cli, unpinned) ran the same scripts with no integrity pinning at all — so this PR is strictly an improvement and I am not blocking on it. But --ignore-scripts is the right end state: Spectral needs no native build, so nothing legitimate depends on those scripts.
Filed as an immediate follow-up: add --ignore-scripts to the npm ci step and re-run the OpenAPI lint to confirm the CLI still resolves.
validate.yml:149has never linted anything:Three stacked failures, each verified rather than assumed:
curl -o /dev/null -w '%{http_code}' -L https://unpkg.com/@stoplight/spectral-openapi/dist/ruleset.js->404. The package it names has never existed:https://registry.npmjs.org/@stoplight/spectral-openapi->404as well. The real package is@stoplight/spectral-rulesets.Could not load ...: Not Found.|| trueturns that into 0. Measured: bare command exit2, with|| trueexit0.The python step at
:151only asserts the document parses and hasopenapi/info/paths. Soopenapi.yaml— the estate's schema source of truth — has been checked for parsing and never for conformance.The fix
|| trueis gone;--fail-severity erroris explicit..spectral.yaml. No URL is fetched at lint time.spectral:oasis built into the CLI, resolved from the@stoplight/spectral-rulesetscopy inside the CLI's own dependency tree.tools/spectral/package-lock.json— 248 packages, every one carrying asha512-integrity hash, installed withnpm ci.@stoplight/spectral-cli6.15.0,@stoplight/spectral-rulesets1.22.6 (sha512-xBwrb2zjx+7AzGS3aX7aOtdd…, matching the registry). This also replaces the floatingnpm install -g @stoplight/spectral-cli.node_modules/added to.gitignore(the repo had no Node tooling before).Deviation from the letter of "tarball +
file:ref", stated plainly: the ruleset is not committed as a.tgz. Vendoring a second copy of@stoplight/spectral-rulesetsalongside the one already inside the CLI would create version skew between the rules and the engine running them, and the rules are function-based JS that cannot be hand-vendored as data. The lockfile gives the same two properties the rule exists for — exact pinning and sha512 verification — with no CDN in the loop.One rule disabled, with a reason
duplicated-entry-in-enumaborts the entire run in CLI 6.15.0 withCannot read properties of null (reading 'enum'), thrown from nimma's traversal, wheneveropenapi.yaml's external./schemas/*.jsonrefs resolve. Isolated to that rule alone: disabling it lets the lint complete, and the crash still happens with every referenced schema replaced by{"type":"object"}— so the schema contents are not the trigger. It is an upstream bug. Leaving it on would mean the lint could only ever crash, which is how a|| truegets added in the first place.openapi.yamlitself is clean under the rest of the ruleset: 0 errors, 1 warning (info-contact).Red-then-green
Exit codes, same tree, three independent conformance breakages:
:151)$ref->schemas/DoesNotExist.jsonoperationIdnotAStatusCodeopenapi.yamlis byte-identical toorigin/mainafterwards (git diff --statempty). The middle column is the paper tiger; the right column is why the parse check never covered it.Not fixed here: the check still is not required
gh api repos/SourceOS-Linux/sourceos-spec/rulesetsreturns exactly one ruleset,rules(id 16110287),enforcement: disabled, containing onlydeletionandnon_fast_forward— no required status checks, andbranches/main/protectionreturns nothing. SoValidate OpenAPIcan go red and still not block a merge. That is a repo-settings change, not a code change, so it is flagged rather than made unilaterally.