Skip to content

fix(ci): make the OpenAPI lint capable of failing, with a vendored ruleset - #219

Merged
mdheller merged 1 commit into
mainfrom
fix/spectral-lint-real
Jul 30, 2026
Merged

fix(ci): make the OpenAPI lint capable of failing, with a vendored ruleset#219
mdheller merged 1 commit into
mainfrom
fix/spectral-lint-real

Conversation

@mdheller

Copy link
Copy Markdown
Contributor

validate.yml:149 has never linted anything:

spectral lint openapi.yaml --ruleset https://unpkg.com/@stoplight/spectral-openapi/dist/ruleset.js || true

Three stacked failures, each verified rather than assumed:

  1. The URL is a 404. curl -o /dev/null -w '%{http_code}' -L https://unpkg.com/@stoplight/spectral-openapi/dist/ruleset.js -> 404. The package it names has never existed: https://registry.npmjs.org/@stoplight/spectral-openapi -> 404 as well. The real package is @stoplight/spectral-rulesets.
  2. So Spectral exits 2 on every run, with Could not load ...: Not Found.
  3. || true turns that into 0. Measured: bare command exit 2, with || true exit 0.

The python step at :151 only asserts the document parses and has openapi/info/paths. So openapi.yaml — the estate's schema source of truth — has been checked for parsing and never for conformance.

The fix

  • || true is gone; --fail-severity error is explicit.
  • The ruleset is the new local .spectral.yaml. No URL is fetched at lint time. spectral:oas is built into the CLI, resolved from the @stoplight/spectral-rulesets copy inside the CLI's own dependency tree.
  • That tree is pinned exactly and integrity-verified by tools/spectral/package-lock.json — 248 packages, every one carrying a sha512- integrity hash, installed with npm ci. @stoplight/spectral-cli 6.15.0, @stoplight/spectral-rulesets 1.22.6 (sha512-xBwrb2zjx+7AzGS3aX7aOtdd…, matching the registry). This also replaces the floating npm install -g @stoplight/spectral-cli.
  • node_modules/ added to .gitignore (the repo had no Node tooling before).

Deviation from the letter of "tarball + file: ref", stated plainly: the ruleset is not committed as a .tgz. Vendoring a second copy of @stoplight/spectral-rulesets alongside the one already inside the CLI would create version skew between the rules and the engine running them, and the rules are function-based JS that cannot be hand-vendored as data. The lockfile gives the same two properties the rule exists for — exact pinning and sha512 verification — with no CDN in the loop.

One rule disabled, with a reason

duplicated-entry-in-enum aborts the entire run in CLI 6.15.0 with Cannot read properties of null (reading 'enum'), thrown from nimma's traversal, whenever openapi.yaml's external ./schemas/*.json refs resolve. Isolated to that rule alone: disabling it lets the lint complete, and the crash still happens with every referenced schema replaced by {"type":"object"} — so the schema contents are not the trigger. It is an upstream bug. Leaving it on would mean the lint could only ever crash, which is how a || true gets added in the first place.

openapi.yaml itself is clean under the rest of the ruleset: 0 errors, 1 warning (info-contact).

Red-then-green

Exit codes, same tree, three independent conformance breakages:

state NEW lint OLD step py parse (:151)
clean 0 0 0
broken $ref -> schemas/DoesNotExist.json 1 0 0
duplicate operationId 1 0 0
invalid response key notAStatusCode 1 0 0
restored 0 0 0

openapi.yaml is byte-identical to origin/main afterwards (git diff --stat empty). The middle column is the paper tiger; the right column is why the parse check never covered it.

Not fixed here: the check still is not required

gh api repos/SourceOS-Linux/sourceos-spec/rulesets returns exactly one ruleset, rules (id 16110287), enforcement: disabled, containing only deletion and non_fast_forwardno required status checks, and branches/main/protection returns nothing. So Validate OpenAPI can go red and still not block a merge. That is a repo-settings change, not a code change, so it is flagged rather than made unilaterally.

…leset

`spectral lint openapi.yaml --ruleset https://unpkg.com/@stoplight/spectral-openapi/dist/ruleset.js || true`
never linted anything. Three stacked failures:

1. The URL returns HTTP 404. The package it names, @stoplight/spectral-openapi,
   does not exist on the npm registry at all (the real one is
   @stoplight/spectral-rulesets), so the ruleset was never loadable.
2. Spectral therefore exits 2 with "Could not load ...: Not Found" on every run.
3. `|| true` converts that to 0, so the step has always reported success.

The adjacent python check at :151 only asserts the document parses and has
openapi/info/paths keys, so openapi.yaml has been checked for parsing and never
for conformance.

The ruleset is now the local .spectral.yaml, and no URL is fetched at lint time.
`spectral:oas` is built into the CLI, resolved from the @stoplight/spectral-rulesets
copy in the CLI's own dependency tree, and that tree is pinned exactly and
sha512-verified by tools/spectral/package-lock.json (248 packages, every one with
an integrity hash) installed via `npm ci`. This also replaces the floating
`npm install -g @stoplight/spectral-cli`.

One rule is disabled with a recorded reason: duplicated-entry-in-enum aborts the
entire run in CLI 6.15.0 with "Cannot read properties of null (reading 'enum')"
whenever openapi.yaml's external ./schemas/*.json refs resolve. Isolated to that
rule alone -- the crash persists with every referenced schema replaced by
{"type":"object"}, so the schema contents are not the trigger.

openapi.yaml itself is clean: 0 errors, 1 warning (info-contact).

Proof that the guard now has teeth, exit codes:

  state                                     NEW lint  OLD step  py parse (:151)
  clean                                        0         0          0
  broken $ref -> schemas/DoesNotExist.json     1         0          0
  duplicate operationId                        1         0          0
  invalid response key notAStatusCode          1         0          0
  restored                                     0         0          0
Copilot AI review requested due to automatic review settings July 30, 2026 03:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The workflow should be adjusted to avoid Node engine mismatch risk and to reduce supply-chain exposure from dependency install scripts during CI.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR fixes the OpenAPI lint in CI so it can actually fail on conformance errors, by replacing a broken remote Spectral ruleset URL (and the || true swallow) with a local .spectral.yaml ruleset and a pinned, lockfile-driven Spectral CLI install under tools/spectral/.

Changes:

  • Add a local .spectral.yaml extending spectral:oas, disabling the one crashing rule (duplicated-entry-in-enum).
  • Vendor a pinned Spectral CLI install (via tools/spectral/package.json + package-lock.json) and switch CI to npm ci + local .bin/spectral.
  • Ignore node_modules/ now that CI installs Node dependencies.
File summaries
File Description
tools/spectral/package.json Introduces a dedicated, non-root Node package to pin Spectral CLI.
tools/spectral/package-lock.json Locks and integrity-pins the Spectral dependency tree used in CI.
.spectral.yaml Defines the local Spectral ruleset (extends spectral:oas) and disables the crashing rule.
.gitignore Ignores node_modules/ for the new Spectral install.
.github/workflows/validate.yml Makes the OpenAPI lint step real: pinned install + fail on error severity.
Review details
  • Files reviewed: 3/5 changed files
  • Comments generated: 2
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@@ -142,11 +142,19 @@ jobs:
with:
node-version: "20"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evaluated — accepted as a real but low-severity risk, not blocking.

The lockfile does declare node: ^16.20 || ^18.18 || >= 20.17 for the pinned CLI. setup-node with node-version: "20" resolves to the newest cached 20.x, which on GitHub-hosted runners has been >= 20.17 since 2024-09, and npm ci does not enforce engines unless engine-strict=true. So the failure mode is "possible on an unusual toolcache state", not "possible today" — and it would fail loudly at install, not silently mislint.

Recorded as a follow-up (pin node-version: "20.17" or add .nvmrc) rather than a change to this PR, because the PR is already establishing the pin discipline and a version bump here would not be covered by the CI evidence attached to this head SHA.

# than `npm install -g @stoplight/spectral-cli`, which floats to whatever
# the registry serves that day.
- name: Install Spectral (pinned, integrity-verified)
run: npm ci --prefix tools/spectral

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evaluated — this is the sharpest finding on the PR, and it is correct.

npm ci runs transitive install scripts by default, and @scarf/scarf in this tree is install-time analytics that phones home. That sits badly next to the estate rule this PR is otherwise serving (vendor it, do not reach out at build time): the PR removes a CDN fetch at lint time and leaves a network side effect at install time.

It is not a regression — the status quo it replaces (npm install -g @stoplight/spectral-cli, unpinned) ran the same scripts with no integrity pinning at all — so this PR is strictly an improvement and I am not blocking on it. But --ignore-scripts is the right end state: Spectral needs no native build, so nothing legitimate depends on those scripts.

Filed as an immediate follow-up: add --ignore-scripts to the npm ci step and re-run the OpenAPI lint to confirm the CLI still resolves.

@mdheller
mdheller merged commit 3efe6b2 into main Jul 30, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants