Merge validation-framework branch into main#260
Merged
Conversation
…nfig Validation framework v1 directory layout alongside existing release-automation assets (shared schemas, validate-release-plan.py interface frozen). New schemas: common findings model, rule metadata definitions, central validation config. Module placeholders for engines, bundling, context, post-filter, and output pipeline. Initial validation-config.yaml with ReleaseTest as first advisory-stage repo.
Replaces flat numeric IDs with engine-prefixed format for readability
and unlimited per-engine growth. Pattern: {prefix}-{NNN}.
Implement config_gate module that reads validation-config.yaml, validates it against its JSON Schema (2020-12), and resolves the effective rollout stage for a given repository. Handles fork owner override logic and advisory/disabled gate decisions. Includes 24 unit tests covering schema validation, stage resolution, fork detection, and gate logic for all stage/trigger combinations.
Implement unified validation context assembly from workflow inputs, release-plan.yaml, and OpenAPI spec files. Includes branch type and trigger type derivation, profile auto-selection, release plan parsing, and API pattern detection (request-response, implicit/explicit subscription). Three focused modules: context_builder (dataclasses + pure derivation), release_plan_parser (Draft 7 schema validation + field extraction), api_pattern_detector (3-tier heuristic from spec paths/callbacks). Includes 47 unit tests covering all derivation paths, graceful degradation for missing files, and dataclass serialization.
Invoke Spectral CLI, parse JSON output, and normalize findings into the common findings model. Includes version-specific ruleset selection with fallback, severity mapping (0→error, 1→warn, 2/3→hint), and graceful error handling for missing CLI, timeouts, and runtime errors. 40 unit tests covering pure functions and mocked subprocess invocation.
yamllint adapter parses --format parsable output, maps warning->warn, and preserves 1-indexed line/column numbers. Errors block downstream Spectral validation (orchestrator decision, not adapter). gherkin-lint adapter parses --format json output, relativizes absolute filePaths, and defaults all findings to warn level (post-filter can override via rule metadata). 51 unit tests covering parsing, severity mapping, and mocked subprocess.
Add the fourth engine adapter for native Python validation checks. Unlike external engine adapters (Spectral, yamllint, gherkin-lint), Python checks run in-process with direct access to ValidationContext. Harness architecture: - python_adapter.py: entry point with per-check error isolation - python_checks/ package: explicit registry, shared types, domain modules - CheckScope (REPO/API): harness iterates per-API checks automatically - make_finding() helper ensures structural consistency Core checks implemented (12 checks across 7 modules): - version_checks: info.version format, server URL version/api-name - filename_checks: kebab-case naming, file existence - test_checks: test directory, test files, version alignment - release_plan_checks: track/type consistency, file existence (ported) - changelog_checks: CHANGELOG existence and format - metadata_checks: license and x-camara-commonalities consistency - release_review_checks: file restriction on snapshot branches Version segment builder implements full CAMARA URL mapping: wip->vwip, 1.0.0->v1, 0.1.0->v0.1, 0.2.0-alpha.2->v0.2alpha2 131 new tests, 293 total passing.
…e blocking Implement the post-filter pipeline that sits between engine adapters and the output stage. Processes raw findings through rule metadata lookup, applicability evaluation, conditional severity resolution, and profile-based blocking to produce pass/fail/error verdicts. Modules: - metadata_loader: YAML loading, frozen dataclasses, (engine, engine_rule) index - condition_evaluator: AND/OR condition logic, version range comparison - level_resolver: first-match-wins overrides, advisory/standard/strict profiles - engine: orchestration entry point (run_post_filter -> PostFilterResult) Handles empty rules directory gracefully (all findings pass through). 114 new tests, 407 total passing.
Simplify rule metadata schema: only id, engine, engine_rule required. Name, hint, and conditional_level are optional — identity-only entries need just 3 fields to assign a stable rule_id. Rule metadata files (96 rules total): - python-rules.yaml: 12 checks with conditional levels and applicability - spectral-rules.yaml: 17 CAMARA custom + 29 built-in OAS (S-001–S-017, S-200–S-228, gap reserved for future CAMARA rules) - gherkin-rules.yaml: 25 enabled rules - yamllint-rules.yaml: 13 enabled rules P-007 (check-test-file-version) suppressed pending fix: checks filename version suffix but should check version inside .feature file content. Integration test validates structural integrity, engine coverage (every enabled engine rule has metadata), and metadata quality. Coverage tracker documents 21 gap rules and 1 fix needed. 428 tests passing.
…018) Separate two use cases previously conflated in a single hint field: - message_override: replaces engine message entirely (rare) - hint: additional fix guidance alongside engine message (common) Both fields are optional; when neither is set the engine message is preserved and no hint is added. Zero rules currently use either field, so this is a non-breaking schema evolution.
Implements the output pipeline that formats post-filter findings for: - Workflow summary ($GITHUB_STEP_SUMMARY) with 900KB truncation - Check annotations (::error/::warning/::notice workflow commands, 50 limit) - PR comment (concise, marker-based create-or-update) - Commit status payload (CAMARA Validation context) - Diagnostic artifacts (full findings JSON, context, summary, engine reports) All modules are pure Python generators — no GitHub API calls. The workflow step (caller workflow, future WP) handles posting via actions/github-script. 6 modules + 6 test files, 94 new tests (526 total, all passing).
…rator Pipeline orchestrator (validation/orchestrator.py) chains all existing modules: config gate -> context builder -> engines -> post-filter -> output. Writes output files for the workflow to post to GitHub surfaces. Reusable workflow (.github/workflows/validation.yml) implements: - 3-tier ref resolution (override -> OIDC -> fallback tag v1-rc) - Single-job sequential pipeline with 16 steps - Annotations, workflow summary, PR comment, commit status, diagnostics - Graceful degradation for fork PRs (continue-on-error) Also includes: - Caller template (validation/workflows/validation-caller.yml) - npm dependencies (Spectral, gherkin-lint) with lock file - 29 unit tests for orchestrator (555 total, zero regressions)
YAML parses unquoted "off" as boolean False, causing the post-filter to miss the suppression check (resolved_level == "off" fails when the actual value is False). Renamed to "muted" which is YAML-safe and familiar from linting tools. Found during smoke test: P-007 appeared in annotations with level: false instead of being filtered out.
OIDC tokens are unavailable for fork PRs, so the fallback ref is used. During testing the fallback must point to hdamker/tooling@validation-framework instead of the production target camaraproject/tooling@v1-rc. TESTING ONLY — revert to camaraproject/tooling@v1-rc before production.
Spectral v6.15.0 appends "No results with a severity of 'error' found!" to stdout after the JSON array, causing json.loads() to fail with "Extra data: line 1 column 3". The --quiet flag suppresses this. Also adds a defensive test for the trailing-diagnostic pattern.
…e gherkin-lint Bump GitHub Actions to match sibling workflows: setup-python@v6, setup-node@v6, upload-artifact@v6. Bump node-version from 20 to 24 (GitHub deprecating Node 20 from June 2026). Pin pip dependencies (pyyaml==6.0.3, jsonschema==4.26.0) for reproducible builds. Exclude unmaintained gherkin-lint from v1 validation pipeline. The engine adapter is retained for re-integration with a replacement tool. Orchestrator now unconditionally skips the gherkin engine.
Revert the gherkin-lint exclusion from the previous commit. Testing shows gherkin-lint works with Node 24 despite being unmaintained. The deprecation warnings are cosmetic npm-level notices, not runtime failures. Better to have gherkin linting with warnings than none.
… creation Spectral may emit absolute runner paths in its JSON output depending on how the shell resolves globs. Previously only the annotation module handled this; diagnostics artifacts retained absolute paths. Now _normalize_path() strips the repo root prefix at finding creation time so all downstream consumers get clean repo-relative paths. Also re-exports PROFILE_ADVISORY/STANDARD/STRICT from validation.context package and updates level_resolver.py to use the public import.
…label Three bugs found during smoke testing on ReleaseTest: 1. yamllint adapter passed glob patterns as literal strings to subprocess.run() — yamllint doesn't expand globs internally, resulting in 0 findings on all files. 2. gherkin-lint's feature-finder mangles ** glob patterns (appends /**.feature), matching nothing. Both adapters now expand globs via Path.glob() before invoking the subprocess. 3. Advisory profile with findings showed "PASS" in summary/PR comment/commit status, which is misleading when errors exist. compute_overall_result() now returns "advisory" when advisory profile has findings, rendered as "ADVISORY" in all surfaces.
Root causes for yamllint/gherkin producing 0 findings on CI: 1. yamllint was never installed — pip only installed pyyaml and jsonschema. python3 -m yamllint exited 1 (module not found), which the adapter treated as "findings found, empty stdout". Fix: add yamllint==1.38.0 to pip install. 2. gherkin-lint was installed in .tooling/validation/node_modules but npx couldn't find it because cwd was the repo root. The workflow already adds node_modules/.bin to PATH, so invoke gherkin-lint directly instead of via npx.
gherkin-lint writes its --format json output to stderr, not stdout. The adapter was parsing result.stdout which was always empty. Now reads from stderr first, falling back to stdout.
Gherkin test file findings are attributed by filename stem which includes the operationId suffix. The column label "API" was misleading for test file entries. Renamed to "API / Test" to reflect that both API specs and test files appear as row keys.
Mint validation app token via create-github-app-token, probe GITHUB_TOKEN write access as fallback, degrade to read-only when neither is available. Token resolution, PR comment, and commit status consolidated into a single step, gated on PR events only. Annotations also restricted to PR events to prevent dispatch duplicates.
Spectral resolves external $ref natively — no pre-bundling needed for validation. Bundling is now an output/artifact step in the workflow that produces standalone specs via Redocly CLI for reviewer download and release automation handoff. Changes: - Spectral adapter: downgrade findings from external files (e.g. code/common/CAMARA_common.yaml) to hint level - Workflow: add Redocly bundle step + artifact upload after validation - Add @redocly/cli to package.json dependencies - Update orchestrator bundling status to reflect workflow-step design
The Redocly CLI binary is in .tooling/validation/node_modules/.bin/ which was only on PATH for the orchestrator step. The bundling step needs it too.
…fallback Part 1 of release automation handoff (WP-06.13): - Remove release-review PR skip logic for yamllint/Spectral (DEC-011 revision: all engines run on release-review PRs for defense-in-depth against bundling/transformation errors) - Add release-metadata.yaml parser with field mapping to ReleasePlanData (enriched tag extraction, api_status derivation from version) - Context builder falls back to release-metadata.yaml when release-plan is absent on snapshot branches (ensures correct Spectral ruleset and per-API check coverage) - Create shared-actions/run-validation composite action for reuse by both validation.yml and RA workflow (pre-snapshot gate in Part 2) - Update token resolution comment per DEC-022 architecture 596 tests passing (24 new).
Part 2 of release automation handoff: - Add pre-snapshot validation steps to create-snapshot job: checkout API repo, setup runtimes, run shared validation action with mode pre-snapshot, gate snapshot creation on validation result - Add bundling to snapshot creator: detect external $ref in API specs, run redocly bundle before mechanical transformations, remove inlined common/modules directories - Add Node 24 + Redocly CLI setup to create-snapshot composite action - Clean up internal decision identifiers from workflow comments 1161 tests passing (596 validation + 565 release automation).
Part 3 of release automation handoff: - Replace inline dep installation + orchestrator invocation with shared-actions/run-validation composite action call - Remove duplicate summary writing step (handled by action) - Simplify result check to use action output (should_fail) - Update all step references from orchestrator to validation Workflow is thinner: setup → action → PR-specific output surfaces. Same action is used by release automation for pre-snapshot gate. 596 tests passing (no behavioral change).
…on-pilot docs: add pilot user documentation for CAMARA Validation
A stray issue_comment on a different issue/PR (e.g. validation-bot posting its result on the Release Review PR) creates a same-named CAMARA Release Automation run that immediately skips via the workflow's if: filter; without a title check find_recent_caller_run could grab the skipped run instead of the slash-command run. Filter run-list candidates by displayTitle matching the Release Issue's title so only runs triggered by comments on the target issue qualify.
…guation fix(regression-runner): disambiguate caller run by displayTitle
Refresh the validation-framework copy of config/validation-settings.yaml with the 14 stage-enabled entries added on main via tooling#245, so the fallback configuration on this branch matches the live configuration on main. Also pulls in the npm/validation dependabot entry. Removes the main-side regression-runner.yml stub during the merge — VF's regression canaries live under different file names, so the stub is misleading here.
chore: merge main into validation-framework
P-007 / P-024 read only line 1 and fired P-024 as error when the Feature line was preceded by a comment or feature-level tag (valid Gherkin in 9 of 49 upstream API repos). Pure validation false positive — release-time transformation is unaffected by Feature-line position. Scan the first 50 lines for `Feature:`, skip preceding comment, tag, and blank lines, and report findings at the actual Feature-line number. Split P-024 into two sub-cases with distinct messages: no `Feature:` line in the scan window, vs. Feature line present without a version token. Drop the implementation-coupling "T1b" phrasing from user-facing messages; revise the P-024 rule comment. Fixes #248
validation: scan early lines for Feature: in .feature test files
- Section 2: collapse two version-related checkboxes into a single "API version(s) used in all files match release-plan.yaml" line. In the alpha branch, drop the "definitions" -> "definition" plural on the info.description checkbox. - Section 3: replace the standalone "Temporary checks during automation introduction" heading + bulleted list with a short italicized reminder paragraph at the end of the Release Management Actions section. The reminder is intentionally not assigned to a single role — it is a transitional check, not an official checkpoint. - Rename "Valid actions" to "Valid next actions for codeowners". Tests: update test_template_loader.py assertions to match the new template strings; all 13 cases pass.
…m491-followup Release Review PR template: apply RM#491 review follow-up
chore: merge main into validation-framework (pre-E2E sync)
fix(ra): skip workflow on repositories with Issues disabled
…abel - Replace per-finding markdown tables in the workflow summary with rule-grouped bullets (bold subject line + bullets + suggestion blockquote) that paste cleanly into GitHub issues. - Add findings.tsv to the validation-diagnostics artifact alongside findings.json for spreadsheet review. - Rename the rendered "Hint:" label to "Suggestion:" across the workflow summary, annotations, and Checks API surfaces. The rule-metadata field is still named hint internally for backward compatibility.
Completes the disambiguation started in the previous commit (rendered label rename Hint: -> Suggestion:) by also renaming the underlying rule-metadata field. The schema field, RuleMetadata dataclass attribute, post-filter propagation, all rule-YAML keys (16 occurrences across spectral-rules.yaml and python-rules.yaml), and every output surface that reads the field now use suggestion. The TSV column header in validation-diagnostics changes from hint to suggestion. Severity-level uses of the literal string "hint" are unchanged — those refer to the severity (error/warn/hint), not the metadata field. The backward-compatibility note that explained the severity-vs-field collision in problem-messages.md is removed: with the field renamed, the source of confusion is gone.
validation: workflow-summary bullets and hint→suggestion rename
Engine messages can contain regex quantifiers, RFC 6901 path encoding, or bracketed identifiers that GitHub-flavored Markdown parses as inline emphasis or links — visible glitches today on S-008 (`(-[a-z0-9]+)*` flips trailing prose to italic) and S-003 (`~1...~1` strikethrough). Escape `*`, `_`, `~`, backtick, `[`, `]`, `<`, and `\` in the bullet body via a shared `escape_gfm_inline` helper in formatting.py. Subject line and Suggestion blockquote are unaffected — they render author-controlled rule metadata. JSON-Pointer fragment decoding (S-003's encoded path) is intentionally out of scope; S-003 is the only rule with that shape and the encoded form is acceptable. Closes #256
…itization validation: escape GFM inline syntax in workflow summary findings
This was referenced May 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
repository management
What this PR does / why we need it:
Merges the
validation-frameworkbranch intomain, completing the v1-rc soak period and consolidating to a single-branch development model.Pre-merge verification (2026-05-04):
validation-frameworkis strictly additive overmain(last sync via chore: merge main into validation-framework (pre-E2E sync) #251 chore-merge + chore: merge main into validation-framework #246 fallback sync).mainof all 16enabledAPI repositories: 5 errors / 430 warnings / 147 hints / 582 total.r1.5 (alpha)(/create-snapshot→/discard-snapshot) confirmed pre-snapshot validation and Release Review PR validation rendering both clean against@v1-rc=8ad56a1(currentvalidation-frameworktip).Tag handling:
@v1-rcdoes not move at merge. The lightweight tag continues to point at8ad56a1, which becomes a parent ofmain's new HEAD via the merge commit; consumers resolving@v1-rcare unaffected. Promotion to@v1is a separate decision after the central defaultadvisory → enabledflip soaks.Which issue(s) this PR fixes:
Closes #254
Closes #256
(#194 and #192 closed manually ahead of merge after their fixes soaked under
@v1-rc.)Tracking issue (kept open through post-merge cutover): #259
Companion documentation issue: #258
Special notes for reviewers:
This is a merge of a long-lived feature branch with no main-side drift; the diff is purely additive. The post-merge cutover sequence (repointing pinned consumers in ReleaseTest, updating Dependabot config, regression workflow triggers, GitHub App
external_url, and only then deleting the branch) is documented in #259 and lands as a separate small PR before the branch is deleted.Use "Create a merge commit" when merging (the default of the three dropdown options on the green merge button) to preserve the v1-rc lineage in
git log main. Do NOT use "Squash and merge" or "Rebase and merge" — both would flatten the 238-commit history and lose the merge-commit topology.Changelog input
Validation framework v1-rc merged into
main. The@v1-rctag continues to reference the same SHA (8ad56a1) and remains the consumer-facing reference until@v1GA.