Add dev-reference target compatibility sweep (compat-check-targets.sh + CI) - #28
Conversation
… + CI) Builds the `dev` reference (supported_targets: '*') for every target/board each feed publishes, on 2024/next, 2024/edge, 2026/next and 2026/edge. Nothing is hardcoded — the feed is the source of truth: - targets from <repo>/<release>/<channel>/targets.json - boards from the avocado-bsp-<board> packages in target/<target>-ext, passed via --target-board (a target with no BSP runs once as itself so the failure is caught, not skipped) Each cell runs in a scratch copy of dev/ with the distro block rewritten, then a verified full reset (stamps, unlock, volume, state, lock). The matrix report (scripts/compat-report.md) is rewritten after every cell and lists what each feed ships, for comparison against the docs support matrix. CI: weekly + workflow_dispatch only, one runner per cell, aggregated report. Any failed cell fails the run; there is no skip-list.
There was a problem hiding this comment.
🟡 Changes recommended
Feed discovery currently treats fetch/parsing/metadata failures as “empty feed/no BSPs”, which can silently reduce coverage and allow false-green compatibility runs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds an automated “dev reference” target/board compatibility sweep that discovers published targets/boards from the feeds, runs avocado install + build per (target, board, release/channel) cell in an isolated scratch copy of dev/, and aggregates results into a markdown matrix report. This extends the repo’s existing verification tooling by adding a dedicated workflow for broad target coverage without affecting other references.
Changes:
- Add
scripts/compat-check-targets.shto plan/run/report a feed-driven target × board × release/channel build matrix fordev/. - Add shared helpers in
scripts/lib/common.sh(reset/prune helpers, feed discovery, failure summarization). - Add
.github/workflows/compat-check.ymlto fan out one CI job per cell and aggregate a report artifact; add.compat-scratch/to.gitignore.
File summaries
| File | Description |
|---|---|
| scripts/lib/common.sh | New shared helpers for feed discovery, docker/avocado cleanup, and error summarization used by the sweep script. |
| scripts/compat-check-targets.sh | Main sweep script: plans cells from feeds, runs install/build per cell in scratch, verifies reset, renders report. |
| scripts/compat-report.md | Initial report template/output file for the sweep report rendering. |
| .github/workflows/compat-check.yml | CI workflow to discover cells, run each cell in parallel, and aggregate/upload the report. |
| .gitignore | Ignore .compat-scratch/ scratch directory and fix .avocado-state ignore entry formatting. |
Review details
- Files reviewed: 4/5 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A transient curl/jq/gunzip failure in feed_targets or feed_boards looked like "nothing published" and silently shrank the matrix, letting a run go green on partial coverage (Copilot review on #28). - _fetch distinguishes 200 / 404 / anything else. 404 keeps its meaning: no targets.json = feed absent, no -ext repodata = bare target. - Any other status, invalid targets.json, missing primary.xml or a decompress failure returns 1 with the URL on stderr. - plan() aborts on rc 1 so neither the local run nor the CI plan job proceeds with an incomplete cell list.
There was a problem hiding this comment.
🟡 Changes recommended
There are a few concrete correctness/operability issues in the new scripts (reset cleanliness gating and network fetch determinism) and a report-quality gap when avocado isn’t installed for report-only mode.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
Previously missed (3) — in code that hasn't changed since the last review.
scripts/lib/common.sh:106
- verify_clean() is used as the gate for a successful reset, but it doesn’t check for a leftover .avocado/ directory. If rm -rf .avocado fails (permissions, transient errors), the cell could still be treated as clean even though build state remains.
scripts/compat-check-targets.sh:29 - The header comment says the tracked repo tree is never touched, but by default REPORT_FILE is under scripts/ and render_report rewrites it. This is misleading for anyone expecting a clean git working tree after running the script.
scripts/compat-check-targets.sh:75 - AVOCADO_VERSION is captured unconditionally, but --plan/--report (and the workflow’s report job) can run without the CLI installed, which renders an empty “CLI:” field in the report. Consider falling back to a clear placeholder when avocado isn’t available.
- Files reviewed: 4/5 changed files
- Comments generated: 1
- Review effort level: Lite
--connect-timeout 10 --max-time 120 so a stalled connection fails the fetch (and aborts the plan) instead of hanging a leg until the job timeout (Copilot review on #28).
There was a problem hiding this comment.
🟡 Changes recommended
The sweep script currently has failure modes where zero cells can be treated as success and where mktemp failure can cause dangerous filesystem operations, both of which should be fixed before relying on CI results.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
scripts/lib/common.sh:103
- verify_clean() claims to assert the project dir is pristine, but it doesn’t check for a leftover .avocado/ directory. If avocado_full_reset fails to remove that directory, the run can be treated as clean even though local build state remains.
- Files reviewed: 4/5 changed files
- Comments generated: 2
- Review effort level: Lite
….avocado/ Copilot review on #28: - plan producing no runnable cells (all combos skipped, malformed CELLS, unknown TARGETS) now exits 1 in --plan and run mode, and --report fails when no cell files exist. "Nothing was checked" must never read green. - run_cell bails out with RESET-FAIL if mktemp fails, before any rm/cd/cp, so an empty $scratch can never turn "$scratch/.avocado" into "/.avocado". - verify_clean also flags a leftover .avocado/ directory.
There was a problem hiding this comment.
🟡 Changes recommended
The workflow’s artifact download path nests compat-logs/compat-logs/... causing report aggregation to miss .cell files, and the aggregator currently can’t fail the run on missing planned cell results.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
scripts/compat-check-targets.sh:27
- Minor grammar nit: “writes ONE one-line result file” reads awkwardly; consider “writes a one-line result file”.
- Files reviewed: 4/5 changed files
- Comments generated: 2
- Review effort level: Lite
A CI leg that dies before uploading its .cell file showed as "…" in the report while the run still passed on the remaining cells. all_passed now requires a result for every planned cell when a plan is known (the report job gets it via CELLS from the plan artifact). Also a comment grammar nit. (Copilot review on #28.)
There was a problem hiding this comment.
🟡 Changes recommended
The sweep script currently uses feed-derived target/board identifiers directly in filesystem paths (logs/scratch/results), which can allow path traversal or malformed path creation without sanitization.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
scripts/compat-check-targets.sh:205
- The TARGETS filter expands
$ONLY_TARGETSunquoted (printf '%s\n' $ONLY_TARGETS), which allows pathname expansion (globbing) and can mis-filter targets depending on the current directory contents. Since the docs say this is a space-separated list (not patterns), split on whitespace explicitly and use fixed-string matching.
- Files reviewed: 4/5 changed files
- Comments generated: 2
- Review effort level: Lite
A transient curl/jq/gunzip failure in feed_targets or feed_boards looked like "nothing published" and silently shrank the matrix, letting a run go green on partial coverage (Copilot review on #28). - _fetch distinguishes 200 / 404 / anything else. 404 keeps its meaning: no targets.json = feed absent, no -ext repodata = bare target. - Any other status, invalid targets.json, missing primary.xml or a decompress failure returns 1 with the URL on stderr. - plan() aborts on rc 1 so neither the local run nor the CI plan job proceeds with an incomplete cell list.
--connect-timeout 10 --max-time 120 so a stalled connection fails the fetch (and aborts the plan) instead of hanging a leg until the job timeout (Copilot review on #28).
….avocado/ Copilot review on #28: - plan producing no runnable cells (all combos skipped, malformed CELLS, unknown TARGETS) now exits 1 in --plan and run mode, and --report fails when no cell files exist. "Nothing was checked" must never read green. - run_cell bails out with RESET-FAIL if mktemp fails, before any rm/cd/cp, so an empty $scratch can never turn "$scratch/.avocado" into "/.avocado". - verify_clean also flags a leftover .avocado/ directory.
Builds the
devreference (supported_targets: '*') for every target/board each feed publishes, on 2024/next, 2024/edge, 2026/next and 2026/edge, to catch build issues on anything we ship. No other reference is touched.Nothing is hardcoded — the feed is the source of truth
https://repo.avocadolinux.org/<release>/<channel>/targets.jsonavocado-bsp-<board>packages intarget/<target>-ext, passed via--target-board. A target with no BSP published runs once as itself so the failure is caught, not skipped.How a cell runs
Scratch copy of
dev/(gitignored.compat-scratch/) with the distro block rewritten →install -f→build→ verified full reset (stamps, unlock, volume, state, lock). A leftover volume/state marks the cellRESET-FAILrather than trusting the result.Report
scripts/compat-report.mdis rewritten after every cell: target × combo matrix, plus a "Published per feed" list to compare against the support matrix for anything we say we support but don't ship, or ship but don't list.CI (
.github/workflows/compat-check.yml)Weekly (Sunday 04:00 UTC) and
workflow_dispatchonly — never on push/PR. Plan job discovers cells from the feed, one runner per cell, aggregate report. Any failed cell fails the run; there is no skip-list.Local usage
Verified locally
imx8mp-var-dart · variscite-sonata@ 2024/next: install ✅, build ❌ — stone validation can't findimx-boot. A real target defect the sweep is meant to surface.workflow_dispatchbecomes available in the Actions tab (and viagh workflow run compat-check.yml -f targets=qemuarm64) once this lands on main — GitHub only registers workflows from the default branch. The workflow has no push/PR triggers, so merging runs nothing on its own.