diff --git a/.claude/rules/ci-change-detection.md b/.claude/rules/ci-change-detection.md index 7e62cabc30..a59396a3f1 100644 --- a/.claude/rules/ci-change-detection.md +++ b/.claude/rules/ci-change-detection.md @@ -53,14 +53,14 @@ never sit behind a *more permissive* condition than a GitHub-hosted one. If you find yourself writing a looser `if:` for the more expensive job, the vocabulary is wrong (see §1) — fix that instead of widening the gate. -## 4. Three axes, layered — and the non-code one is subtracted first +## 4. Four axes, layered — and the non-code one is subtracted first | Axis | Output | Answers | | ---- | ------ | ------- | | non-code | `non_code_only` | can this diff change what the code does at all? | | architecture | `a2a3_changed` / `a5_changed` | which silicon can it reach? | | test category | `st_affected` / `ut_affected` | which suite can it break? | -| example corpus | `examples_only` | can it reach a job that builds the product and runs a fixed payload? | +| corpus | `examples_only` / `tests_only` | can it reach a job that builds the product and runs a fixed payload? | They are layered, not redundant. **Every arch and category flag subtracts `NON_CODE` before deciding**, so a non-code-only change already makes all four @@ -72,24 +72,36 @@ A job composes the axes it is actually subject to: | Job family | Gate | | ---------- | ---- | | `st-sim-*`, `st-onboard-*` | `_changed && st_affected` | -| `profiling-flags-smoke` | `(a2a3_changed \|\| a5_changed) && !examples_only` | +| `profiling-flags-smoke` | `(a2a3_changed \|\| a5_changed) && !examples_only && !tests_only` | | `ut`, `ut-a2a3`, `ut-a5` | `non_code_only != true && ut_affected` | -| `packaging-matrix` | `non_code_only != true && !examples_only` | +| `packaging-matrix` | `non_code_only != true && !examples_only && !tests_only` | `packaging-matrix` and `profiling-flags-smoke` build and install the product and -then exercise it with a **fixed, tiny payload** — one entry-point script and one -`vector_example` respectively. They are the only jobs that consume neither test -suite as a corpus, which is why they take the example axis rather than the -category one. Note what stays in: `tests/` still triggers packaging, because -`tools/verify_packaging.sh` runs a `tests/st/` file as its entry-point smoke and -`tests/` is in the sdist `include`. Only `examples/` is provably absent from -both jobs. +then exercise it with a **fixed, tiny payload** — packaging's entry-point smoke +is one `tests/st/` file, profiling's is one `vector_example`. They are the only +jobs that consume neither test suite as a corpus, which is why they take the +corpus axis rather than the category one — and the corpus axis comes in the +same shape on both sides, so a diff confined to `examples/` or to `tests/` +cannot reach either job. That is not a gap: `wheel.packages` is +`["simpler_setup", "python/simpler"]`, so both partitions are provably absent +from the product, and a payload file changed under either is still exercised by +the scene-test job that reads the same corpus. A change to a payload file was +the historical reason `tests/` stayed in — a now-deleted claim that `tests/` +was "in the sdist include", which no configuration ever made true. The UT jobs stay off the arch axis on purpose — unit tests cover shared contracts, so the cost of a falsely-skipped regression outweighs the minutes. That is a decision about the *arch* axis only; the category axis is a different question, because a scene-test-only change genuinely cannot break a unit test. +The vocabulary exists twice. `ci-self-cpu.yml` runs its own lane-local +`detect-changes` with the same outputs, and its file header says the two must +be kept in sync. That has now failed twice — `examples_only` never reached the +lane (#1607), and `tests_only` did not either (#1635) — because the gate tables +here only name `ci.yml`. Any change to an axis therefore means two files: +`.github/workflows/ci.yml` and `.github/workflows/ci-self-cpu.yml`, same commit, +and the outputs-consistency grep in §7 runs against both. + ### Write every axis in the same shape The arch and category axes use one pattern: **a partition is unaffected only @@ -165,7 +177,15 @@ bug shows up as a green check, so "CI passed" is not evidence.** Where the change is to `NON_CODE` itself, replay the pattern locally against representative file lists — a docs-only set, a `.gitignore`-only set, a single-arch set, a `ci.yml` set — and check all four land where you intended -before pushing. +before pushing. Where the change is to an *axis*, also verify both workflow +copies expose the same output set — the lane drifted twice because only +`ci.yml` was checked: + +```bash +for f in ci.yml ci-self-cpu.yml; do + echo "== $f"; grep -oE 'outputs\.[a-z_]+' ".github/workflows/$f" | sort -u +done # the two lists must be identical +``` ## Relation to the other rules diff --git a/.github/workflows/ci-self-cpu.yml b/.github/workflows/ci-self-cpu.yml index 91760b179b..e10268d376 100644 --- a/.github/workflows/ci-self-cpu.yml +++ b/.github/workflows/ci-self-cpu.yml @@ -65,6 +65,8 @@ jobs: non_code_only: ${{ steps.check.outputs.non_code_only }} st_affected: ${{ steps.check.outputs.st_affected }} ut_affected: ${{ steps.check.outputs.ut_affected }} + examples_only: ${{ steps.check.outputs.examples_only }} + tests_only: ${{ steps.check.outputs.tests_only }} steps: - name: Checkout target PR head uses: actions/checkout@v5 @@ -89,6 +91,8 @@ jobs: echo "a5_changed=true" >> "$GITHUB_OUTPUT" echo "st_affected=true" >> "$GITHUB_OUTPUT" echo "ut_affected=true" >> "$GITHUB_OUTPUT" + echo "examples_only=false" >> "$GITHUB_OUTPUT" + echo "tests_only=false" >> "$GITHUB_OUTPUT" exit 0 fi @@ -134,6 +138,25 @@ jobs: echo "ut_affected=false" >> "$GITHUB_OUTPUT" fi + # Corpus axis, mirroring ci.yml: a diff confined to examples/ or to + # tests/ cannot reach the product jobs, whose payload files live in + # those partitions but are exercised by the scene-test job instead. + EXAMPLES_ONLY='^examples/' + NON_EXAMPLE=$(echo "$FILES" | grep -vE "$EXAMPLES_ONLY" | grep -vE "$NON_CODE" || true) + if [ -n "$NON_EXAMPLE" ]; then + echo "examples_only=false" >> "$GITHUB_OUTPUT" + else + echo "examples_only=true" >> "$GITHUB_OUTPUT" + fi + + TESTS_ONLY='^(tests/)' + NON_TEST=$(echo "$FILES" | grep -vE "$TESTS_ONLY" | grep -vE "$NON_CODE" || true) + if [ -n "$NON_TEST" ]; then + echo "tests_only=false" >> "$GITHUB_OUTPUT" + else + echo "tests_only=true" >> "$GITHUB_OUTPUT" + fi + # ---------- T1: no-hardware Linux, on the cpu runner ---------- pre-commit: runs-on: [self-hosted, cpu] @@ -228,7 +251,7 @@ jobs: packaging: needs: [detect-changes] - if: needs.detect-changes.outputs.non_code_only != 'true' + if: needs.detect-changes.outputs.non_code_only != 'true' && needs.detect-changes.outputs.examples_only != 'true' && needs.detect-changes.outputs.tests_only != 'true' runs-on: [self-hosted, cpu] timeout-minutes: 60 steps: @@ -261,7 +284,7 @@ jobs: profiling-flags-smoke: needs: [detect-changes] - if: (needs.detect-changes.outputs.a2a3_changed == 'true' || needs.detect-changes.outputs.a5_changed == 'true') && needs.detect-changes.outputs.st_affected == 'true' + if: (needs.detect-changes.outputs.a2a3_changed == 'true' || needs.detect-changes.outputs.a5_changed == 'true') && needs.detect-changes.outputs.examples_only != 'true' && needs.detect-changes.outputs.tests_only != 'true' runs-on: [self-hosted, cpu] timeout-minutes: 30 steps: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0734fb0edb..db2a66ca80 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -62,7 +62,7 @@ jobs: # See docs/python-packaging.md and tools/verify_packaging.sh. packaging-matrix: needs: [detect-changes, pre-commit] - if: needs.detect-changes.outputs.non_code_only != 'true' && needs.detect-changes.outputs.examples_only != 'true' + if: needs.detect-changes.outputs.non_code_only != 'true' && needs.detect-changes.outputs.examples_only != 'true' && needs.detect-changes.outputs.tests_only != 'true' runs-on: ${{ matrix.os }} timeout-minutes: 60 strategy: @@ -375,7 +375,7 @@ jobs: # corresponding profiling block — no cross-contamination from cmake cache. profiling-flags-smoke: needs: [detect-changes, pre-commit] - if: (needs.detect-changes.outputs.a2a3_changed == 'true' || needs.detect-changes.outputs.a5_changed == 'true') && needs.detect-changes.outputs.examples_only != 'true' + if: (needs.detect-changes.outputs.a2a3_changed == 'true' || needs.detect-changes.outputs.a5_changed == 'true') && needs.detect-changes.outputs.examples_only != 'true' && needs.detect-changes.outputs.tests_only != 'true' runs-on: ubuntu-latest timeout-minutes: 30 @@ -725,6 +725,7 @@ jobs: st_affected: ${{ steps.check.outputs.st_affected }} ut_affected: ${{ steps.check.outputs.ut_affected }} examples_only: ${{ steps.check.outputs.examples_only }} + tests_only: ${{ steps.check.outputs.tests_only }} steps: - name: Checkout repository uses: actions/checkout@v5 @@ -760,6 +761,7 @@ jobs: echo "st_affected=true" >> "$GITHUB_OUTPUT" echo "ut_affected=true" >> "$GITHUB_OUTPUT" echo "examples_only=false" >> "$GITHUB_OUTPUT" + echo "tests_only=false" >> "$GITHUB_OUTPUT" exit 0 fi @@ -837,12 +839,15 @@ jobs: fi # `packaging-matrix` and `profiling-flags-smoke` build and install the - # product and then exercise it with a fixed, tiny payload. Nothing - # under examples/ reaches either: it is absent from both - # `wheel.packages` and the sdist `include`, and neither job reads the - # example corpus. Everything else stays in — `tests/` in particular, - # because `tools/verify_packaging.sh` runs one `tests/st/` file as an - # entry-point smoke and `tests/` is in the sdist include. + # product and then exercise it with a fixed, tiny payload — + # profiling-flags-smoke's lives under examples/ (vector_example), + # packaging-matrix's under tests/ (one tests/st/ file is the + # entry-point smoke of tools/verify_packaging.sh). Neither job reads + # either corpus: `wheel.packages` is ["simpler_setup", + # "python/simpler"], so both partitions are provably absent from the + # product, and a payload file changed under either is exercised by + # the scene-test job that also reads it. Each partition therefore + # gets the same axis, written in the same shape. EXAMPLES_ONLY='^examples/' NON_EXAMPLE=$(echo "$FILES" | grep -vE "$EXAMPLES_ONLY" | grep -vE "$NON_CODE" || true) if [ -n "$NON_EXAMPLE" ]; then @@ -852,6 +857,15 @@ jobs: echo "All changes are under examples/ or non-code; skipping packaging and the profiling-flag smoke" fi + TESTS_ONLY='^(tests/)' + NON_TEST=$(echo "$FILES" | grep -vE "$TESTS_ONLY" | grep -vE "$NON_CODE" || true) + if [ -n "$NON_TEST" ]; then + echo "tests_only=false" >> "$GITHUB_OUTPUT" + else + echo "tests_only=true" >> "$GITHUB_OUTPUT" + echo "All changes are under tests/ or non-code; skipping packaging and the profiling-flag smoke" + fi + # ---------- Unit tests (a5 hardware, Python + C++) ---------- ut-a5: needs: [detect-changes, pre-commit] diff --git a/docs/ci.md b/docs/ci.md index bfb566a10a..1ec6866eb6 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -27,10 +27,10 @@ The complete test-type × hardware-tier matrix. Empty cells have no tests yet; o ```text PullRequest ├── pre-commit (ubuntu-latest) - ├── packaging-matrix (ubuntu + macOS) — [needs !examples_only] - ├── profiling-flags-smoke (ubuntu-latest) — (a2a3_changed || a5_changed) && !examples_only + ├── packaging-matrix (ubuntu + macOS) — [needs !examples_only && !tests_only] + ├── profiling-flags-smoke (ubuntu-latest) — (a2a3_changed || a5_changed) && !examples_only && !tests_only ├── ut (ubuntu + macOS) — Python + C++ UT, no hardware [needs ut_affected] - ├── detect-changes (ubuntu-latest) — outputs non_code_only, a{2a3,5}_changed, {st,ut}_affected, examples_only + ├── detect-changes (ubuntu-latest) — outputs non_code_only, a{2a3,5}_changed, {st,ut}_affected, examples_only, tests_only ├── st-sim-a2a3 (ubuntu + macOS) — a2a3_changed && st_affected ├── st-sim-a5 (ubuntu + macOS) — a5_changed && st_affected ├── ut-a2a3 (a2a3 self-hosted) — Python + C++ UT, a2a3 hardware [needs ut_affected] @@ -131,12 +131,12 @@ not need `--max-parallel` manually. ### Scheduling constraints - Sim scene tests and no-hardware unit tests run on github-hosted runners (no hardware). -- `detect-changes` computes four axes from the PR diff — non-code (`non_code_only`), architecture (`a2a3_changed` / `a5_changed`), test category (`st_affected` / `ut_affected`), and example corpus (`examples_only`) — **all of them derived from one `NON_CODE` set**: `docs/`, `.docs/`, `.claude/`, `mkdocs.yml`, `.github/workflows/docs.yml`, `.gitignore`, `.pre-commit-config.yaml`, and any `*.md` file anywhere. Membership follows a file's *effect*, not its path — `mkdocs.yml` and `docs.yml` are docs tooling that happens to live outside `docs/`. An arch flag is `false` only when every changed file is in the opposite platform's tree (`src/{arch}/`, `examples/{arch}/`, `tests/{st,ut/cpp}/{arch}/`) or in `NON_CODE`. Anything else — shared C++ (`src/common/`), Python (`python/`, `simpler_setup/`), build files (`CMakeLists.txt`, `pyproject.toml`), shared test infra (`tests/ut/py/`, `tests/lint/`), tooling (`tools/`), or **`.github/workflows/ci.yml` itself** — flips both flags to `true`. `ci.yml` is deliberately excluded from `NON_CODE`: a change to the gates must run everything, including whatever it just switched off. +- `detect-changes` computes four axes from the PR diff — non-code (`non_code_only`), architecture (`a2a3_changed` / `a5_changed`), test category (`st_affected` / `ut_affected`), and corpus (`examples_only` / `tests_only`) — **all of them derived from one `NON_CODE` set**: `docs/`, `.docs/`, `.claude/`, `mkdocs.yml`, `.github/workflows/docs.yml`, `.gitignore`, `.pre-commit-config.yaml`, and any `*.md` file anywhere. Membership follows a file's *effect*, not its path — `mkdocs.yml` and `docs.yml` are docs tooling that happens to live outside `docs/`. An arch flag is `false` only when every changed file is in the opposite platform's tree (`src/{arch}/`, `examples/{arch}/`, `tests/{st,ut/cpp}/{arch}/`) or in `NON_CODE`. Anything else — shared C++ (`src/common/`), Python (`python/`, `simpler_setup/`), build files (`CMakeLists.txt`, `pyproject.toml`), shared test infra (`tests/ut/py/`, `tests/lint/`), tooling (`tools/`), or **`.github/workflows/ci.yml` itself** — flips both flags to `true`. `ci.yml` is deliberately excluded from `NON_CODE`: a change to the gates must run everything, including whatever it just switched off. - **Test-category axis:** `ST_ONLY='^(tests/st/|examples/)'` and `UT_ONLY='^tests/ut/'`, applied in the same shape as the arch patterns — a category is unaffected only when *every* changed file is exclusively the other's. `st_affected` gates the four scene-test jobs; `ut_affected` gates `ut`, `ut-a2a3`, `ut-a5`. Anything belonging to neither (root `conftest.py`, `pyproject.toml`, `simpler_setup/`, `tests/lint/`) flips both, so shared infrastructure always runs both suites. -- **Example-corpus axis:** `EXAMPLES_ONLY='^examples/'`, same shape again. `packaging-matrix` and `profiling-flags-smoke` build and install the product and then exercise it with a fixed, tiny payload — one entry-point script and one `vector_example` — so they read neither suite as a corpus and nothing under `examples/` can reach them: it is absent from both `wheel.packages` and the sdist `include`. `tests/` deliberately stays in, because `tools/verify_packaging.sh` runs a `tests/st/` file as its entry-point smoke. +- **Corpus axis:** `EXAMPLES_ONLY='^examples/'` and `TESTS_ONLY='^(tests/)'`, same shape again, one per side of the product jobs' payload. `packaging-matrix` and `profiling-flags-smoke` build and install the product and then exercise it with a fixed, tiny payload — one entry-point script (a `tests/st/` file) and one `vector_example` — so neither reads either suite as a corpus, and a diff confined to `examples/` or to `tests/` cannot reach them: `wheel.packages` is `["simpler_setup", "python/simpler"]`, so both partitions are provably absent from the product. A payload file changed under either is still exercised by the scene-test job that reads the same corpus. - **Gated jobs (scene tests):** `st-sim-{a2a3,a5}`, `st-onboard-{a2a3,a5}` run iff their platform's flag **and** `st_affected` are `true`. -- **Platform-independent jobs (all UT + packaging):** `ut`, `ut-a2a3`, `ut-a5`, `packaging-matrix` ignore the *platform* flags — unit tests exercise shared contracts (nanobind bindings, RuntimeBuilder, ring buffers, etc.) and the risk of silently skipping a regression outweighs the CI minutes saved. The `tests/ut/cpp/{arch}/` entry in the gating regex only *attributes* an arch-specific C++ UT change to that platform (so it does not spuriously flip the other arch's scene-test flag); it does not gate the UT jobs themselves. The three UT jobs do respect `ut_affected`, which is a statement about test category rather than silicon. `packaging-matrix` respects `examples_only`, which is a statement about which corpus a job reads: `examples/` is in neither `wheel.packages` nor the sdist `include`, and `tools/verify_packaging.sh` never opens it. -- **`non_code_only` is the same `NON_CODE` set, not a narrower one.** It is `true` when no changed file falls outside it. Nothing in the set can change what the code does, and no workflow consumes any of it beyond its own gate: `pre-commit` is ungated so it always exercises `.pre-commit-config.yaml`, `docs.yml` is unconditional on every PR so it always exercises `mkdocs.yml` / `docs/`, and **no workflow invokes anything under `.claude/`** (`grep -rn '\.claude' .github/workflows/` finds only comments). An **empty diff short-circuits the whole step**: attribution is impossible, so a single guard sets `non_code_only=false`, every arch and category flag `true`, and `examples_only=false`, then returns — running the full matrix, packaging and the profiling smoke included. That guard is deliberately one place; testing emptiness per flag is what previously left `non_code_only` false while both arch flags also came out false, running UT and packaging but skipping every scene test. +- **Platform-independent jobs (all UT + packaging):** `ut`, `ut-a2a3`, `ut-a5`, `packaging-matrix` ignore the *platform* flags — unit tests exercise shared contracts (nanobind bindings, RuntimeBuilder, ring buffers, etc.) and the risk of silently skipping a regression outweighs the CI minutes saved. The `tests/ut/cpp/{arch}/` entry in the gating regex only *attributes* an arch-specific C++ UT change to that platform (so it does not spuriously flip the other arch's scene-test flag); it does not gate the UT jobs themselves. The three UT jobs do respect `ut_affected`, which is a statement about test category rather than silicon. `packaging-matrix` respects the corpus axis, which is a statement about which corpus a job reads: neither `examples/` nor `tests/` is in `wheel.packages`, so a diff confined to either partition cannot change what the packaging job builds. +- **`non_code_only` is the same `NON_CODE` set, not a narrower one.** It is `true` when no changed file falls outside it. Nothing in the set can change what the code does, and no workflow consumes any of it beyond its own gate: `pre-commit` is ungated so it always exercises `.pre-commit-config.yaml`, `docs.yml` is unconditional on every PR so it always exercises `mkdocs.yml` / `docs/`, and **no workflow invokes anything under `.claude/`** (`grep -rn '\.claude' .github/workflows/` finds only comments). An **empty diff short-circuits the whole step**: attribution is impossible, so a single guard sets `non_code_only=false`, every arch and category flag `true`, and both corpus flags (`examples_only`, `tests_only`) `false`, then returns — running the full matrix, packaging and the profiling smoke included. That guard is deliberately one place; testing emptiness per flag is what previously left `non_code_only` false while both arch flags also came out false, running UT and packaging but skipping every scene test. The arch flags subtract `NON_CODE` before deciding, so a non-code-only change already makes both `false`. An arch-gated job therefore needs no separate non-code check. See [`.claude/rules/ci-change-detection.md`](../.claude/rules/ci-change-detection.md) for the invariants these gates must keep.