diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 0c150dbae9..4e3029b74c 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -47,26 +47,16 @@ source-compiling installs cascaded into bash-tool interrupts that blocked even `pwd` and `echo`. (Pre-built single-script installers like Determinate Nix's are fine — see **Weekly Maintenance: MSRV & Toolchain** for the one we use. The block is specifically about long-running cargo compiles.) Instead, -query Codecov directly: - -```bash -REPO=$(gh repo view --json nameWithOwner --jq '.nameWithOwner') -curl -sL "https://api.codecov.io/api/v2/gh/${REPO%/*}/repos/${REPO#*/}/compare/?pullid=" > /tmp/codecov.json - -# Patch-level summary per file: -jq '.files[] | {name: .name.head, patch: .totals.patch}' /tmp/codecov.json - -# Uncovered added lines in a specific changed file -# (coverage.head is a LineType enum: 0=hit, 1=miss, 2=partial — filter on 1=miss): -jq '.files[] | select(.name.head == "") | .lines[] | select(.is_diff and .added and .coverage.head == 1) | {line: .number.head, code: .value}' /tmp/codecov.json -``` +query Codecov directly, following `tests/CLAUDE.md` → **Coverage +Investigation** for the endpoints and their traps. If the Codecov API markers aren't enough, download the `code-coverage-report` -artifact from the PR head's `ci` workflow run — it contains a `cobertura.xml` -with per-line hit counts: +artifact from the PR head's `coverage` workflow run — it contains a +`cobertura.xml` with per-line hit counts: ```bash -# Find the ci run on the PR head SHA: +REPO=$(gh repo view --json nameWithOwner --jq '.nameWithOwner') +# Find the coverage run on the PR head SHA: CI_RUN=$(gh api "repos/$REPO/commits//check-runs" --jq '.check_runs[] | select(.name == "code-coverage") | .details_url | capture("runs/(?[0-9]+)") | .id') # List artifacts, then download the coverage one: gh api "repos/$REPO/actions/runs/$CI_RUN/artifacts" --jq '.artifacts[] | {name, id}' @@ -297,8 +287,8 @@ Pinned third-party versions in CI are invisible to Dependabot — it follows `Ca For each weekly run, check upstream and bump: -- **`baptiste0928/cargo-install@v3` blocks** in `.github/workflows/ci.yaml`, `.github/workflows/nightly.yaml`, and `.github/actions/{test,claude}-setup/action.yaml` — every `version: "=X.Y.Z"` against `cargo info `. Today: `cargo-insta`, `cargo-nextest`, `cargo-llvm-cov`, `cargo-msrv`, `cargo-udeps`, `lychee`, `worktrunk`. The `cargo-affected` install has no version pin (follows default branch) — leave it alone. Verify each crate's `rust-version` against the pinned toolchain and note compatibility in the PR body (see PR #1657 for the format). -- **`hustcer/setup-nu@v3`** `version:` input — latest from `gh api repos/nushell/nushell/releases/latest --jq '.tag_name'`. Four call sites: `ci.yaml` (`code-coverage`), `nightly.yaml` (`feature-powerset`), `benchmarks.yaml` (`benchmarks`), and `actions/test-setup/action.yaml`. +- **`baptiste0928/cargo-install@v3` blocks** in `.github/workflows/{ci,coverage,nightly}.yaml` and `.github/actions/{test,claude}-setup/action.yaml` — every `version: "=X.Y.Z"` against `cargo info `. Today: `cargo-insta`, `cargo-nextest`, `cargo-llvm-cov`, `cargo-msrv`, `cargo-udeps`, `lychee`, `worktrunk`. The `cargo-affected` install has no version pin (follows default branch) — leave it alone. Verify each crate's `rust-version` against the pinned toolchain and note compatibility in the PR body (see PR #1657 for the format). +- **`hustcer/setup-nu@v3`** `version:` input — latest from `gh api repos/nushell/nushell/releases/latest --jq '.tag_name'`. Four call sites: `coverage.yaml` (`code-coverage`), `nightly.yaml` (`feature-powerset`), `benchmarks.yaml` (`benchmarks`), and `actions/test-setup/action.yaml`. - **`taiki-e/install-action@v2.x`** `tool: zola@` in the `check-docs` job — latest from `gh api repos/getzola/zola/releases/latest --jq '.tag_name'`. - **Runner images** — `ubuntu-24.04`, `macos-15`, `windows-2022`. Keep `windows-2022` pinned (actions/runner-images#12677 — windows-2025 lacks the D: drive). diff --git a/.claude/skills/running-tend/references/nightly-cleaner.md b/.claude/skills/running-tend/references/nightly-cleaner.md index 307b6a0494..d14dffe8fb 100644 --- a/.claude/skills/running-tend/references/nightly-cleaner.md +++ b/.claude/skills/running-tend/references/nightly-cleaner.md @@ -12,4 +12,4 @@ For each `.rs` file in the survey, also check: ## Repo-Wide CI Breakage -A failure that reproduces identically on `main` and every PR (a broken system-package install in `code-coverage`, say) belongs to `tend-ci-fix`, which fires on any `ci`-workflow `failure` on `main`; a non-required job failing is enough to trigger it. It does not fire on runs that end `cancelled`, so record the breakage in the summary rather than assuming the handoff lands. +A failure that reproduces identically on `main` and every PR (a broken system-package install in `code-coverage`, say) belongs to `tend-ci-fix`, which fires on any watched-workflow `failure` on `main` — the list lives in `.config/tend.yaml` under `ci-fix.watched_workflows`; a non-required job failing is enough to trigger it. It does not fire on runs that end `cancelled`, so record the breakage in the summary rather than assuming the handoff lands. diff --git a/.config/tend.yaml b/.config/tend.yaml index a440a76d29..970e3c58bf 100644 --- a/.config/tend.yaml +++ b/.config/tend.yaml @@ -8,6 +8,11 @@ workflows: ci-fix: watched_workflows: - ci + - coverage + # `affected-tests` is continue-on-error, so watching this workflow only + # signs Tend up for `collect-affected` failures — a stale DB degrades + # selection on every later PR, so it is worth fixing. + - affected - publish-docs # nightly hosts `full-tests` (the cargo-affected safety net), so a red # nightly is actionable: Tend fixes it instead of leaving main's gaps diff --git a/.github/.codecov.yaml b/.github/.codecov.yaml index 456a22e89f..534c3a179e 100644 --- a/.github/.codecov.yaml +++ b/.github/.codecov.yaml @@ -1,7 +1,10 @@ comment: false +# `cargo llvm-cov` reports only `src/`, so this is a guard rather than a live +# filter. It has to be `tests/**`: codecov compiles `**/tests/**` to +# `.*/tests/.*`, which needs a leading directory and so misses `tests/` itself. ignore: - - "**/tests/**" + - "tests/**" coverage: status: diff --git a/.github/workflows/affected.yaml b/.github/workflows/affected.yaml new file mode 100644 index 0000000000..da10c8463a --- /dev/null +++ b/.github/workflows/affected.yaml @@ -0,0 +1,272 @@ +name: affected + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: + +# Grouped per-ref on main, which samples: a merge train cancels the queued run +# and only the newest one collects. That's the intended cadence — a DB a few +# commits old still anchors a correct superset (see the cache strategy below), +# and collecting per-commit would mean three ~2h matrices per merge. Sampling +# is also why these jobs sit in their own workflow: `collect` holds a group for +# ~2h, so anything sharing it gets sampled too, whether or not that suits it. +# PR runs supersede their own. +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +env: + # Both jobs restore the rust-cache that `test` saves, and Swatinem/rust-cache + # hashes CARGO*/RUST* into the key, so these must match ci.yaml exactly + # (.github/CLAUDE.md → Build environment). + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: 0 + RUSTFLAGS: -C debuginfo=0 + +jobs: + # cargo-affected: runs only the tests whose recorded coverage overlaps the + # diff. Advisory at this stage — `test (linux/macos/windows)` still runs the + # full suite. The signal is the runtime delta on the exact-match path and + # whether selection misses any failures the full suite catches. + # + # Both jobs run as the same linux/macos/windows matrix as `test`. The + # fingerprint embeds `rustc -vV` (host triple), so a DB collected on one OS + # cache-misses on the others — each OS's `affected-tests` leg pairs with a + # `collect-affected` DB from the same OS via the `runner.os` cache key. + # + # TODO(affected-only PRs): the fast pre-merge gates are already split into the + # `fast-checks` job, so `test (linux/macos/windows)` is now purely the full + # nextest suite — the long-pole, and a drop-in shape for the affected legs. + # Once `affected tests (*)` has a track record of catching every failure the + # full suite catches on all three OSes, flip: make the affected legs required + # and delete `ci`'s PR `test` job — its full-matrix coverage already runs as + # `full-tests` in the `nightly` workflow. `fast-checks` stays required and + # unchanged. Confidence bar: a sustained stretch of PRs where the advisory + # legs and the required full matrix agree — no failure the full suite caught + # that affected selection missed. + # + # Until then the full `test` matrix stays required and runs on every PR, + # unchanged. Do NOT reach affected-only by making the required `test (*)` + # checks skip=pass on PRs — relaxing the merge gate is a deliberate + # branch-protection change owned by the repo admin (see `.github/CLAUDE.md`), + # not a silent workflow skip. + # + # The full suite must keep running after the flip, because cargo-affected + # misses non-Rust inputs (`include_str!`, templates, SQL), build-time inputs + # not in its fingerprint (build.rs, rust-toolchain.toml, .cargo/config.toml), + # and proc-macro source edits. Its home is the `full-tests` matrix in the + # `nightly` workflow (nightly cron, the `nightly` label, and before a + # release) — NOT push-to-main: a failure there means affected under-selected, + # and that must not redden main. Nightly failures are non-blocking and + # Tend-fixable. The Linux `--unreferenced reject` orphan check, intrinsically + # full-suite, rides `full-tests` too. + # + # Cache strategy: + # - `collect-affected` (push to main) saves `target/affected/coverage.db` + # to actions/cache keyed on the main commit sha. We cache only the DB, + # not the parent dir — cargo-affected drops its profraw staging dir + # at the end of every successful collect, but caching the path + # explicitly keeps the contract obvious and ~10 GB of profile bundles + # from leaking into the cache if that cleanup ever regresses. + # - `collect-affected` also restores the most recent prior main DB before + # collecting, so the new DB accumulates rows for up to FINGERPRINT_KEEP + # (=10) recent main-tip env_fingerprints (LRU-evicted in `Db::gc`). + # PRs whose manifests match any of those fingerprints get exact-match + # selection instead of the all-or-nothing single-fingerprint cache. + # Don't "simplify" by removing the restore — it's load-bearing. + # - `affected-tests` (PRs) restores the cache. Primary key is the + # PR/main merge-base — when collect ran on that exact commit, we get a + # tight diff (`PR changes only`) and the smallest possible selection. + # - Restore-keys fall back to the most recent main DB. Its `collect_sha` + # is typically a sibling of the PR's HEAD, not a strict ancestor. + # cargo-affected uses any sha still in the repo as a diff anchor, so + # the fallback drives normal selection — over-includes tests touched + # by main commits between the merge-base and collect_sha (correct + # superset), but never widens to "run everything" unless the cache is + # missing entirely. + # - Cache keys include `runner.os` (fingerprint embeds `rustc -vV`) and a + # manual `db-v{N}` marker. Bump the marker if cargo-affected ships an + # on-disk schema change; the cache is otherwise version-agnostic. + collect-affected: + name: collect affected coverage (${{ matrix.name }}) + if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'workflow_dispatch' }} + strategy: + fail-fast: false + matrix: + include: + - os: ubuntu-24.04 + name: linux + - os: macos-15 + name: macos + - os: windows-2022 + name: windows + runs-on: ${{ matrix.os }} + steps: + - name: 📂 Checkout code + uses: actions/checkout@v7 + with: + # `cargo affected run` later diffs PR HEAD against the sha that was + # HEAD when collect ran. That sha must be reachable. + fetch-depth: 0 + + - uses: ./.github/actions/test-setup + with: + # Restore-only: collect builds coverage-instrumented artifacts (it adds + # llvm-tools below), which must not overwrite the shared cache that + # `test` saves for everyone else. + save-cache: "false" + + - name: "Use fast D: drive for temp files (Windows)" + if: runner.os == 'Windows' + shell: pwsh + run: | + New-Item -ItemType Directory -Force -Path "D:\tmp" | Out-Null + echo "TEMP=D:\tmp" >> $env:GITHUB_ENV + echo "TMP=D:\tmp" >> $env:GITHUB_ENV + + - name: Install cargo-affected + uses: baptiste0928/cargo-install@v3 + with: + crate: cargo-affected + git: https://github.com/max-sixty/cargo-affected + # No rev/branch/tag → action resolves to latest default-branch sha. + + - name: Install llvm-tools + run: rustup component add llvm-tools + + - name: 💾 Restore prior coverage DB + uses: actions/cache/restore@v6 + with: + path: target/affected/coverage.db + # Primary key matches the current sha (no-op on first push of this + # commit; on re-runs of the same sha, lets us skip rebuilding from + # scratch). + key: cargo-affected-db-v1-${{ runner.os }}-${{ github.sha }} + # Fall back to any prior main DB. cargo-affected preserves rows for + # up to FINGERPRINT_KEEP (=10) distinct fingerprints in one DB, + # evicting LRU on each collect. By feeding a prior DB into the new + # collect, we accumulate fingerprint snapshots across main commits — + # PRs whose manifests match any of the last ~10 main fingerprints + # get exact-match selection, instead of the all-or-nothing + # single-fingerprint cache. + restore-keys: | + cargo-affected-db-v1-${{ runner.os }}- + + - name: 📊 Collect coverage data + run: cargo affected collect -- --features shell-integration-tests + + - name: 💾 Save coverage DB + uses: actions/cache/save@v6 + with: + # Only the SQLite DB. `target/affected/profraw-/` holds raw + # profile bundles (~5 GB on this repo) that aren't needed past the + # current collect — caching them blows past the 10 GB repo cache + # cap and forces eviction of every prior cache. + path: target/affected/coverage.db + key: cargo-affected-db-v1-${{ runner.os }}-${{ github.sha }} + + affected-tests: + name: affected tests (${{ matrix.name }}, advisory) + if: github.event_name == 'pull_request' + strategy: + fail-fast: false + matrix: + include: + - os: ubuntu-24.04 + name: linux + - os: macos-15 + name: macos + - os: windows-2022 + name: windows + runs-on: ${{ matrix.os }} + # Advisory while we calibrate selection accuracy against the full suite. + # The full matrix (`test (linux/macos/windows)`) is still required for + # merge. + continue-on-error: true + steps: + - name: 📂 Checkout code + uses: actions/checkout@v7 + with: + # Need the cached `collect_sha` reachable from HEAD for the diff; + # see collect-affected above. + fetch-depth: 0 + + - name: Compute merge-base for cache key + id: mb + shell: bash + env: + PR_HEAD: ${{ github.event.pull_request.head.sha }} + run: | + git fetch --no-tags --depth=200 origin main + sha=$(git merge-base origin/main "$PR_HEAD") + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "merge-base with origin/main: $sha" + + - uses: ./.github/actions/test-setup + with: + # PR-only job: restore `test`'s shared main cache, never save. This is + # the job that most needs the shared key — a per-job cache would have + # no main baseline (it never runs on main) and cold-build every PR. + save-cache: "false" + + - name: "Use fast D: drive for temp files (Windows)" + if: runner.os == 'Windows' + shell: pwsh + run: | + New-Item -ItemType Directory -Force -Path "D:\tmp" | Out-Null + echo "TEMP=D:\tmp" >> $env:GITHUB_ENV + echo "TMP=D:\tmp" >> $env:GITHUB_ENV + + # Restore AFTER test-setup so we land on top of rust-cache's tar + # extraction. rust-cache's restore step calls `cleanTargetDir` on a + # partial cache hit (`full match: false`), which deletes file children + # of `target/affected/` — including a `coverage.db` we'd just dropped + # there. Order: rust-cache populates `target/` first; we then drop the + # DB on top, where nothing else touches it. + - name: 💾 Restore coverage DB + uses: actions/cache/restore@v6 + with: + path: target/affected/coverage.db + # Exact match: collect ran on the PR's merge-base → tight diff. + key: cargo-affected-db-v1-${{ runner.os }}-${{ steps.mb.outputs.sha }} + # Fallback: most recent main DB. cargo-affected runs all tests when + # the cached `collect_sha` is missing from the repo (rebased and + # pruned, beyond a shallow clone boundary). Sibling shas — including + # PR-vs-main-tip — drive normal selection. + restore-keys: | + cargo-affected-db-v1-${{ runner.os }}- + + - name: Install cargo-affected + uses: baptiste0928/cargo-install@v3 + with: + crate: cargo-affected + git: https://github.com/max-sixty/cargo-affected + # No rev/branch/tag → action resolves to latest default-branch sha. + + - name: 🎯 Run affected tests + env: + # An empty affected selection is a valid outcome — a diff can touch + # nothing with recorded coverage. nextest's default exits 4 ("no + # tests to run"), which fails this advisory job spuriously. Windows + # hits this every run: the `#[cfg_attr(windows, ignore)]` tests never + # execute during `collect` (nextest skips ignored tests), so they get + # no coverage rows and cargo-affected keeps re-flagging them as `new` + # and selecting them — then `nextest run` skips them as ignored too, + # so zero run. `warn` exits 0 and leaves the breadcrumb in the log. + NEXTEST_NO_TESTS: warn + run: cargo affected run --report-json target/affected/report.json -- --features shell-integration-tests + + # Report writes BEFORE nextest runs, so it survives test failures — + # uploading on `!cancelled()` makes it the most useful diagnostic + # when tests fail (cache state, fingerprint divergence, selection + # reasons). Schema documented at: + # https://github.com/max-sixty/cargo-affected/blob/main/docs/report-json.md + - name: 📊 Upload cargo-affected report + if: ${{ !cancelled() }} + uses: actions/upload-artifact@v7 + with: + name: cargo-affected-report-${{ matrix.name }} + path: target/affected/report.json diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 59d8381514..034ffb2a7f 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -233,295 +233,3 @@ jobs: - name: 🕷️ Build docs run: zola build working-directory: docs - - # cargo-affected: runs only the tests whose recorded coverage overlaps the - # diff. Advisory at this stage — `test (linux/macos/windows)` still runs the - # full suite. The signal is the runtime delta on the exact-match path and - # whether selection misses any failures the full suite catches. - # - # Both jobs run as the same linux/macos/windows matrix as `test`. The - # fingerprint embeds `rustc -vV` (host triple), so a DB collected on one OS - # cache-misses on the others — each OS's `affected-tests` leg pairs with a - # `collect-affected` DB from the same OS via the `runner.os` cache key. - # - # TODO(affected-only PRs): the fast pre-merge gates are already split into the - # `fast-checks` job, so `test (linux/macos/windows)` is now purely the full - # nextest suite — the long-pole, and a drop-in shape for the affected legs. - # Once `affected tests (*)` has a track record of catching every failure the - # full suite catches on all three OSes, flip: make the affected legs required - # and delete this PR `test` job — its full-matrix coverage already runs as - # `full-tests` in the `nightly` workflow. `fast-checks` stays required and - # unchanged. Confidence bar: a sustained stretch of PRs where the advisory - # legs and the required full matrix agree — no failure the full suite caught - # that affected selection missed. - # - # Until then the full `test` matrix stays required and runs on every PR, - # unchanged. Do NOT reach affected-only by making the required `test (*)` - # checks skip=pass on PRs — relaxing the merge gate is a deliberate - # branch-protection change owned by the repo admin (see `.github/CLAUDE.md`), - # not a silent workflow skip. - # - # The full suite must keep running after the flip, because cargo-affected - # misses non-Rust inputs (`include_str!`, templates, SQL), build-time inputs - # not in its fingerprint (build.rs, rust-toolchain.toml, .cargo/config.toml), - # and proc-macro source edits. Its home is the `full-tests` matrix in the - # `nightly` workflow (nightly cron, the `nightly` label, and before a - # release) — NOT push-to-main: a failure there means affected under-selected, - # and that must not redden main. Nightly failures are non-blocking and - # Tend-fixable. The Linux `--unreferenced reject` orphan check, intrinsically - # full-suite, rides `full-tests` too. - # - # Cache strategy: - # - `collect-affected` (push to main) saves `target/affected/coverage.db` - # to actions/cache keyed on the main commit sha. We cache only the DB, - # not the parent dir — cargo-affected drops its profraw staging dir - # at the end of every successful collect, but caching the path - # explicitly keeps the contract obvious and ~10 GB of profile bundles - # from leaking into the cache if that cleanup ever regresses. - # - `collect-affected` also restores the most recent prior main DB before - # collecting, so the new DB accumulates rows for up to FINGERPRINT_KEEP - # (=10) recent main-tip env_fingerprints (LRU-evicted in `Db::gc`). - # PRs whose manifests match any of those fingerprints get exact-match - # selection instead of the all-or-nothing single-fingerprint cache. - # Don't "simplify" by removing the restore — it's load-bearing. - # - `affected-tests` (PRs) restores the cache. Primary key is the - # PR/main merge-base — when collect ran on that exact commit, we get a - # tight diff (`PR changes only`) and the smallest possible selection. - # - Restore-keys fall back to the most recent main DB. Its `collect_sha` - # is typically a sibling of the PR's HEAD, not a strict ancestor. - # cargo-affected uses any sha still in the repo as a diff anchor, so - # the fallback drives normal selection — over-includes tests touched - # by main commits between the merge-base and collect_sha (correct - # superset), but never widens to "run everything" unless the cache is - # missing entirely. - # - Cache keys include `runner.os` (fingerprint embeds `rustc -vV`) and a - # manual `db-v{N}` marker. Bump the marker if cargo-affected ships an - # on-disk schema change; the cache is otherwise version-agnostic. - collect-affected: - name: collect affected coverage (${{ matrix.name }}) - if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'workflow_dispatch' }} - strategy: - fail-fast: false - matrix: - include: - - os: ubuntu-24.04 - name: linux - - os: macos-15 - name: macos - - os: windows-2022 - name: windows - runs-on: ${{ matrix.os }} - steps: - - name: 📂 Checkout code - uses: actions/checkout@v7 - with: - # `cargo affected run` later diffs PR HEAD against the sha that was - # HEAD when collect ran. That sha must be reachable. - fetch-depth: 0 - - - uses: ./.github/actions/test-setup - with: - # Restore-only: collect builds coverage-instrumented artifacts (it adds - # llvm-tools below), which must not overwrite the shared cache that - # `test` saves for everyone else. - save-cache: "false" - - - name: "Use fast D: drive for temp files (Windows)" - if: runner.os == 'Windows' - shell: pwsh - run: | - New-Item -ItemType Directory -Force -Path "D:\tmp" | Out-Null - echo "TEMP=D:\tmp" >> $env:GITHUB_ENV - echo "TMP=D:\tmp" >> $env:GITHUB_ENV - - - name: Install cargo-affected - uses: baptiste0928/cargo-install@v3 - with: - crate: cargo-affected - git: https://github.com/max-sixty/cargo-affected - # No rev/branch/tag → action resolves to latest default-branch sha. - - - name: Install llvm-tools - run: rustup component add llvm-tools - - - name: 💾 Restore prior coverage DB - uses: actions/cache/restore@v6 - with: - path: target/affected/coverage.db - # Primary key matches the current sha (no-op on first push of this - # commit; on re-runs of the same sha, lets us skip rebuilding from - # scratch). - key: cargo-affected-db-v1-${{ runner.os }}-${{ github.sha }} - # Fall back to any prior main DB. cargo-affected preserves rows for - # up to FINGERPRINT_KEEP (=10) distinct fingerprints in one DB, - # evicting LRU on each collect. By feeding a prior DB into the new - # collect, we accumulate fingerprint snapshots across main commits — - # PRs whose manifests match any of the last ~10 main fingerprints - # get exact-match selection, instead of the all-or-nothing - # single-fingerprint cache. - restore-keys: | - cargo-affected-db-v1-${{ runner.os }}- - - - name: 📊 Collect coverage data - run: cargo affected collect -- --features shell-integration-tests - - - name: 💾 Save coverage DB - uses: actions/cache/save@v6 - with: - # Only the SQLite DB. `target/affected/profraw-/` holds raw - # profile bundles (~5 GB on this repo) that aren't needed past the - # current collect — caching them blows past the 10 GB repo cache - # cap and forces eviction of every prior cache. - path: target/affected/coverage.db - key: cargo-affected-db-v1-${{ runner.os }}-${{ github.sha }} - - affected-tests: - name: affected tests (${{ matrix.name }}, advisory) - if: github.event_name == 'pull_request' - strategy: - fail-fast: false - matrix: - include: - - os: ubuntu-24.04 - name: linux - - os: macos-15 - name: macos - - os: windows-2022 - name: windows - runs-on: ${{ matrix.os }} - # Advisory while we calibrate selection accuracy against the full suite. - # The full matrix (`test (linux/macos/windows)`) is still required for - # merge. - continue-on-error: true - steps: - - name: 📂 Checkout code - uses: actions/checkout@v7 - with: - # Need the cached `collect_sha` reachable from HEAD for the diff; - # see collect-affected above. - fetch-depth: 0 - - - name: Compute merge-base for cache key - id: mb - shell: bash - env: - PR_HEAD: ${{ github.event.pull_request.head.sha }} - run: | - git fetch --no-tags --depth=200 origin main - sha=$(git merge-base origin/main "$PR_HEAD") - echo "sha=$sha" >> "$GITHUB_OUTPUT" - echo "merge-base with origin/main: $sha" - - - uses: ./.github/actions/test-setup - with: - # PR-only job: restore `test`'s shared main cache, never save. This is - # the job that most needs the shared key — a per-job cache would have - # no main baseline (it never runs on main) and cold-build every PR. - save-cache: "false" - - - name: "Use fast D: drive for temp files (Windows)" - if: runner.os == 'Windows' - shell: pwsh - run: | - New-Item -ItemType Directory -Force -Path "D:\tmp" | Out-Null - echo "TEMP=D:\tmp" >> $env:GITHUB_ENV - echo "TMP=D:\tmp" >> $env:GITHUB_ENV - - # Restore AFTER test-setup so we land on top of rust-cache's tar - # extraction. rust-cache's restore step calls `cleanTargetDir` on a - # partial cache hit (`full match: false`), which deletes file children - # of `target/affected/` — including a `coverage.db` we'd just dropped - # there. Order: rust-cache populates `target/` first; we then drop the - # DB on top, where nothing else touches it. - - name: 💾 Restore coverage DB - uses: actions/cache/restore@v6 - with: - path: target/affected/coverage.db - # Exact match: collect ran on the PR's merge-base → tight diff. - key: cargo-affected-db-v1-${{ runner.os }}-${{ steps.mb.outputs.sha }} - # Fallback: most recent main DB. cargo-affected runs all tests when - # the cached `collect_sha` is missing from the repo (rebased and - # pruned, beyond a shallow clone boundary). Sibling shas — including - # PR-vs-main-tip — drive normal selection. - restore-keys: | - cargo-affected-db-v1-${{ runner.os }}- - - - name: Install cargo-affected - uses: baptiste0928/cargo-install@v3 - with: - crate: cargo-affected - git: https://github.com/max-sixty/cargo-affected - # No rev/branch/tag → action resolves to latest default-branch sha. - - - name: 🎯 Run affected tests - env: - # An empty affected selection is a valid outcome — a diff can touch - # nothing with recorded coverage. nextest's default exits 4 ("no - # tests to run"), which fails this advisory job spuriously. Windows - # hits this every run: the `#[cfg_attr(windows, ignore)]` tests never - # execute during `collect` (nextest skips ignored tests), so they get - # no coverage rows and cargo-affected keeps re-flagging them as `new` - # and selecting them — then `nextest run` skips them as ignored too, - # so zero run. `warn` exits 0 and leaves the breadcrumb in the log. - NEXTEST_NO_TESTS: warn - run: cargo affected run --report-json target/affected/report.json -- --features shell-integration-tests - - # Report writes BEFORE nextest runs, so it survives test failures — - # uploading on `!cancelled()` makes it the most useful diagnostic - # when tests fail (cache state, fingerprint divergence, selection - # reasons). Schema documented at: - # https://github.com/max-sixty/cargo-affected/blob/main/docs/report-json.md - - name: 📊 Upload cargo-affected report - if: ${{ !cancelled() }} - uses: actions/upload-artifact@v7 - with: - name: cargo-affected-report-${{ matrix.name }} - path: target/affected/report.json - - code-coverage: - runs-on: ubuntu-24.04 - steps: - - name: 📂 Checkout code - uses: actions/checkout@v7 - - - uses: baptiste0928/cargo-install@v3 - with: - crate: cargo-llvm-cov - version: "=0.8.7" - - - name: 💰 Cache - uses: Swatinem/rust-cache@v2 - with: - # Own cache — coverage-instrumented builds differ from the shared test - # cache. Already main-gated; cache-bin:false matches everywhere else. - cache-bin: "false" - save-if: ${{ github.ref == 'refs/heads/main' }} - - - name: Install shells (zsh, fish) - run: sudo apt-get update && sudo apt-get install -y zsh fish - - - name: Install nushell - uses: hustcer/setup-nu@v3 - with: - version: '0.114.1' - - # Ensure nothing remains from caching - - run: cargo llvm-cov clean --workspace - - - name: 📊 Generate coverage report - run: cargo llvm-cov --features shell-integration-tests --cobertura --output-path=cobertura.xml - - - name: Upload code coverage results - uses: actions/upload-artifact@v7 - with: - name: code-coverage-report - path: cobertura.xml - - - name: Upload to codecov.io - uses: codecov/codecov-action@v7.0.0 - with: - files: cobertura.xml - # Soft-fail on fork PRs: secrets aren't available there, so we rely on - # tokenless upload, which can hiccup. Keep hard-fail on the main repo. - fail_ci_if_error: ${{ github.repository_owner == 'max-sixty' }} - token: ${{ secrets.CODECOV_TOKEN }} diff --git a/.github/workflows/coverage.yaml b/.github/workflows/coverage.yaml new file mode 100644 index 0000000000..5b8022ac41 --- /dev/null +++ b/.github/workflows/coverage.yaml @@ -0,0 +1,77 @@ +name: coverage + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: + +# Keyed per-commit on main, so a merge train can't cancel a queued upload: +# GitHub cancels the pending run whenever a newer one joins the group, and +# codecov picks a PR's base by walking back to the newest ancestor that has a +# report. A main commit that skips its upload therefore leaves later PRs +# comparing against a stale base. That's why coverage sits outside `ci`, whose +# group is shared with jobs that hold a run open for hours. PR runs still +# supersede their own. +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +env: + # Swatinem/rust-cache hashes CARGO*/RUST* into the cache key, so these track + # ci.yaml (.github/CLAUDE.md → Build environment). + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: 0 + RUSTFLAGS: -C debuginfo=0 + +jobs: + code-coverage: + runs-on: ubuntu-24.04 + steps: + - name: 📂 Checkout code + uses: actions/checkout@v7 + + - uses: baptiste0928/cargo-install@v3 + with: + crate: cargo-llvm-cov + version: "=0.8.7" + + - name: 💰 Cache + uses: Swatinem/rust-cache@v2 + with: + # Own cache — coverage-instrumented builds differ from the shared test + # cache. Already main-gated; cache-bin:false matches everywhere else. + cache-bin: "false" + save-if: ${{ github.ref == 'refs/heads/main' }} + + - name: Install shells (zsh, fish) + run: sudo apt-get update && sudo apt-get install -y zsh fish + + - name: Install nushell + uses: hustcer/setup-nu@v3 + with: + version: '0.114.1' + + # Ensure nothing remains from caching + - run: cargo llvm-cov clean --workspace + + - name: 📊 Generate coverage report + run: cargo llvm-cov --features shell-integration-tests --cobertura --output-path=cobertura.xml + + - name: Upload code coverage results + uses: actions/upload-artifact@v7 + with: + name: code-coverage-report + path: cobertura.xml + + - name: Upload to codecov.io + uses: codecov/codecov-action@v7.0.0 + with: + files: cobertura.xml + # Soft-fail on fork PRs: secrets aren't available there, so we rely on + # tokenless upload, which can hiccup. Keep hard-fail on the main repo. + # The head repo is what tells the two apart — `github.repository_owner` + # is the base repo's owner on a fork PR too. + fail_ci_if_error: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }} + token: ${{ secrets.CODECOV_TOKEN }} diff --git a/.github/workflows/tend-ci-fix.yaml b/.github/workflows/tend-ci-fix.yaml index e9fa3a0aee..498066864d 100644 --- a/.github/workflows/tend-ci-fix.yaml +++ b/.github/workflows/tend-ci-fix.yaml @@ -9,7 +9,7 @@ name: tend-ci-fix on: workflow_run: - workflows: ["ci", "publish-docs", "nightly"] + workflows: ["ci", "coverage", "affected", "publish-docs", "nightly"] types: [completed] branches: ["main"] diff --git a/tests/CLAUDE.md b/tests/CLAUDE.md index ea4934725e..805543d0a1 100644 --- a/tests/CLAUDE.md +++ b/tests/CLAUDE.md @@ -20,7 +20,7 @@ A target-filtered run (`--lib`, `--test integration`, …) on a fresh `target/` ## Coverage Investigation -`task coverage` runs the suite and writes an HTML report to `target/llvm-cov/html/index.html`. Both CI (`code-coverage` job) and local `task coverage` pass `--features shell-integration-tests`, so code behind that flag is compiled and measured. +`task coverage` runs the suite and writes an HTML report to `target/llvm-cov/html/index.html`. Both CI (the `coverage` workflow) and local `task coverage` pass `--features shell-integration-tests`, so code behind that flag is compiled and measured. When `codecov/patch` fails, investigate before declaring ready (the merge gate itself is in the root `CLAUDE.md` → Coverage): @@ -29,7 +29,34 @@ task coverage cargo llvm-cov report --show-missing-lines | grep # authoritative miss list; matches codecov line-for-line ``` -For each uncovered function, either write a test (integration tests via `assert_cmd_snapshot!` do capture subprocess coverage) or document why it's intentionally untested. If codecov's compare API must be queried directly, `coverage.head` is a `LineType` enum: `0=hit`, `1=miss`, `2=partial`, and per-file `.totals.head.diff` (`[files, lines, hits, misses, partials, coverage, …]`) is what reproduces the posted patch percentage — the top-level `totals.base.diff` reports different numbers. Prefer measuring: the API is for disputing a posted check, not a substitute for `task coverage`. +For each uncovered function, either write a test (integration tests via `assert_cmd_snapshot!` do capture subprocess coverage) or document why it's intentionally untested. + +**Querying codecov directly** serves two cases the local report can't: disputing a posted check, and running in CI, where `task coverage` isn't installed. Prefer measuring everywhere else. + +```bash +API=https://api.codecov.io/api/v2/github/max-sixty/repos/worktrunk +# Full SHAs throughout; an abbreviation 404s. `?pullid=N` compares the PR's +# *current* head, so name both SHAs to ask about an earlier commit. +curl -sL "$API/compare/?base=&head=" > /tmp/codecov.json + +# Patch coverage per file. `.name` is an object, and the files carrying patch +# lines are the ones with `has_diff`: +jq '.files[] | select(.has_diff) | {name: .name.head, patch: .totals.patch}' /tmp/codecov.json + +# The missed patch lines in one file. `.coverage.head` is a LineType enum +# (0=hit, 1=miss, 2=partial), and `.added` keeps context lines inside a hunk +# from reading as patch misses: +jq '.files[] | select(.name.head == "") | .lines[] + | select(.is_diff and .added and .coverage.head == 1) | {line: .number.head, code: .value}' /tmp/codecov.json + +# Whole-file line coverage at one commit. No trailing slash after the path — +# the route swallows it and answers 404 "coverage info not found": +curl -sL "$API/file_report/?sha=" +``` + +Per-file `.totals.patch` (equivalently `.totals.head.diff`, `[files, lines, hits, misses, …]`) holds that file's patch numbers, and the posted percentage aggregates them over the files in the PR's own diff. The top-level `totals.base.diff` is a different quantity: the base's coverage of those lines. Commit messages arrive with raw newlines in them, so the JSON is strictly invalid — `jq` copes, Python needs `json.loads(…, strict=False)`. + +**A compare listing files the PR never touched** means the merge-base has no report, so codecov walked back to the newest ancestor that does and diffed from there. The posted patch check still scopes to the PR's own diff; it's the API object that widens. Every main commit uploads from the `coverage` workflow, so this points at a failed or missing run on the base commit. **`skim` fails with E0554 (`#![feature]` on stable):** the local `cargo-llvm-cov` predates 0.7.0, which stopped putting the coverage flags in global `RUSTFLAGS` and started instrumenting only workspace crates. Older versions leak `--cfg=coverage` into every dependency, and `skim` gates a nightly feature on it. Install the version the `code-coverage` job pins rather than working around it (`--no-cfg-coverage` also avoids it; `--no-rustc-wrapper` reinstates it).