ci: fuse duplicate PR gates and reduce repeated setup #975

Summary

Ready-for-review pull requests currently fan out into 15 GitHub Actions jobs. Most compilation jobs start on fresh hosted runners, repeat checkout and dependency setup, and discard their local build outputs when the job ends.

Recent ordinary PR CI runs usually complete in about 6–7 minutes of wall time, with an observed 11-minute outlier. sccache is already enabled and working, so this issue is about reducing duplicated job work and runner isolation.

The release workflow is explicitly out of scope.

Measured baseline

Recent non-release PR runs:

• Typical run: 6m39s
• Slow run: 11m00s

Representative steps from the typical run:

sccache was active in that run:

• Qwen3 CUDA compile: 79.7% hit rate
• Qwen3 CUDA Clippy: 84.8%
• Gemma 4 CUDA Clippy: 92.6%
• Qwen3.5 CUDA Clippy: 98.8%

The slow run also showed that cache hits alone do not bound wall time:

• Qwen3.5 CUDA compile: 98.7% hits, but the Cargo step took 8m21s
• Simulated frontend E2E: 100% hits (511/511), but the job took 9m44s

The typical run also reported cache write errors in several parallel jobs, including 94 in Qwen3 compile and 76 in Qwen3.5 compile. The logs do not expose the backend error detail, so the cause should be verified before changing cache behavior.

Current duplication

1. The Gemma 4/Qwen3.5 matrix runs both cargo check and cargo clippy with the same package, feature, release profile, and --all-targets surface.
2. Qwen3 has separate CUDA compile and Clippy jobs with largely overlapping package and target coverage.
3. CPU Clippy, four CPU unit-test jobs, and simulated frontend E2E each repeat Rust/sccache setup; most also repeat protobuf installation and dependency materialization.
4. Every CUDA job repeats recursive submodule checkout and CUDA toolkit installation.
5. pegainfer-kernels/build.rs launches nvcc directly, so those CUDA translation units are outside the Rust RUSTC_WRAPPER=sccache path and are rebuilt on every fresh runner.

Production invariant

A ready PR must continue to fail when any currently covered surface fails:

• DCO, attribution scripts, Rust formatting, and locked Cargo metadata
• CPU Clippy
• CPU unit tests for pegainfer-build, pegainfer-frontend, pegainfer-sim, and kvbm-logical
• simulated frontend E2E
• Qwen3, Qwen3.5, and Gemma 4 release-mode CUDA all-target compilation and lint coverage for sm_80

Before deleting a job, document which retained command covers every package, feature, and target previously checked by that job.

Proposed work

1. Remove the feature-model compile matrix entries where the retained Clippy invocation demonstrably covers the identical package/feature/--all-targets surface.
2. Map the Qwen3 compile and Clippy target surfaces. Remove the compile job only if the retained Clippy command fully subsumes it; otherwise consolidate both commands into one runner.
3. Evaluate one fused CPU gate that installs dependencies once and runs CPU Clippy, unit tests, and simulated frontend E2E against the same workspace.
4. Keep model lines in separate CUDA jobs so feature-specific build requirements remain isolated.
5. Avoid adding composite-action wrappers solely to reduce YAML duplication; they do not share runner state.

Acceptance criteria

• Every current gate is mapped to a retained command, with no package/feature/target coverage loss.
• Report before/after:
  • workflow wall time
  • summed job runtime
  • per-job setup and Cargo-step duration
  • sccache hit/miss/write-error statistics
• Use multiple completed ordinary PR runs and retain links to the raw Actions results.
• Do not claim a CI performance improvement until the before/after data is measured.
• If CPU fusion reduces summed runner time but increases wall time, report both results and make the tradeoff explicit.
• Release CI remains unchanged.

Follow-ups outside this issue

• Route direct nvcc compilation through a supported cache launcher.
• Evaluate a prebuilt CI image with CUDA, protobuf, libibverbs, and Triton.
• Investigate the GitHub Actions cache write errors.
• Evaluate persistent or self-hosted runners separately.