PR #8443 (CE AR no pipeline) added src/device/ce_reduce.cc containing all 40 (type, redop) kernel instantiations of ncclCeLocalReduceKernelVec in a single translation unit. LLVM's runtime loop-unroller auto-unrolled the rank-reduction loop (nRanks is a runtime value, not a compile-time constant) by up to 64x on top of the existing manual unroll/vectorization, producing ~900 PHI nodes and ~150 shufflevector ops per kernel for int8_t/uint8_t Min/Max alone. This single translation unit became one unparallelizable compile job that took ~48 minutes by itself, dominating the whole build's critical path regardless of available cores.
Fixed in #9756, which splits the 40 instantiations into one TU per (type, redop) and disables the pathological auto-unroll. Each kernel now compiles in under 2 seconds (verified with an empty ccache), removing the serialized 48-minute bottleneck, with no correctness regressions and a net performance improvement.
PR #8443 (CE AR no pipeline) added
src/device/ce_reduce.cccontaining all 40(type, redop)kernel instantiations ofncclCeLocalReduceKernelVecin a single translation unit. LLVM's runtime loop-unroller auto-unrolled the rank-reduction loop (nRanksis a runtime value, not a compile-time constant) by up to 64x on top of the existing manual unroll/vectorization, producing ~900 PHI nodes and ~150shufflevectorops per kernel forint8_t/uint8_tMin/Max alone. This single translation unit became one unparallelizable compile job that took ~48 minutes by itself, dominating the whole build's critical path regardless of available cores.Fixed in #9756, which splits the 40 instantiations into one TU per
(type, redop)and disables the pathological auto-unroll. Each kernel now compiles in under 2 seconds (verified with an empty ccache), removing the serialized 48-minute bottleneck, with no correctness regressions and a net performance improvement.