Support: persist scene-test kernel compilation cache

[doraemonmj]

Summary

• persist content-addressed ChipCallable artifacts under build/cache/kernels, with source/include-closure, compiler identity/flags/schema, and ABI invalidation, plus integrity validation, atomic publication, and per-key locking
• add a collection-only scene-test compile command that defaults to serial compilation; simpler CI explicitly uses --compile-workers 8 before task-submit without acquiring an NPU
• restore and save kernel artifacts with actions/cache, while keeping the existing batch-level device acquisition and separate SDMA execution
• honor active pytest skip conditions and cover cache reuse, invalidation, corruption, concurrency, and configurable warm-up parallelism

Lock-boundary measurement

qwen3_14b_decode::StressBatch16Seq3500 on a2a3, with both device jobs acquiring immediately:

Path	Lock-free warm-up	task-submit wall	Total wall
baseline: full pytest under lock	—	131.62 s	131.62 s
current: cold cache	80.16 s	54.20 s	134.36 s
current: warm cache	17.23 s	54.20 s	71.43 s

The cold-cache path moves compilation out of the device allocation rather than shortening first-run wall time. Device-lock wall falls by 77.42 s (58.8%); restored caches also reduce total wall time.

Fixture construction and golden computation remain inside the locked pytest invocation and are intentionally outside this PR's scope.

Testing

• full Python UT: 1076 passed, 19 skipped, 1 xfailed
• focused cache/warm-up UT after rebase: 15 passed
• local a2a3sim cold-cache warm-up repeated successfully; a full eight-class run observed eight concurrent compiler child processes
• real CLI validation covered --compile-workers 8 and rejection of zero workers
• all commit hooks passed: headers, English-only, YAML, Markdown, Ruff, and Pyright
• a2a3 onboard qwen3_14b_decode::StressBatch16Seq3500: baseline, cold warm-up, cached device run, and hot warm-up all passed

Close #1604

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds persistent, content-sensitive caching for compiled scene-test ChipCallable objects. It integrates cache-aware compilation, adds a pytest collection warm-up tool, and runs warm-up commands with cache restoration in hardware CI.

Changes

Scene-Test Compilation Cache

Layer / File(s)	Summary
Cache keys, validation, and locking simpler_setup/scene_test_cache.py, tests/ut/py/test_scene_test_cache.py	The cache fingerprints ABI data, metadata, source contents, and include dependencies. It validates artifacts, publishes them atomically, and prevents duplicate concurrent compilation.
Compiler input and ChipCallable integration simpler_setup/kernel_compiler.py, simpler_setup/scene_test.py	Compilation now collects orchestration and in-core inputs and retrieves compiled artifacts through the persistent cache.
Collection tool, CI wiring, and documentation simpler_setup/tools/scene_test_compile.py, simpler_setup/tools/README.md, .github/workflows/_st-npu.yml, docs/ci.md, docs/testing.md, docs/investigations/..., tests/ut/py/test_scene_test_compile.py	The new tool compiles selected collected classes without device initialization. CI restores the kernel cache and warms selected scene-test batches before pytest. Documentation describes the workflow and lock boundary.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CI as Hardware CI
  participant Pytest as pytest
  participant Warmup as scene_test_compile
  participant Cache as scene_test_cache
  participant Compiler as scene_test.py
  CI->>CI: Restore build/cache/kernels
  CI->>Warmup: Run selected tests with --collect-only
  Warmup->>Pytest: Collect scene-test items
  Pytest-->>Warmup: Return collected classes
  Warmup->>Compiler: Compile selected classes
  Compiler->>Cache: Load or compile ChipCallable
  Cache-->>Compiler: Return cached artifact
  CI->>Pytest: Execute hardware tests
  Pytest->>Cache: Reuse compiled artifacts

Loading

Possibly related issues

• [Code Health] Scene tests hold an NPU lock through their CPU phases, and recompile every kernel on every CI run #1604 — Directly covers persistent scene-test compilation caching and lock-free kernel warm-up.

Possibly related PRs

• hw-native-sys/simpler#1418 — Modifies scene-test and kernel compilation caching in the same compiler integration areas.
• hw-native-sys/simpler#1590 — Shares scene-test CI workflow and docs/ci.md changes.
• hw-native-sys/simpler#1609 — Shares scene-test CI selection and hardware test command changes.

Poem

A rabbit warms kernels before the run,
Cache keys bloom beneath the sun.
Headers change, stale blobs flee,
Locks guard one compile efficiently.
pytest hops on, artifacts stay—
Faster trails through tests today.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 17.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely identifies the primary change: persistent scene-test kernel compilation caching.
Description check	✅ Passed	The description directly covers persistent caching, CI warm-up, lock boundaries, testing, and documentation changes in the pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@simpler_setup/scene_test.py`:
- Around line 1032-1055: The artifact metadata built in the compile_artifact_key
call must include a compiler cache token covering the selected compiler version,
relevant compile/link flags, and a schema token for KernelCompiler behavior.
Update the surrounding artifact-key generation to use this token, and add a
regression test proving that changing the schema token invalidates the artifact
key.

In `@simpler_setup/tools/scene_test_compile.py`:
- Around line 24-25: Update the warm-up filtering around item.iter_markers() to
evaluate active pytest skipif conditions, not only markers named "skip", and
skip compilation whenever pytest would skip the collected item. Add a regression
test covering a truthy pytest.mark.skipif condition while preserving compilation
for inactive conditions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bdf8b3c9-b100-41e8-8b49-59279b55f34a

📥 Commits

Reviewing files that changed from the base of the PR and between a032884 and ae6e02a.

📒 Files selected for processing (11)

• .github/workflows/_st-npu.yml
• docs/ci.md
• docs/investigations/2026-07-qwen-scene-test-406s-decomposition.md
• docs/testing.md
• simpler_setup/kernel_compiler.py
• simpler_setup/scene_test.py
• simpler_setup/scene_test_cache.py
• simpler_setup/tools/README.md
• simpler_setup/tools/scene_test_compile.py
• tests/ut/py/test_scene_test_cache.py
• tests/ut/py/test_scene_test_compile.py

[ChaoZheng109]

Rebased onto upstream/main (7fd34fe9) and pushed a follow-up commit a1dcb88d addressing the review findings. The rebase conflict was in tests/ut/py/test_scene_test_cache.py against #1450 (module-qualified cache keys) and was resolved as a union — l3_compile_cache_key's new module argument and test_compile_cache_keys_include_module are preserved alongside the cache tests.

必须处理

1. Warm-up 失败不再阻断整个 onboard job. list(executor.map(...)) 在第一个异常处抛出 → pytest_collection_finish INTERNALERROR → step 非零退出 → 后面所有 pytest / SDMA / dfx step 全部被跳过。改动前一个 kernel 编译不过只失败一个用例，改动后会让整个 job 零信号。现在 compile_collected_scene_tests 返回 (count, failures)，逐个上报，把归因留给加锁的 pytest。
2. 不再依赖手工 bump _COMPILE_CACHE_SCHEMA. key 里新增了 kernel_compiler.py / toolchain.py / elf_parser.py 的内容摘要 —— 改 extract_text_section、_compile_incore_sim、link/编译逻辑现在会自动失效缓存。常量保留给唯一没人覆盖的输入（scene_test 把二进制装配成 ChipCallable 的那一步），并在注释里写明了这一点。
3. 文档与实现对齐. warm-up 只接进了 _st-npu-a2a3.yml / _st-npu-a5.yml，docs/testing.md 的 "Linux and macOS" 与 docs/ci.md 的 "every runner architecture" 都不成立（sim job 在 ubuntu/macos 上完全没有 warm-up step）。已改为点名两个 onboard job，并说明 sim 每次冷编译。
4. 缓存不再无限增长. 内容寻址意味着旧 entry 永不覆盖；叠加 actions/cache 每次 push 存一份新 entry + 前缀恢复，会挤占仓库 10 GB 配额并淘汰其它 job 的 pip/cmake 缓存。现在每次运行 prune 掉 14 天未使用的 entry，命中即刷新 mtime，lock 文件挪到 .locks/ 走同一把尺子。

建议修复

5. evaluate_skip_marks 是 pytest 私有 API，import 失败现在降级为"照常编译"，不会同时打挂 CLI 和引用它的 UT。
6. build/ 不可写（wheel 装进只读 site-packages）时降级为进程内直接编译并告警，而不是从 mkdir 抛 PermissionError 打挂原本能过的 scene test。
7. _orchestration_link_flags() 内部已经判过 sys.platform，两个互斥的外层 guard 合成一次无条件 extend；build-id 的不变量注释移到函数上。
8. _source_closure 补了 docstring：闭包只含能在 -I 列表内解析到的头，ccec 内置 / CANN 头不在 key 里（由编译器 --version identity 覆盖），#if 不求值属于过近似。

可以考虑

9. per-file 摘要与 include 指令按 (path, mtime_ns, size) memoize —— 一个 scene test 的各编译单元共享大部分闭包，之前 pto-isa 头每个单元都要重读重 hash 一遍。
10. key 含 owning class 的限定名（所以并非严格 content-addressed，同源两个 scene test 各存一份），已在 docs/ci.md 写明。
11. --compile-workers N 每个 worker 一个编译器进程 —— 文档写明 CI 的 8 是"runner CPU 独占"前提，本地别照抄。
12. sim job 无 warm-up、无 cache 恢复，每次冷编译 —— 与 [Code Health] Scene tests hold an NPU lock through their CPU phases, and recompile every kernel on every CI run #1604 的原始判断一致，已在 docs/ci.md 写明而不是留给读者猜。

新增测试 (test_scene_test_cache.py / test_scene_test_compile.py)：compile 失败上报而不抛、缺少 pytest 私有 helper 时的降级、logic token 覆盖三个模块、过期 entry 与 lock 被 prune、命中刷新 mtime、只读目录降级、per-file 摘要 memo 生效。

验证：tests/ut/py 1122 passed / 19 skipped；ruff check + ruff format + 全部 pre-commit hook（headers / english-only / markdownlint / pyright）通过；a2a3sim 上 vector_example 冷→热跑通并命中缓存（warm-up 14.6 s → 5.5 s）。

一个无关的既有缺陷（不在本 PR 范围，已单独记录）：python/simpler/worker.py:3776-3792 的 _forked_child_main 只把 setup() 包进 try/except _StartupCancelled，却在该块之后才恢复 SIGTERM handler。信号落在这个窗口里时 _StartupCancelled 从恢复那一行抛出，绕过所有 handler，unwind 进父进程 _start_hierarchical 的 fork 副本 —— 正是该函数 docstring 里 "load-bearing invariant" 明令禁止的。表现为 TestNextLevelStartupFailure::test_second_child_failure_reaps_first 在满负载全量跑时约 1/3 概率失败（单跑必过），且 forked child 会继续打印自己的 pytest summary。与本 PR 无关，本 PR 只是改变了调度时序更容易撞上。