deflake ccov: Use _exit() when a child is killed by SIGUSR1 - #4251
deflake ccov: Use _exit() when a child is killed by SIGUSR1#4251rainsupreme wants to merge 1 commit into
Conversation
sigKillChildHandler called exitFromChild(), which in coverage builds (COVERAGE_TEST) calls exit() to flush gcov profile data. Running exit() from a signal handler is not async-signal-safe, and the gcov atexit dump lock-merges .gcda files shared by every process built from the same tree. In the cluster-migrateslots tests, which cancel migrations in tight loops across many parallel test servers, a killed child's exit() was observed stalling 30-90+ seconds on this contention. The parent correctly reports an active child the whole time, so subsequent migrations sit in waiting-to-snapshot until test waits expire -- the dominant residual flake in this file on the code-coverage CI job (observed at ~1.5% of soak iterations in 200-iteration gcov soaks; unaffected by raising test wait budgets 10s->30s). Killed children now skip the coverage dump via _exit(). Their coverage contribution is redundant with children that complete normally, and the kill path regains async-signal-safety. Signed-off-by: Rain Valentine <rsg000@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesChild exit handling
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## unstable #4251 +/- ##
============================================
+ Coverage 76.69% 76.86% +0.17%
============================================
Files 162 162
Lines 81461 81487 +26
============================================
+ Hits 62474 62634 +160
+ Misses 18987 18853 -134
🚀 New features to boost your workflow:
|
roshkhatri
left a comment
There was a problem hiding this comment.
LGTM, Thanks for the fix @rainsupreme
|
Thanks for the review @roshkhatri! 😁 |
Problem. cluster-migrateslots is the dominant source of coverage-run failures on the Codecov job (#4153). PR #4223 contains the blast radius of a failure; this PR removes the largest remaining cause of the failures themselves. Investigation of gcov soaks (test runs and analysis by an AI agent; runs linked below) found that every residual failing iteration shares one engine-side signature — a migration stuck in waiting-to-snapshot behind a child process that was killed long before but never exited:
Mechanism.
sigKillChildHandlercallsexitFromChild(), which under COVERAGE_TEST (set bymake gcov) callsexit()so coverage data is flushed. Two problems. First,exit()is not async-signal-safe, and this call runs inside a signal handler. Second, the gcov atexit dump writes and lock-merges every .gcda file in the build tree, which is shared by all servers and children the test harness spawns across its 16 parallel clients — and tests that cancel migrations in tight loops kill a fresh child every few hundred milliseconds, so a killed child'sexit()can stall for tens of seconds on the lock convoy. The parent correctly reports an active child throughout, the next migration legitimately cannot snapshot, and the test's wait expires regardless of its budget: harness-wide 3x wait scaling (#4153) and a controlled 10s-vs-30s A/B soak (200 iterations per arm, https://github.com/valkey-rainfall/valkey/actions/runs/29782958430) both left the failure rate unchanged. Corroborating detail: children reaped asterminated by signal 10(killed before installing the SIGUSR1 handler, so no coverage flush) never stall; only the handler/exit()path does.Fix. Killed children call
_exit()directly: the kill path regains async-signal-safety, and a killed child's coverage contribution — redundant with children that complete normally — is intentionally skipped. Non-coverage builds are behaviorally unchanged (_exit()either way).Verification
Full cluster-migrateslots file passes locally 105/105 (this file exercises the kill path heavily). Code coverage for children is now discarded but this was redundant -- coverage is exactly the same after this change except for missing
sigKillChildHandlercoverage.A/B soak on gcov-instrumented builds matching the Codecov job, 50 loops x 4 shards per arm (https://github.com/valkey-rainfall/valkey/actions/runs/29860994762): baseline (parent commit) failed in all 4 shards — one [err] plus one cascade abort each (~5.4% per iteration), with the waiting-before-snapshotting stall visible in every failure's dumped logs — while the fixed arm completed all 4 shards fully clean: 5250/5250 [ok] per shard, zero errors, zero aborts, across 200 iterations (Fisher p ≈ 0.005).