Skip to content

deflake ccov: Use _exit() when a child is killed by SIGUSR1 - #4251

Open
rainsupreme wants to merge 1 commit into
valkey-io:unstablefrom
valkey-rainfall:fix/coverage-child-kill-exit
Open

deflake ccov: Use _exit() when a child is killed by SIGUSR1#4251
rainsupreme wants to merge 1 commit into
valkey-io:unstablefrom
valkey-rainfall:fix/coverage-child-kill-exit

Conversation

@rainsupreme

Copy link
Copy Markdown
Contributor

Problem. cluster-migrateslots is the dominant source of coverage-run failures on the Codecov job (#4153). PR #4223 contains the blast radius of a failure; this PR removes the largest remaining cause of the failures themselves. Investigation of gcov soaks (test runs and analysis by an AI agent; runs linked below) found that every residual failing iteration shares one engine-side signature — a migration stuck in waiting-to-snapshot behind a child process that was killed long before but never exited:

22:21:13.318 * Started child process 78673 for slot migration {...}
22:21:13.322 * Slot migration target dropped, killing fork child.
22:21:13.322 * Killing running slot migration child: 78673
22:21:13.430 * Slot migration initiated through CLUSTER MIGRATESLOTS {...}
22:21:14.946 * waiting before snapshotting due to active child process.
   (repeats every 5s; 31s in this instance, 93+s observed elsewhere)
22:21:44.196 * connection lost / state transition: waiting-to-snapshot -> failed

Mechanism. sigKillChildHandler calls exitFromChild(), which under COVERAGE_TEST (set by make gcov) calls exit() so coverage data is flushed. Two problems. First, exit() is not async-signal-safe, and this call runs inside a signal handler. Second, the gcov atexit dump writes and lock-merges every .gcda file in the build tree, which is shared by all servers and children the test harness spawns across its 16 parallel clients — and tests that cancel migrations in tight loops kill a fresh child every few hundred milliseconds, so a killed child's exit() can stall for tens of seconds on the lock convoy. The parent correctly reports an active child throughout, the next migration legitimately cannot snapshot, and the test's wait expires regardless of its budget: harness-wide 3x wait scaling (#4153) and a controlled 10s-vs-30s A/B soak (200 iterations per arm, https://github.com/valkey-rainfall/valkey/actions/runs/29782958430) both left the failure rate unchanged. Corroborating detail: children reaped as terminated by signal 10 (killed before installing the SIGUSR1 handler, so no coverage flush) never stall; only the handler/exit() path does.

Fix. Killed children call _exit() directly: the kill path regains async-signal-safety, and a killed child's coverage contribution — redundant with children that complete normally — is intentionally skipped. Non-coverage builds are behaviorally unchanged (_exit() either way).

Verification

Full cluster-migrateslots file passes locally 105/105 (this file exercises the kill path heavily). Code coverage for children is now discarded but this was redundant -- coverage is exactly the same after this change except for missing sigKillChildHandler coverage.

A/B soak on gcov-instrumented builds matching the Codecov job, 50 loops x 4 shards per arm (https://github.com/valkey-rainfall/valkey/actions/runs/29860994762): baseline (parent commit) failed in all 4 shards — one [err] plus one cascade abort each (~5.4% per iteration), with the waiting-before-snapshotting stall visible in every failure's dumped logs — while the fixed arm completed all 4 shards fully clean: 5250/5250 [ok] per shard, zero errors, zero aborts, across 200 iterations (Fisher p ≈ 0.005).

sigKillChildHandler called exitFromChild(), which in coverage builds
(COVERAGE_TEST) calls exit() to flush gcov profile data. Running exit()
from a signal handler is not async-signal-safe, and the gcov atexit dump
lock-merges .gcda files shared by every process built from the same
tree. In the cluster-migrateslots tests, which cancel migrations in
tight loops across many parallel test servers, a killed child's exit()
was observed stalling 30-90+ seconds on this contention. The parent
correctly reports an active child the whole time, so subsequent
migrations sit in waiting-to-snapshot until test waits expire -- the
dominant residual flake in this file on the code-coverage CI job
(observed at ~1.5% of soak iterations in 200-iteration gcov soaks;
unaffected by raising test wait budgets 10s->30s).

Killed children now skip the coverage dump via _exit(). Their coverage
contribution is redundant with children that complete normally, and the
kill path regains async-signal-safety.

Signed-off-by: Rain Valentine <rsg000@gmail.com>
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 028af4c8-a50b-4307-98a6-27a9c56a6827

📥 Commits

Reviewing files that changed from the base of the PR and between f4dc3ca and 783e8f7.

📒 Files selected for processing (1)
  • src/server.c

📝 Walkthrough

Walkthrough

Changes

Child exit handling

Layer / File(s) Summary
Direct child exit handling
src/server.c
sigKillChildHandler now calls _exit(SERVER_CHILD_NOERROR_RETVAL) directly, with comments describing the skipped exit() and coverage atexit path; checkChildrenDone() documentation is updated accordingly.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: quanyeyang

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: switching killed children to _exit() to fix coverage-build flakiness.
Description check ✅ Passed The description accurately explains the SIGUSR1 child-exit change and its coverage/flakiness motivation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@rainsupreme
rainsupreme marked this pull request as ready for review July 21, 2026 21:29
@roshkhatri
roshkhatri self-requested a review July 21, 2026 21:46
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.86%. Comparing base (50a0397) to head (783e8f7).
⚠️ Report is 5 commits behind head on unstable.

Files with missing lines Patch % Lines
src/server.c 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #4251      +/-   ##
============================================
+ Coverage     76.69%   76.86%   +0.17%     
============================================
  Files           162      162              
  Lines         81461    81487      +26     
============================================
+ Hits          62474    62634     +160     
+ Misses        18987    18853     -134     
Files with missing lines Coverage Δ
src/server.c 89.37% <0.00%> (-0.14%) ⬇️

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread src/server.c

@roshkhatri roshkhatri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks for the fix @rainsupreme

@rainsupreme

Copy link
Copy Markdown
Contributor Author

Thanks for the review @roshkhatri! 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants