Skip to content

Add nextest slow-timeout to fail hung tests loudly - #193

Open
Saurabh Singh (saurabh500) wants to merge 2 commits into
mainfrom
dev/saurabh/nextest-slow-timeout
Open

Add nextest slow-timeout to fail hung tests loudly#193
Saurabh Singh (saurabh500) wants to merge 2 commits into
mainfrom
dev/saurabh/nextest-slow-timeout

Conversation

@saurabh500

@saurabh500 Saurabh Singh (saurabh500) commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a nextest slow-timeout so a hung or spinning test is terminated and fails loudly instead of hanging cargo btest forever. We just hit a 30+ minute forever-hang because the runner has retries but no timeout.

Config change (exact TOML diff)

diff --git a/.config/nextest.toml b/.config/nextest.toml
--- a/.config/nextest.toml
+++ b/.config/nextest.toml
@@ -3,6 +3,7 @@ path = "junit.xml"
 
 [profile.ci]
 retries = 5
+slow-timeout = { period = "60s", terminate-after = 3 }
+
+[profile.default]
+slow-timeout = { period = "60s", terminate-after = 3 }
 
 # Test groups for tests that need to run sequentially

A test running longer than period (60s) is flagged SLOW; after terminate-after (3) such periods (~180s) nextest terminates it and marks it failed. Added to both [profile.ci] (used by cargo btest --profile ci) and [profile.default] (plain cargo nextest run) so local runs are protected too. Retries and all other keys are unchanged.

Condition (a): no healthy test legitimately runs past 60s ✅

Pulled the slowest passing tests from the CI JUnit timings (target/nextest/ci/junit.xml, 2237 passing tests):

Time Test
20.33s mssql-tds connection::transport::localdb::tests::test_resolve_mssqllocaldb
15.68s test_no_protocol_resolution::...test_msf_with_instance_and_explicit_port_allowed
14.72s test_multi_subnet_failover::...test_msf_connection_refused
10.70s test_multi_subnet_failover::...test_msf_invalid_host
8.51s connection_provider::...retry_with_unreachable_host

Slowest healthy test is 20.3s — comfortably under 40s, giving a ~3x margin to the 60s threshold. 60s is safe; keeping it (no bump to 90s needed).

Condition (b): real worst-case bound

[profile.ci] has retries = 5 → 6 attempts. Retries apply to any non-success outcome, and a deterministic hang is terminated after 3 × 60s = 180s (~3 min) per attempt.

Worst case = 6 attempts × 3 min = ~18 min for a deterministic hang. This is bounded and a large improvement over the previous 30+ min forever-hang, but it is honestly higher than the earlier "≤3 min" estimate (that was per-attempt, before accounting for retries).

Investigation: can we avoid retrying terminations?

Checked nextest 0.9.88 (the pinned toolchain version):

  • There is no option to distinguish a slow-timeout termination from an assertion failure for retry purposes. Retry controls are limited to count / backoff / delay and per-test-name overrides — none filter by outcome type. A hang is retried the full 5×.
  • on-timeout = "pass" (treat timeouts as success) exists only from 0.9.115, and it changes pass/fail semantics, not retry behavior.
  • On Windows, the grace-period is ignored for timeouts (immediate job-object kill), so each terminated attempt is ~180s exactly.

Conclusion: no no-retry-on-termination option is available in this version, so retries are left as-is and the ~18 min worst-case applies. A reviewer may optionally choose to lower retries for the terminated case, but that would also reduce flake-retry coverage for genuine failures, so it's left untouched here.

Validation

  • cargo bfmt — clean
  • cargo bclippy — clean (warnings-as-errors)
  • cargo nextest list --profile ci and --profile default — both exit 0 (config parses, runner accepts it). Pre-existing integration-test failures require a live SQL Server and are unrelated to this TOML-only change.

Note

TOML-only change to shared CI config. Values approved upstream: period = "60s", terminate-after = 3, mirrored into [profile.default].

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e2c378f8-3ba1-4b48-9ebe-5a4ea2bd2761
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e2c378f8-3ba1-4b48-9ebe-5a4ea2bd2761
@saurabh500
Saurabh Singh (saurabh500) marked this pull request as ready for review August 8, 2026 20:00
@saurabh500
Saurabh Singh (saurabh500) requested a review from a team as a code owner August 8, 2026 20:00
Copilot AI balanced review requested due to automatic review settings August 8, 2026 20:00
@saurabh500
Saurabh Singh (saurabh500) marked this pull request as draft August 8, 2026 20:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds bounded timeouts to prevent hung tests from blocking CI and local nextest runs indefinitely.

Changes:

  • Terminates tests after three 60-second slow periods.
  • Applies the timeout to both CI and default profiles.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@saurabh500
Saurabh Singh (saurabh500) marked this pull request as ready for review August 8, 2026 20:02
@saurabh500
Saurabh Singh (saurabh500) enabled auto-merge (squash) August 8, 2026 20:03
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

100%

🎯 Overall Coverage

91.0%

📦 Project: mssql-tds + mssql-odbc + mssql-py-core
ℹ️ Note: diff coverage is reported, not enforced.


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.


🔗 Quick Links

View Azure DevOps Build · Coverage Report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants