VMR Source-Only Validation hangs: SB_Fedora43_Offline_PreviousSourceBuiltSdk_Validation_x64 scenario-tests times out at 4h

[nagilson]

Summary

The VMR_SourceOnly_Validation stage of the dotnet-unified-build pipeline intermittently hangs on a single validation leg, which runs the source-built SDK against the scenario-tests. The leg produces no test output and runs until the 4-hour task timeout, failing the whole build.

Concrete occurrence

• Build: 3034195 — dotnet-unified-build, release/10.0.4xx (source version 9cfaa55c)
• Stage: VMR_SourceOnly_Validation (VMR Source-Only Validation)
• Hung leg: SB_Fedora43_Offline_PreviousSourceBuiltSdk_Validation_x64
• Symptoms:
  • Warning: No test result files matching '[ 'artifacts/TestResults/**/scenario-tests/*.xml' ]' were found.
  • Error: The task has timed out. (task ran the full 4h with no .trx produced)
• Exactly one leg hung; all other distro validation legs (AlmaLinux8, Alpine323, CentOSStream10, Ubuntu2404, Ubuntu2404 arm64) in the same stage completed.

Signature / suspected cause

A single-leg, no-output, full-timeout failure with no test assertion error is the classic signature of an infra / network-isolation stall (e.g. a scenario-test process blocking on a network call that offline network isolation neither serves nor fails fast) rather than a genuine test failure. The offline PreviousSourceBuiltSdk validation legs appear most susceptible.

Similar timeouts have been observed on the Fedora43 offline validation legs in other recent unified-build runs (across 3xx/4xx and internal branches).

Actions taken

• Retried the VMR_SourceOnly_Validation stage on build 3034195 to confirm whether it is transient.

Next steps

• Confirm transient vs. persistent from the retry.
• If it recurs, pull the SB_Fedora43_Offline_PreviousSourceBuiltSdk_Validation_x64 scenario-tests task log and identify the stuck step (likely a hung child process). Consider adding a per-test/inner timeout so the leg fails fast with diagnostics instead of consuming the full 4h stage budget.

Filed from source-build build-duty triage.

[mthalman]

It looks like the retry got the same failure. I've encountered this as well and found that a retry did succeed. So it's intermittent but seems common enough to be investigated further to see what's causing this.

[nagilson]

Diagnosis synthesis (build-duty follow-up)

Pulling together the signals gathered on this hang:

Signature (unchanged, now corroborated): single-leg, no-output, full-timeout with no .trx and no assertion error — the classic signature of an infra / network-isolation stall during scenario-test execution (a test process blocking on something that offline network isolation neither serves nor fails fast), not a genuine test failure.

Evidence:

1. Not machine flakiness in the random sense — reproduced 2/2 on the same source (build 3034195, incl. an explicit stage retry). @mthalman independently confirms it's intermittent but common, and that a later retry has succeeded for him — so it's a probabilistic stall, not deterministic.
2. Environment-specific, not a universal test bug — isolated to the Fedora43 Offline legs while AlmaLinux8 / Alpine323 / CentOSStream10 / Ubuntu2404 (x64+arm64) run the same scenario-tests against the same source-built SDK and pass.
3. Offline PreviousSourceBuiltSdk variant is the most susceptible; recurs cross-branch (4xx + 3xx and internal branches).
4. Not a recent image regression — Fedora was bumped 41→43 back in Implement Vertical Build for Browser-Wasm Multithreaded Mono #3836 (merged 2025-12-15), so the image has been baking ~7 months. This points to an intermittent Fedora43 + offline-isolation interaction rather than a fresh breakage.

Ranked hypotheses:

• (H1, most likely) Fedora43-specific network/DNS stall under offline isolation — a scenario test spawns a child process that makes a network/DNS call which, under Fedora43's newer glibc/systemd-resolved/openssl stack, blocks indefinitely instead of failing fast. Explains: Fedora-only, offline-only, no output, full-timeout, intermittent.
• (H2) Hung child process / no inner timeout — regardless of root cause, the leg has no per-test timeout, so any stuck child consumes the entire 4h task budget with zero diagnostics.
• (H3) Genuine test bug — effectively ruled out: other distros pass the identical suite.

Recommended next steps (in priority order):

1. Add a per-scenario / inner timeout to the scenario-tests invocation so a stuck test fails fast and emits diagnostics (partial .trx, last-started test name) instead of a blind 4h task timeout. This is the single highest-value change — it converts every future occurrence into an actionable log and reclaims ~4h of pipeline budget.
2. With that in place, capture which scenario test is last-started on a Fedora43 offline leg — that names the culprit. (Currently unobtainable: the hung leg produces no .trx and the internal build timeline/log isn't reachable via the anonymous Helix tooling; a team member with a dnceng PAT could pull the leg's scenario-tests console log directly.)
3. If confirmed as a DNS/network stall, mitigate by making the offending scenario test honor offline mode (skip or short-circuit the network call) on the source-only validation path.

Filed from source-build build-duty triage (continuation of the crashed session).

[nagilson]

Root cause identified — concrete fix

I traced the hang into the scenario-tests harness and it's not a Fedora-specific test bug per se — it's a generic process-teardown deadlock in the shared runner that Fedora43 just exposes most often. Here's the mechanism.

The deadlock

The suite that produces artifacts/TestResults/**/scenario-tests/*.xml runs web/run/watch templates (dotnet run on ASP.NET apps — note ExecuteHelper.ValidateExitCode even special-cases Microsoft.AspNetCore.Connections.AddressInUseException). Every process-teardown path uses the same anti-pattern:

src/Microsoft.DotNet.ScenarioTests.Common/ExecuteHelper.cs:

process.WaitForExit(millisecondTimeout);   // bounded
if (!process.HasExited)
{
    process.Kill(true);
    process.WaitForExit();                 // ← UNBOUNDED
    ...
}

and again in SdkTemplateTests/DotNetSdkHelper.cs (ExecuteRunWeb, ExecuteRunUIApp):

process.Kill(true);
process.WaitForExit();                      // ← UNBOUNDED

Because stdout/stderr are redirected and read asynchronously (BeginOutputReadLine), the parameterless WaitForExit() blocks until the async readers hit EOF on the pipes, not merely until the process exits. dotnet run launches the web app as a child/grandchild that inherits those pipe write handles. Kill(true) signals the process tree known at that instant, but if any descendant survives the kill for a moment (double-fork / reparent to pid 1 / tree-walk race), it keeps the pipe's write end open → EOF never arrives → WaitForExit() blocks forever → the leg burns the full 4h task budget and xUnit never writes a .trx. That exactly matches this issue's signature (single leg, no output, no .trx, full timeout).

Why Fedora43, and why intermittent

It's a race on process-tree teardown/pipe-closure timing. Fedora 43 is the newest image in the matrix (kernel + systemd + cgroup v2 reaping/reparenting differ from AlmaLinux8/CentOSStream10/Ubuntu2404), so a lingering child at the moment of WaitForExit() is most likely there. Offline PreviousSourceBuiltSdk legs force the timeout→kill path more often (the web/run dance can't complete), which is why that variant is the most frequent repro. Being a race, a retry sometimes passes (as @mthalman observed).

Concrete fix (upstream: dotnet/scenario-tests, flows to the VMR)

Bound every post-kill wait so a stuck pipe can never hang the leg — accept possibly-truncated output on the already-failing/timeout path:

process.Kill(true);
if (!process.WaitForExit(KillGraceMilliseconds))   // e.g. 30_000
{
    outputHelper.WriteLine("Process tree did not fully exit within the kill grace period; " +
        "abandoning wait to avoid hanging on inherited stdout/stderr handles.");
}

Apply to all three sites: ExecuteHelper.ExecuteProcess, DotNetSdkHelper.ExecuteRunWeb, DotNetSdkHelper.ExecuteRunUIApp. (The VMR's own copy test/TestUtilities/ExecuteHelper.cs has the identical pattern and should get the same bound.)

This guarantees the leg fails fast with diagnostics instead of consuming the 4h stage budget, and removes the deadlock rather than papering over it. A per-scenario inner timeout (already suggested above) is still worth adding as defense-in-depth, but the unbounded WaitForExit() is the actual root cause.

Continuation of build-duty triage.

[nagilson]

Root cause identified — concrete fix

I traced the hang into the scenario-tests harness and it's not a Fedora-specific test bug per se — it's a generic process-teardown deadlock in the shared runner that Fedora43 just exposes most often. Here's the mechanism.

The deadlock

The suite that produces artifacts/TestResults/**/scenario-tests/*.xml runs web/run/watch templates (dotnet run on ASP.NET apps — note ExecuteHelper.ValidateExitCode even special-cases Microsoft.AspNetCore.Connections.AddressInUseException). Every process-teardown path uses the same anti-pattern:

src/Microsoft.DotNet.ScenarioTests.Common/ExecuteHelper.cs:

process.WaitForExit(millisecondTimeout); // bounded
if (!process.HasExited)
{
process.Kill(true);
process.WaitForExit(); // ← UNBOUNDED
...
}
and again in SdkTemplateTests/DotNetSdkHelper.cs (ExecuteRunWeb, ExecuteRunUIApp):

process.Kill(true);
process.WaitForExit(); // ← UNBOUNDED
Because stdout/stderr are redirected and read asynchronously (BeginOutputReadLine), the parameterless WaitForExit() blocks until the async readers hit EOF on the pipes, not merely until the process exits. dotnet run launches the web app as a child/grandchild that inherits those pipe write handles. Kill(true) signals the process tree known at that instant, but if any descendant survives the kill for a moment (double-fork / reparent to pid 1 / tree-walk race), it keeps the pipe's write end open → EOF never arrives → WaitForExit() blocks forever → the leg burns the full 4h task budget and xUnit never writes a .trx. That exactly matches this issue's signature (single leg, no output, no .trx, full timeout).

Why Fedora43, and why intermittent

It's a race on process-tree teardown/pipe-closure timing. Fedora 43 is the newest image in the matrix (kernel + systemd + cgroup v2 reaping/reparenting differ from AlmaLinux8/CentOSStream10/Ubuntu2404), so a lingering child at the moment of WaitForExit() is most likely there. Offline PreviousSourceBuiltSdk legs force the timeout→kill path more often (the web/run dance can't complete), which is why that variant is the most frequent repro. Being a race, a retry sometimes passes (as @mthalman observed).

Concrete fix (upstream: dotnet/scenario-tests, flows to the VMR)

Bound every post-kill wait so a stuck pipe can never hang the leg — accept possibly-truncated output on the already-failing/timeout path:

process.Kill(true);
if (!process.WaitForExit(KillGraceMilliseconds)) // e.g. 30_000
{
outputHelper.WriteLine("Process tree did not fully exit within the kill grace period; " +
"abandoning wait to avoid hanging on inherited stdout/stderr handles.");
}
Apply to all three sites: ExecuteHelper.ExecuteProcess, DotNetSdkHelper.ExecuteRunWeb, DotNetSdkHelper.ExecuteRunUIApp. (The VMR's own copy test/TestUtilities/ExecuteHelper.cs has the identical pattern and should get the same bound.)

This guarantees the leg fails fast with diagnostics instead of consuming the 4h stage budget, and removes the deadlock rather than papering over it. A per-scenario inner timeout (already suggested above) is still worth adding as defense-in-depth, but the unbounded WaitForExit() is the actual root cause.

Continuation of build-duty triage.

This investigation overall seems reasonable and plausible, so I'll forward this to the owning group for scenario-tests.

[nagilson]

Fix PR opened upstream: dotnet/scenario-tests#354 — bounds the post-kill WaitForExit() at all three harness sites so a lingering child holding the stdout/stderr pipe can no longer hang the leg for the full 4h. cc @akoeplinger @ViktorHofer are reviewing there. The VMR's sibling copy test/TestUtilities/ExecuteHelper.cs still needs the same bound applied separately.