TUI hangs at 'Loading: N skills' on NFS/GPFS: SIGCHLD race when Tokio spawns 'which gh' subprocess with 30+ concurrent threads

[raylim]

Summary

In TUI mode on Linux with home directories on GPFS/NFS, the CLI hangs indefinitely at "Loading: N skills". The binary stops logging almost immediately after startup and never becomes responsive. The hang occurs even with no MCP servers configured.

Symptom

The log line gh CLI availability check completed never appears. MCP clients never start. The binary displays "Loading: N skills" forever.

Hanging run (GPFS, heavy concurrent I/O):
T+0s   Starting Copilot CLI: 1.0.69
T+∞    Loading: 1 instruction, N skills   ← stuck
       [gh CLI check callback never fires, MCP clients never start]

Working run:
T+0.09s  Registering foreground session
T+0.27s  gh CLI availability check completed
T+1.04s  Starting MCP clients...
T+5-20s  AI response received ✓

Environment

• Linux (RHEL 8), home directory on GPFS or NFS
• Hangs intermittently without any MCP configuration; hangs more reliably when command:-based MCP servers that exit immediately are configured (e.g. misconfigured MCPs that crash on startup)
• Not reproducible on macOS (Darwin uses kqueue for child-process events, not signalfd)

What is known

During startup, the binary spawns a which gh subprocess (via clone(SIGCHLD)) to check if the gh CLI is available. On GPFS systems with many concurrent I/O threads, the callback for this subprocess completion never fires. Without it, MCP clients never start and the TUI hangs.

strace -qq -e trace=none -f -o /dev/null wrapping the binary consistently and immediately fixes the hang. The -f flag (attach ptrace to all threads via PTRACE_O_TRACECLONE) is required — without it the hang persists. This strongly suggests the issue is related to how SIGCHLD is delivered across threads under heavy I/O load on Linux.

The suspected mechanism is a race in libuv's signalfd-based child-process tracking: SIGCHLD from the which gh subprocess is missed when all Tokio worker threads are busy with concurrent GPFS I/O. However, this has not been confirmed at the kernel level — the exact reason strace -f fixes it is inferred, not directly observed.

Having command:-based MCP servers that exit immediately (crashed/misconfigured MCPs) makes the hang more likely to occur, but is not required — the hang can occur with no MCP configuration at all under sufficient GPFS load.

Workaround

#!/bin/bash
# ~/bin/copilot wrapper
exec strace -qq -e trace=none -f -o /dev/null /path/to/copilot-linux-x64/copilot "$@"

Zero syscall-tracing overhead. The -f flag is required.

Suggested fix

1. Add a timeout/fallback for the gh CLI check — if the subprocess does not complete within ~500ms, assume gh is unavailable and continue startup. This would prevent the hang regardless of the signal delivery issue.
2. Decouple MCP client startup from the gh CLI check — MCP initialization should not be gated on an optional subprocess check. Starting MCP clients unconditionally would eliminate the hang.
3. Investigate libuv signalfd behavior on Linux under high thread/I/O load — specifically whether SIGCHLD from short-lived subprocesses can be silently dropped when all threads are busy.

Option 1 or 2 would be the most impactful: a hung subprocess check should never be able to prevent the TUI from becoming usable.

Version

• Copilot CLI: 1.0.69
• OS: Linux (RHEL 8), home directory on GPFS
• Not reproducible on macOS

[raylim]

Investigation Update: Revised Root Cause

After extensive testing (20+ controlled experiments with isolated LD_PRELOAD shims and strace), the root cause is different from what I previously described. The fsync/auto-update hypothesis was wrong.

True Root Cause

SIGCHLD delivery race condition on multi-threaded GPFS deployments

What actually happens

1. During startup, the binary spawns a which gh subprocess (via clone(SIGCHLD)) from a Tokio worker thread to check if the gh CLI is available
2. The subprocess exits quickly (~7ms), sending SIGCHLD to the process
3. On systems with 30+ concurrent I/O threads hammering a GPFS filesystem, there is a race condition where Node.js/libuv's SIGCHLD handler does not receive the child exit event
4. The gh CLI availability check callback never fires
5. Without the gh CLI check completing, MCP clients never start (this is the gate that triggers MCP initialization)
6. The binary displays "Loading: 1 instruction, 34 skills" indefinitely

Normal startup (working):
T+0.09s  Registering foreground session (SQLite write)
T+0.27s  gh CLI availability check completed  ← gh subprocess exits, SIGCHLD handled
T+1.04s  Starting MCP clients (github-mcp-server, etc.)
T+5-20s  AI response received ✓

Broken startup (GPFS, 30+ threads):
T+0.09s  Registering foreground session
T+0.20s  Login status unknown
         [gh subprocess spawned but SIGCHLD never delivered to libuv]
T+5.4s   resolve repo IDs failed (background check)
T+∞      Loading: 1 instruction, 34 skills  ← stuck, no MCP, no AI ✗

Why it's environment-specific

The race is consistently triggered when:

• Home directory on GPFS (many threads doing concurrent GPFS metadata operations)
• The Rust/Tokio runtime has 30+ worker threads all making concurrent GPFS I/O
• The subprocess's SIGCHLD is delivered during a window when libuv's epoll/signalfd cannot process it

Diagnostic evidence

strace -e trace=none -f -o /dev/null (minimal ptrace attachment, zero system-call tracing overhead) immediately and consistently fixes the hang by controlling SIGCHLD delivery timing through the ptrace mechanism.

Without -f (main process only): still hangs. Requires tracing all threads to fix.

Workaround

# Minimal: run under strace with no syscall overhead
exec strace -qq -e trace=none -f -o /dev/null /path/to/copilot "$@"

Suggested Fix

The gh CLI availability check should be made robust against SIGCHLD delivery failures:

1. Add a timeout with retry for the subprocess-based gh check
2. Check gh availability using access(path, X_OK) instead of spawning a subprocess
3. Move MCP client startup earlier, independent of the gh CLI check (preferred)

Option 3 seems most impactful: the gh CLI check is for an optimization, not core functionality. MCP clients (especially github-mcp-server) should start regardless.

[gokhanarkan]

Triage summary

Current code confirms src/runtime/src/agents/util.rs launches which or where.exe and waits without a timeout. The exact GPFS SIGCHLD theory is not proven, but the immediate defect is clear.

Use bounded in-process PATH resolution and ensure MCP startup continues with an actionable unavailable result.

Route: agent-ready.

This triage summary was prepared with GitHub Copilot and reviewed before posting.