perf: parallelize tool server init and reduce LLM retry overhead

[wangbinluo]

Summary

Addresses evaluation pipeline performance bottlenecks identified in #137.

Three changes targeting the slowest parts of BC benchmark evaluation.

Closes #137

Changes

• P0: Parallelize MCP tool server initialization — manager.py get_all_tool_definitions() now uses asyncio.gather() instead of sequential for loop.
• P0: Reduce LLM retry overhead — openai_client.py base_wait_time 30s → 10s, max_retries 10 → 5. Prevents 60-90s wasted on retries.
• P1: httpx connection pooling — search_and_scrape_webpage.py reuses a shared httpx.AsyncClient instead of creating a new one per request (~346 search calls per BC task).

Benchmark Results

Tool server initialization time (3 runs on dev_wbl)

Run	dev_wbl (parallel)
Run 1 (cold start)	29.3s
Run 2	12.1s
Run 3	9.6s
Average	17.0s

Comparison with main branch (extracted from existing evaluation logs)

	main (sequential)	dev_wbl (parallel)	Speedup
Average	234s	17.0s	13.8x
Min	22s	9.6s	2.3x
Max	945s (15.8 min)	29.3s	32.3x

Note: main branch baseline was extracted from BC-ZH evaluation logs (qwen_xxg_negative_r10_new1_step50, 30 tasks). The high average (234s) includes E2B sandbox queueing under high concurrency (MAX_CONCURRENT=60). Per-server breakdown on main: tool-python avg=145s (max=631s), search avg=69s, jina avg=19s.

Test plan

• Run bench_init_time.py on dev_wbl — 17.0s avg vs main baseline 234s avg
• Run BC-ZH evaluation end-to-end and verify no regression in accuracy
• Verify LLM retry behavior works correctly with reduced parameters
• Check Serper API connection pooling doesn't cause stale connection issues

[wangbinluo]

@shawnlimn @xingxuanli Could you review this PR when you get a chance?

This PR addresses the performance bottlenecks identified in #137 with two optimizations:

1. Parallel tool server init (asyncio.gather) — measured 13.8x speedup (234s → 17s avg)
2. httpx connection pooling — reuse TCP connections for Serper API (~346 calls per BC task)

Note: PR #138 by @JasonOA888 covers only item 1; this PR is a superset with additional optimization and benchmark data.

[Vanint]

Solid PR — persistent sessions + parallel init is a big perf win, and this supersedes #138 with better exception handling. A few issues to address:

1. Subprocess leak in _connect if session init fails: aenter/aexit are called manually, bypassing async with safety. If session.aenter() or initialize() throws, client.aexit is never called and the subprocess leaks:

async def _connect(self):
    ...
    read, write = await self._client.__aenter__()
    try:
        self._session = ClientSession(read, write, sampling_callback=None)
        await self._session.__aenter__()
        await self._session.initialize()
    except Exception:
        await self._client.__aexit__(None, None, None)
        self._client = None
        raise

2. Sessions not cleaned up on abnormal exit: close_all_sessions() is only called at the end of a successful run_main_agent. If the orchestrator throws mid-execution, subprocesses are leaked. Wrap in try/finally:

try:
    # ... existing run_main_agent logic ...
finally:
    await self.main_agent_tool_manager.close_all_sessions()
    for tm in self.sub_agent_tool_managers.values():
        await tm.close_all_sessions()

3. Reconnect logic in call_tool is not concurrency-safe: If two coroutines fail concurrently, both enter close() → ensure_connected(), potentially tearing down a connection the other just established. Currently the orchestrator calls tools sequentially so this won't bite, but PersistentMCPSession looks like a reusable component — worth either adding a lock around the reconnect path or documenting the single-caller assumption.

4. Reconnect failure lacks context: If the retry in call_tool also fails, the exception propagates without any indication that a reconnect was attempted. Consider logging or wrapping the error (e.g., raise RuntimeError(f"Reconnect to {self.server_name} also failed") from e).

5. get_all_tool_definitions connects twice per server: _get_single_server_tools opens a temporary connection to list tools, then execute_tool_call creates a new persistent session. Intentional (to avoid holding long-lived connections during init)? If so, a brief comment would help. If not, could reuse the persistent session for the initial tool listing too.

6. Re: #138: This PR covers everything in #138 with better exception handling (uses zip to preserve server name, retains fallback error entries). If this lands, #138 can be closed.

Fix these problems, and this is good to merge.