[Python SDK] Mixing threads and async code

[pocesar]

Describe the bug

The code contains a dangerous mix of threading.Lock, background threads, asyncio counterparts, requests blocking calls, dynamic lock creation and no timeout on most acquired locks.

Since most of the time you'll be using FastAPI + decorators for skill / reasoner, it'll always be a single event loop that can be blocked by blocking or nested calls.

It's very hard to debug when this happens, and some runs can stay stuck in time for hours. I recommend adding mypy and ruff for strict checks of async / blocking code

Steps to reproduce

1. Go to '...'
2. Run '...'
3. See error

Expected behavior

Screenshots / Logs

Environment

• Control plane version: 1.88
• SDK version (if applicable): 1.85
• Deployment environment (local, docker, kubernetes, etc.): Local, docker, kubernetes

Additional context

[7vignesh]

@santoshkumarradha
I'd like to pick this up, but it's a broad, systemic report rather than a single bug, so I want to align on scope before opening PRs.

Here's how I'm reading the hazards called out, and where things stand today:

#	Hazard	Status
1	Loop-bound asyncio primitives torn down from the wrong loop	Largely fixed in #799 (ResultCache / AsyncExecutionManager / ConnectionManager now use a shared loop-aware teardown helper)
2	requests blocking calls on the event-loop thread (e.g. the sync poll path in client._await_execution_sync)	Open
3	asyncio.run() invoked from inside a running loop (agent.py:1243, agent_serverless.py:151, agent.py:5040)	Open
4	Locks acquired with no timeout (~40 .acquire() / with lock sites) — the "stuck for hours" symptom	Open
5	Dynamic/lazy lock creation (client.py:424, agent.py:3558, memory_events.py:142)	Open
6	Background threads mixing with asyncio.run() (heartbeat, notes, logger, multimodal)	Open
7	Add static enforcement (ruff async ruleset + mypy) to catch blocking/async mistakes	ruff check runs in CI, but the ASYNC ruleset isn't enabled and there's no mypy

Since this is too big for one PR, my proposal is to split it into focused, independently-reviewable slices — roughly:

1. Static enforcement first — turn on ruff's ASYNC ruleset (and optionally a mypy pass over the async modules). This is the preventive fix you explicitly asked for and it surfaces the remaining hazards as lint errors, giving a concrete backlog. Non-behavioural, easy to review.
2. asyncio.run() in a running loop — guard the sync reasoner-dispatch paths so they detect a running loop and schedule onto it instead of raising RuntimeError: asyncio.run() cannot be called from a running event loop.
3. Blocking requests on the event loop — make the sync poll/execute path safe when reached from a running loop (offload to a thread executor, or route through the async client).
4. Lock hygiene — add timeouts to the long-lived lock acquisitions and remove lazy lock creation, so a wedged lock surfaces as a timeout error instead of hanging for hours.

What I'd like guidance on:

• Do you want this tracked as an epic with sub-issues (like the trigger-parity epics [Epic] [TypeScript SDK] Trigger system parity (#506) #507/[Epic] [Go SDK] Trigger system parity (#506) #508), or a series of standalone PRs referencing this issue?
• For the static-enforcement slice: are you open to enabling ruff ASYNC rules repo-wide now, or would you prefer it scoped to the async-heavy modules (client.py, async_execution_manager.py, http_connection_manager.py, memory_events.py) first to keep the initial diff small?
• Is mypy in scope for this issue, or should that be its own separate initiative?

If the split sounds right, I'm happy to start with slice 1 (ruff ASYNC enablement) since it's self-contained and directly addresses the reporter's request, then work down the list. Let me know how you'd like to scope it.

[github-actions]

Thanks for picking this up, @7vignesh! 🎉

Here's what to expect:

• We'll check in after 7 days if we haven't heard an update
• After 14 days, we'll send a second reminder
• After 21 days without activity, we'll unassign to let others contribute

Tips for success:

• Drop a comment when you start working on it
• Share any blockers - we're here to help!
• Link your PR with Fixes #620 in the description

Happy coding! 🚀

[santoshkumarradha]

Thanks @7vignesh , assigning you. @AbirAbbas could you take a look please 🙏🏻

[7vignesh]

Thanks @santoshkumarradha! 🙏

@AbirAbbas no rush on the questions above. While you take a look, I'll go ahead and start on slice 1 (turning on ruff's ASYNC rules) as a draft PR. It doesn't change any behaviour, so it's safe to start early, and it'll actually show us the full list of blocking/async spots as lint warnings which should make it easier to decide whether this needs to be an epic or just a few separate PRs.

I'll hold off on the bigger behaviour changes (the blocking calls, asyncio.run, lock timeouts) until we agree on how to split this. Will share the draft soon.