fix(defer): make shared EventLoopThread teardown safe for siblings and re-entrant kills

[LHMQ878]

Description

EventLoopThread instances are cached per thread name, but DeferredTask.kill(terminate_thread=True) tore the loop down as if it owned it. Two failures followed, both on the paths that chat deletion and scheduler task deletion take.

Fixes #1793

Based on ready, following the branch recent merged PRs use.

Root cause

1. Killing one task cancelled every sibling on the same name. The drain in kill() cancels all pending tasks on the loop, and TaskScheduler gives every scheduled task one fixed name (helpers/task_scheduler.py:1013), so deleting one running task cancelled all of them. Reached from api/scheduler_task_delete.py:37, api/chat_remove.py:12 and api/chat_reset.py:13, all of which pass terminate_thread=True. THREAD_BACKGROUND is shared process-wide the same way.

The failure is silent: the cancellation surfaces only on the abandoned task's own future, which nobody holds, so there is no log line — the sibling task simply stops.

2. kill(terminate_thread=True) deadlocked permanently when it ran on the loop's own thread. _on_task_done is a concurrent.futures done-callback, so it runs on the loop thread, and it kills children. A child sharing the parent's thread name therefore reached

cleanup_future = asyncio.run_coroutine_threadsafe(self._drain_event_loop_tasks(), loop)
cleanup_future.result()          # no timeout

from inside the loop, waiting on a coroutine only that thread could advance. terminate() had the mirror problem in its unbounded thread.join(), and its loop.close() would raise on a loop that is still running. The stack of the wedged thread in #1793 shows it stopped at defer.py:176 and never moved; every other task on that loop is stuck with it, and the thread being a daemon means no shutdown error surfaces either.

The change

Refcount the shared thread. Each DeferredTask registers with acquire() on construction and drops its claim in kill(). release(terminate) reports whether the caller may tear the loop down, and only the last user may:

def release(self, terminate: bool) -> bool:
    with self.__class__._lock:
        if self._users > 0:
            self._users -= 1
        if not (terminate and self._users == 0):
            return False
        if self.__class__._instances.get(self.thread_name) is self:
            del self.__class__._instances[self.thread_name]
        return True

Unregistering happens under the same lock that makes the decision, so a task constructed after this point gets a fresh thread rather than one on its way out.

Never wait on the loop from inside it. The in-loop path chains the drain and the stop onto the loop instead:

if event_loop_thread.thread is threading.current_thread():
    async def drain_then_terminate() -> None:
        try:
            await self._drain_event_loop_tasks()
        finally:
            event_loop_thread.terminate()

    loop.create_task(drain_then_terminate())
    return

Chained rather than scheduled independently, so the stop cannot land before the drain finishes. terminate() on its own thread likewise only requests the stop; the loop is closed from inside _run_event_loop's finally, since close() raises on a running loop and the in-loop caller cannot join first.

Bound the off-thread waits with DRAIN_TIMEOUT. kill(terminate_thread=True) runs on request threads, so a task that swallows cancellation must not hang chat deletion. The previous .result() and join() had no bound at all.

Two details that are consequences of the above rather than separate choices:

• _start treats a closed loop or a dead thread as absent. The in-loop teardown path cannot null out loop/thread — it is running inside a callback the loop still has to return from — so a torn-down instance keeps them attached, and a later task on the same name would be handed a closed loop.
• _start rebuilds the loop and thread together, and _run_event_loop takes its loop as an argument. Replacing only the loop would leave it unattended, because the surviving thread runs the loop it was handed.

Deliberately not changed: _drain_event_loop_tasks still cancels every task on the loop. That is correct once the loop provably has one user, which the refcount is what establishes.

Relationship to #1781

Complementary, different files, no conflict. #1781 moves parallel_tools off the shared THREAD_BACKGROUND onto per-job names, which avoids the shared-thread problem for that one caller by not sharing. It does not touch defer.py, so TaskScheduler, THREAD_BACKGROUND, the settings and plugin callers and the memorize extensions all still share a name. #1781 also adds a kill(terminate_thread=True) on timeout, which makes reaching failure mode 2 more likely rather than less. Either can merge first.

Tests

tests/test_defer_lifecycle.py covered the single-task lifecycle only, and every existing test uses a fresh uuid-suffixed thread name — so nothing exercised two tasks on one name, which is why neither failure was caught. Seven tests added:

test	asserts
test_killing_one_task_does_not_cancel_siblings_on_the_shared_thread	failure 1: a sibling on the same name runs to completion after another task's kill(terminate_thread=True)
test_killing_from_the_loops_own_thread_does_not_deadlock	failure 2: the loop is still responsive after a done-callback killed a child sharing its thread
test_last_task_to_be_killed_still_tears_the_thread_down	deferring teardown to the last user does not become never tearing down
test_teardown_requested_from_inside_the_loop_still_completes	the in-loop path closes the loop, stops the thread, and leaves the name reusable
test_a_restarted_task_keeps_its_claim_on_the_thread	restart() goes through kill(), so it must re-register or a sibling's later kill stops the loop under it
test_killing_the_same_task_twice_does_not_release_the_thread_twice	__del__ also calls kill(), and close_runtime_sync kills in a finally after a possibly-already-killed task
test_a_new_task_after_teardown_gets_a_working_thread	a guard on the fix, not the bug — see below

Three things worth flagging for review:

• test_killing_one_task_does_not_cancel_siblings_on_the_shared_thread asserts on the survivor's own result, not on a progress counter. A cancelled task raises there, whereas asserting that the counter advanced could pass on work completed before the kill landed.
• test_killing_from_the_loops_own_thread_does_not_deadlock probes by scheduling onto the loop, not by elapsed time. The parent's result arrives before its done-callback runs, so a wedge introduced by that callback would otherwise go unnoticed.
• test_a_new_task_after_teardown_gets_a_working_thread passes on the unfixed code too, and the docstring says so. It guards the risk the fix introduces — teardown now leaves a closed loop attached in the in-loop case — rather than the original bug. An earlier draft of test_teardown_requested_from_inside_the_loop_still_completes also passed against unfixed source, because it reached the in-loop branch only by winning a race with the done-callback; it now invokes kill() on the loop directly.

Control experiment

Reverting only helpers/defer.py and keeping the tests, run per-test:

test	on unfixed source
..._does_not_cancel_siblings_on_the_shared_thread	failed
..._last_task_to_be_killed_still_tears_the_thread_down	failed
..._a_restarted_task_keeps_its_claim_on_the_thread	failed
..._killing_the_same_task_twice_does_not_release_the_thread_twice	failed
..._teardown_requested_from_inside_the_loop_still_completes	failed
..._killing_from_the_loops_own_thread_does_not_deadlock	hung until the pytest timeout fired
..._a_new_task_after_teardown_gets_a_working_thread	passed (by design, see above)
the 3 pre-existing tests	passed

Run per-test because the deadlock takes the whole file down with it — worth knowing when reviewing: a regression here stalls the run rather than reporting, so --timeout is needed. That is noted in the module's DOX.

Verification

suite	result
tests/test_defer_lifecycle.py	10 passed (3 pre-existing + 7 new), green on 5 consecutive runs
full collectible suite	941 passed, failing-test set name-identical to base: 101 names before, 101 after, no additions or removals

The 101 pre-existing failures and 12 collection errors are this Windows environment: POSIX-only modules (fcntl, resource), symlink and permission semantics, absent optional dependencies and API keys. Verified by running the same suite on the base commit and diffing failing test names rather than counts — counts alone shift when tests are added to a file.

helpers/defer.py.dox.md updated per the helpers/ DOX contract: the new public methods, the shared-thread and re-entrancy contracts, and the timeout note for running the tests.