Skip to content

Commit be8cc87

Browse files
mvanhornclaude
andcommitted
gh-142041: Improve concurrent.interpreters documentation
- Add complete Queue method documentation (put, get, empty, full, qsize, maxsize, put_nowait, get_nowait) with parameter descriptions - Expand prepare_main docs explaining its purpose with an example - Fix exec() signature (remove incorrect dedent parameter) - Add detailed exec() and call() descriptions with error handling info - Expand call_in_thread docs explaining thread behavior - Document create_queue parameters (maxsize, unbounditems) - Add is_shareable() function documentation - Rewrite Basic usage section with clearer examples for exec, call, prepare_main, queues, and parallel execution patterns - Add tip pointing to InterpreterPoolExecutor for simpler use cases Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 0156133 commit be8cc87

File tree

2 files changed

+208
-33
lines changed

2 files changed

+208
-33
lines changed

Doc/library/concurrent.interpreters.rst

Lines changed: 202 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -224,11 +224,25 @@ This module defines the following functions:
224224
Initialize a new (idle) Python interpreter
225225
and return a :class:`Interpreter` object for it.
226226

227-
.. function:: create_queue()
227+
.. function:: create_queue(maxsize=0, *, unbounditems=UNBOUND)
228228

229229
Initialize a new cross-interpreter queue and return a :class:`Queue`
230230
object for it.
231231

232+
*maxsize* sets the upper bound on the number of items that can be placed
233+
in the queue. If *maxsize* is less than or equal to zero, the queue
234+
size is infinite.
235+
236+
*unbounditems* sets the default behavior when getting an item from the
237+
queue whose original interpreter has been destroyed.
238+
See :meth:`Queue.put` for supported values.
239+
240+
.. function:: is_shareable(obj)
241+
242+
Return ``True`` if the object can be sent to another interpreter
243+
without using :mod:`pickle`, and ``False`` otherwise.
244+
See :ref:`interp-object-sharing`.
245+
232246

233247
Interpreter objects
234248
^^^^^^^^^^^^^^^^^^^
@@ -261,27 +275,71 @@ Interpreter objects
261275

262276
Finalize and destroy the interpreter.
263277

264-
.. method:: prepare_main(ns=None, **kwargs)
278+
.. method:: prepare_main(ns=None, /, **kwargs)
279+
280+
Bind the given objects into the interpreter's :mod:`!__main__`
281+
module namespace. This is the primary way to pass data to code
282+
running in another interpreter.
283+
284+
*ns* is an optional :class:`dict` mapping names to values.
285+
Any additional keyword arguments are also bound as names.
286+
287+
The values must be shareable between interpreters. Some objects
288+
are actually shared, some are copied efficiently, and most are
289+
copied via :mod:`pickle`. See :ref:`interp-object-sharing`.
265290

266-
Bind objects in the interpreter's :mod:`!__main__` module.
291+
For example::
267292

268-
Some objects are actually shared and some are copied efficiently,
269-
but most are copied via :mod:`pickle`. See :ref:`interp-object-sharing`.
293+
interp = interpreters.create()
294+
interp.prepare_main(name='world')
295+
interp.exec('print(f"Hello, {name}!")')
270296

271-
.. method:: exec(code, /, dedent=True)
297+
This is equivalent to setting variables in the interpreter's
298+
:mod:`!__main__` module before calling :meth:`exec` or :meth:`call`.
299+
The names are available as global variables in the executed code.
300+
301+
.. method:: exec(code, /)
272302

273303
Run the given source code in the interpreter (in the current thread).
274304

305+
*code* is a :class:`str` of Python source code. It is executed as
306+
though it were the body of a script, using the interpreter's
307+
:mod:`!__main__` module as the globals namespace.
308+
309+
There is no return value. To get a result back, use :meth:`call`
310+
instead, or communicate through a :class:`Queue`.
311+
312+
If the code raises an unhandled exception, an :exc:`ExecutionFailed`
313+
exception is raised in the calling interpreter. The actual
314+
exception object is not preserved because objects cannot be shared
315+
between interpreters directly.
316+
317+
This blocks the current thread until the code finishes.
318+
275319
.. method:: call(callable, /, *args, **kwargs)
276320

277-
Return the result of calling running the given function in the
278-
interpreter (in the current thread).
321+
Call *callable* in the interpreter (in the current thread) and
322+
return the result.
323+
324+
Nearly all callables, args, kwargs, and return values are supported.
325+
All "shareable" objects are supported, as are "stateless" functions
326+
(meaning non-closures that do not use any globals). For other
327+
objects, this method falls back to :mod:`pickle`.
328+
329+
If the callable raises an exception, an :exc:`ExecutionFailed`
330+
exception is raised in the calling interpreter.
279331

280332
.. _interp-call-in-thread:
281333

282334
.. method:: call_in_thread(callable, /, *args, **kwargs)
283335

284-
Run the given function in the interpreter (in a new thread).
336+
Start a new :class:`~threading.Thread` that calls *callable* in
337+
the interpreter and return the thread object.
338+
339+
This is a convenience wrapper that combines
340+
:mod:`threading` with :meth:`call`. The thread is started
341+
immediately. Call :meth:`~threading.Thread.join` on the returned
342+
thread to wait for it to finish.
285343

286344
Exceptions
287345
^^^^^^^^^^
@@ -318,19 +376,86 @@ Communicating Between Interpreters
318376

319377
.. class:: Queue(id)
320378

321-
A wrapper around a low-level, cross-interpreter queue, which
322-
implements the :class:`queue.Queue` interface. The underlying queue
323-
can only be created through :func:`create_queue`.
379+
A cross-interpreter queue that can be used to pass data safely
380+
between interpreters. It provides the same interface as
381+
:class:`queue.Queue`. The underlying queue can only be created
382+
through :func:`create_queue`.
383+
384+
When an object is placed in the queue, it is prepared for use in
385+
another interpreter. Some objects are actually shared and some are
386+
copied efficiently, but most are copied via :mod:`pickle`.
387+
See :ref:`interp-object-sharing`.
324388

325-
Some objects are actually shared and some are copied efficiently,
326-
but most are copied via :mod:`pickle`. See :ref:`interp-object-sharing`.
389+
:class:`Queue` objects themselves are shareable between interpreters
390+
(they reference the same underlying queue), making them suitable for
391+
use with :meth:`Interpreter.prepare_main`.
327392

328393
.. attribute:: id
329394

330395
(read-only)
331396

332397
The queue's ID.
333398

399+
.. attribute:: maxsize
400+
401+
(read-only)
402+
403+
The maximum number of items allowed in the queue. A value of zero
404+
means the queue size is infinite.
405+
406+
.. method:: empty()
407+
408+
Return ``True`` if the queue is empty, ``False`` otherwise.
409+
410+
.. method:: full()
411+
412+
Return ``True`` if the queue is full, ``False`` otherwise.
413+
414+
.. method:: qsize()
415+
416+
Return the number of items in the queue.
417+
418+
.. method:: put(obj, block=True, timeout=None, *, unbounditems=None)
419+
420+
Put *obj* into the queue. If *block* is true (the default),
421+
block if necessary until a free slot is available. If *timeout*
422+
is a positive number, block at most *timeout* seconds and raise
423+
:exc:`QueueFullError` if no free slot is available within that time.
424+
425+
If *block* is false, put *obj* in the queue if a free slot is
426+
immediately available, otherwise raise :exc:`QueueFullError`.
427+
428+
*unbounditems* controls what happens when the item is retrieved
429+
via :meth:`get` after the interpreter that called :meth:`put` has
430+
been destroyed. If ``None`` (the default), the queue's default
431+
(set via :func:`create_queue`) is used. Supported values:
432+
433+
* :data:`UNBOUND` -- :meth:`get` returns the :data:`UNBOUND`
434+
sentinel in place of the original object.
435+
* :data:`UNBOUND_ERROR` -- :meth:`get` raises
436+
:exc:`ItemInterpreterDestroyed`.
437+
* :data:`UNBOUND_REMOVE` -- the item is silently removed from
438+
the queue when the original interpreter is destroyed.
439+
440+
.. method:: put_nowait(obj, *, unbounditems=None)
441+
442+
Equivalent to ``put(obj, block=False)``.
443+
444+
.. method:: get(block=True, timeout=None)
445+
446+
Remove and return an item from the queue. If *block* is true
447+
(the default), block if necessary until an item is available.
448+
If *timeout* is a positive number, block at most *timeout* seconds
449+
and raise :exc:`QueueEmptyError` if no item is available within
450+
that time.
451+
452+
If *block* is false, return an item if one is immediately
453+
available, otherwise raise :exc:`QueueEmptyError`.
454+
455+
.. method:: get_nowait()
456+
457+
Equivalent to ``get(block=False)``.
458+
334459

335460
.. exception:: QueueEmptyError
336461

@@ -354,31 +479,75 @@ Creating an interpreter and running code in it::
354479

355480
interp = interpreters.create()
356481

357-
# Run in the current OS thread.
482+
# Run source code directly.
483+
interp.exec('print("Hello from a subinterpreter!")')
358484

359-
interp.exec('print("spam!")')
485+
# Call a function and get the result.
486+
def add(x, y):
487+
return x + y
360488

361-
interp.exec("""if True:
362-
print('spam!')
363-
""")
489+
result = interp.call(add, 3, 4)
490+
print(result) # 7
364491

365-
from textwrap import dedent
366-
interp.exec(dedent("""
367-
print('spam!')
368-
"""))
492+
# Run a function in a new thread.
493+
def worker():
494+
print('Running in a thread!')
369495

370-
def run(arg):
371-
return arg
496+
t = interp.call_in_thread(worker)
497+
t.join()
372498

373-
res = interp.call(run, 'spam!')
374-
print(res)
499+
Passing data with :meth:`~Interpreter.prepare_main`::
375500

376-
def run():
377-
print('spam!')
501+
interp = interpreters.create()
378502

379-
interp.call(run)
503+
# Bind variables into the interpreter's __main__ namespace.
504+
interp.prepare_main(greeting='Hello', name='world')
505+
interp.exec('print(f"{greeting}, {name}!")')
380506

381-
# Run in new OS thread.
507+
# Can also use a dict.
508+
config = {'host': 'localhost', 'port': 8080}
509+
interp.prepare_main(config)
510+
interp.exec('print(f"Connecting to {host}:{port}")')
382511

383-
t = interp.call_in_thread(run)
384-
t.join()
512+
Using queues to communicate between interpreters::
513+
514+
interp = interpreters.create()
515+
516+
# Create a queue and share it with the subinterpreter.
517+
queue = interpreters.create_queue()
518+
interp.prepare_main(queue=queue)
519+
520+
# The subinterpreter puts results into the queue.
521+
interp.exec("""
522+
import math
523+
queue.put(math.factorial(10))
524+
""")
525+
526+
# The main interpreter reads from the same queue.
527+
result = queue.get()
528+
print(result) # 3628800
529+
530+
Running CPU-bound work in parallel using threads and interpreters::
531+
532+
import time
533+
from concurrent import interpreters
534+
535+
def compute(n):
536+
total = sum(range(n))
537+
return total
538+
539+
interp1 = interpreters.create()
540+
interp2 = interpreters.create()
541+
542+
# Each interpreter runs in its own thread and does not share
543+
# the GIL, enabling true parallel execution.
544+
t1 = interp1.call_in_thread(compute, 50_000_000)
545+
t2 = interp2.call_in_thread(compute, 50_000_000)
546+
t1.join()
547+
t2.join()
548+
549+
.. tip::
550+
551+
For many use cases, :class:`~concurrent.futures.InterpreterPoolExecutor`
552+
provides a higher-level interface that combines threads with
553+
interpreters automatically.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Improve :mod:`concurrent.interpreters` documentation with clearer
2+
descriptions of :class:`~concurrent.interpreters.Queue` methods,
3+
:meth:`~concurrent.interpreters.Interpreter.prepare_main` usage,
4+
:meth:`~concurrent.interpreters.Interpreter.exec` parameters,
5+
and expanded usage examples. Also fix the ``exec`` signature which
6+
incorrectly listed a ``dedent`` parameter.

0 commit comments

Comments
 (0)