Skip to content

Parallel streams: move the results into the list without reference counting - #14

Merged
lemire merged 1 commit into
mainfrom
parallel-results
Nov 14, 2026
Merged

lemire merged 1 commit into
mainfrom
parallel-results

Conversation

@lemire

@lemire lemire commented Nov 14, 2026

Copy link
Copy Markdown
Member

After a parallel run, the calling thread copied each chunk's results into its list (one Py_NewRef per result), then dropped the chunk's list (one Py_DECREF per result). Other threads own these objects, so on free-threaded Python each change is atomic, and the calling thread did all of it alone after the workers had finished. perf showed only 41 of 64 CPUs busy on average.

Chunks now keep their results in a std::vector, and the final list takes the references (PyList_SET_ITEM without incref). On a failure, the results past the failing chunk are released as before.

map_many with two fields on 7.6 GB of GH Archive events (file mapped once), big4 (Xeon Gold 6548N), 3.14.10t:

threads before after
16 27.5 GB/s 29.7 GB/s
64 56.6 GB/s 62.9 GB/s
128 65.9 GB/s 73.7 GB/s

Full suite passes on 3.14.10t (with #13 applied).

…unting

The calling thread copied the results of each chunk into its list (one
Py_NewRef per result), then dropped the chunk's list (one Py_DECREF per
result). Other threads own these objects, so each change was atomic,
and the calling thread did it alone after the workers had finished:
2.2 million results took a large share of a run on 64 threads. Chunks
now keep their results in a vector and the list takes the references.

map_many with fields on 7.6 GB of GH Archive events, mapped once,
big4 (Xeon Gold 6548N), 3.14.10t: 56.6 to 62.9 GB/s with 64 threads,
65.9 to 73.7 GB/s with 128.
@lemire
lemire merged commit 9d8430f into main Nov 14, 2026
35 checks passed
lemire added a commit that referenced this pull request Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in
  place (#13).
- Parallel streams move their results into the list without reference
  counting (#14).
- fields= reads the documents with On-Demand (#15).
- map_many(..., drop_none=True) leaves out the results that are None (#16).
- Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets
  without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16
  bytes or more) and for parse_many under Pyodide (#17).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant