Skip to content

Streams read a read-only memoryview in place - #13

Merged
lemire merged 2 commits into
mainfrom
stream-readonly-buffer
Nov 14, 2026
Merged

lemire merged 2 commits into
mainfrom
stream-readonly-buffer

Conversation

@lemire

@lemire lemire commented Nov 14, 2026

Copy link
Copy Markdown
Member

loads_many, parse_many and map_many copied every memoryview before parsing, on one thread. For a large file mapped with mmap.ACCESS_READ, the copy dominated: on 7.6 GB of GH Archive events (big4, 64 threads, free-threaded 3.14.10), map_many with two fields ran at 5 GB/s because of the copy, against 35 to 40 GB/s now from a fresh mapping (72 GB/s once the pages are mapped).

A read-only, contiguous memoryview is now parsed in place. The stream keeps a memoryview of its own, so the buffer stays exported: the caller may release its view, and the mmap cannot be closed while a stream reads it. bytearray and writable memoryviews are still copied.

Tests: mmap input with and without threads, with fields, release of the caller's view, BufferError on close, and writable buffers still copied. Full suite passes on 3.14.10 and 3.14.10t.

loads_many, parse_many and map_many copied every memoryview, on one
thread, before parsing: on a 7.6 GB file mapped with mmap.ACCESS_READ,
the copy took 1.5 s and map_many with 64 threads ran at 5 GB/s instead
of 35 to 70 GB/s. A read-only, contiguous memoryview is now parsed in
place, through a view of our own so that the buffer stays exported.
A read-only memoryview says nothing of what it shows:
memoryview(bytearray).toreadonly(), or a read-only view of a writable
mmap, can still be written through another reference while simdjson
reads the buffer (from its stage-1 thread, after next() has returned).
The view is now read in place only when its exporter is bytes, or an
mmap that refuses a writable buffer (opened with ACCESS_READ); any other
view is copied, as before.
@lemire
lemire merged commit 9aa651a into main Nov 14, 2026
35 checks passed
lemire added a commit that referenced this pull request Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in
  place (#13).
- Parallel streams move their results into the list without reference
  counting (#14).
- fields= reads the documents with On-Demand (#15).
- map_many(..., drop_none=True) leaves out the results that are None (#16).
- Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets
  without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16
  bytes or more) and for parse_many under Pyodide (#17).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant