Skip to content

fields=: read the documents with On-Demand - #15

Merged
lemire merged 3 commits into
mainfrom
ondemand-fields
Nov 14, 2026
Merged

lemire merged 3 commits into
mainfrom
ondemand-fields

Conversation

@lemire

@lemire lemire commented Nov 14, 2026

Copy link
Copy Markdown
Member

With fields, loads_many and map_many built the whole DOM tape of every document to pick out a few values. perf on map_many with two fields over GH Archive events: stage 2 (tape building) took 41% of the cycles.

With the formats whitespace and lines, the documents are now read with simdjson's On-Demand API (iterate_many): only the keys on the way to the pointers and the values found there are read and validated; the rest of each document is skipped. This is documented in the README: an invalid value elsewhere in a document goes unnoticed.

  • Strings, integers, floats, booleans and null at the pointers are converted directly (same string caches as the DOM path). Objects and arrays are built by the DOM from their text, as before.
  • Any error, or anything the walk does not handle (a big integer, a document larger than a batch, an incomplete last document), switches the stream to the DOM path from that document: errors and the json fallback are as before.
  • Pointers that are the whole document, or that lead into the value of another (["/a", "/a/b"]), keep the DOM path, as do the other formats.
  • Parallel chunks borrow a per-thread On-Demand parser, like the DOM chunk parser; release() frees it.

New tests: skipped values are not validated, values read are (errors, json fallback: big integers, NaN, 1e400, lone surrogates), escaped and repeated keys, non-object documents, and random documents against a pure-Python JSON Pointer, with and without threads. Full suite passes on 3.14.10 and 3.14.10t.

map_many with two fields on 7.6 GB of GH Archive events (file mapped once), big4 (Xeon Gold 6548N), 3.14.10t, with #13 and #14 applied:

threads before after
1 2.5 GB/s 3.6 GB/s
16 31.3 GB/s 40.5 GB/s
64 63.9 GB/s 72.5 GB/s (map_many alone: 105 GB/s; the rest is the caller's Counter)

With fields, loads_many and map_many built the whole DOM tape of every
document to pick out a few values: stage 2 took 41% of the cycles of
map_many with two fields on GH Archive events. With the formats
whitespace and lines, the documents are now read with simdjson's
On-Demand API: only the keys on the way to the pointers and the values
found there are read and validated, and the rest of each document is
skipped. Any error, or a case the walk does not handle (a big integer,
a document larger than a batch, an incomplete last document), switches
the stream to the DOM path from that document, which decides as before.

Objects and arrays found at the pointers are built by the DOM from their
text, as before. Pointers that are the whole document, or that lead into
the value of another, keep the DOM path.

map_many with two fields on 7.6 GB of GH Archive events (mapped once),
big4, 3.14.10t, with #13 and #14: 2.5 to 3.6 GB/s on one thread, 31 to
40 GB/s on 16, 64 to 72 GB/s on 64 (map_many alone: 105 GB/s).
…calars reached

The walk stopped reading a document once its fields were found, and the
stream then skipped the rest without checking it. A malformed document
could thus move where the next one starts ({"a":1,"b":[1}{"a":2}}
gave [1]), and an incomplete last document gave its value before the
error. A scalar reached by the walk (a document such as tru, or a step
such as {"a": tru} with /a/b) was taken for a missing field: type()
looks at its first byte only.

The top level of each document is now read to its end, so its closing
bracket, its separators and its completeness are checked; the values
that no pointer reaches are still skipped without being checked. A
scalar that the walk reaches is read, and an invalid one goes to the
DOM path. Tests compare these cases with the whole documents, with both
formats (lines and whitespace split documents alike on the DOM path).

Cost on map_many with two fields over GH Archive events (big4): 3.59 to
3.41 GB/s on one thread, 73.4 to 72.3 GB/s on 64.
@lemire
lemire merged commit a5f266f into main Nov 14, 2026
35 checks passed
lemire added a commit that referenced this pull request Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in
  place (#13).
- Parallel streams move their results into the list without reference
  counting (#14).
- fields= reads the documents with On-Demand (#15).
- map_many(..., drop_none=True) leaves out the results that are None (#16).
- Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets
  without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16
  bytes or more) and for parse_many under Pyodide (#17).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant