Repository navigation
fields=: read the documents with On-Demand - #15
Merged
Merged
Conversation
With fields, loads_many and map_many built the whole DOM tape of every document to pick out a few values: stage 2 took 41% of the cycles of map_many with two fields on GH Archive events. With the formats whitespace and lines, the documents are now read with simdjson's On-Demand API: only the keys on the way to the pointers and the values found there are read and validated, and the rest of each document is skipped. Any error, or a case the walk does not handle (a big integer, a document larger than a batch, an incomplete last document), switches the stream to the DOM path from that document, which decides as before. Objects and arrays found at the pointers are built by the DOM from their text, as before. Pointers that are the whole document, or that lead into the value of another, keep the DOM path. map_many with two fields on 7.6 GB of GH Archive events (mapped once), big4, 3.14.10t, with #13 and #14: 2.5 to 3.6 GB/s on one thread, 31 to 40 GB/s on 16, 64 to 72 GB/s on 64 (map_many alone: 105 GB/s).
…calars reached
The walk stopped reading a document once its fields were found, and the
stream then skipped the rest without checking it. A malformed document
could thus move where the next one starts ({"a":1,"b":[1}{"a":2}}
gave [1]), and an incomplete last document gave its value before the
error. A scalar reached by the walk (a document such as tru, or a step
such as {"a": tru} with /a/b) was taken for a missing field: type()
looks at its first byte only.
The top level of each document is now read to its end, so its closing
bracket, its separators and its completeness are checked; the values
that no pointer reaches are still skipped without being checked. A
scalar that the walk reaches is read, and an invalid one goes to the
DOM path. Tests compare these cases with the whole documents, with both
formats (lines and whitespace split documents alike on the DOM path).
Cost on map_many with two fields over GH Archive events (big4): 3.59 to
3.41 GB/s on one thread, 73.4 to 72.3 GB/s on 64.
lemire
added a commit
that referenced
this pull request
Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in place (#13). - Parallel streams move their results into the list without reference counting (#14). - fields= reads the documents with On-Demand (#15). - map_many(..., drop_none=True) leaves out the results that are None (#16). - Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16 bytes or more) and for parse_many under Pyodide (#17).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
fields,loads_manyandmap_manybuilt the whole DOM tape of every document to pick out a few values. perf onmap_manywith two fields over GH Archive events: stage 2 (tape building) took 41% of the cycles.With the formats
whitespaceandlines, the documents are now read with simdjson's On-Demand API (iterate_many): only the keys on the way to the pointers and the values found there are read and validated; the rest of each document is skipped. This is documented in the README: an invalid value elsewhere in a document goes unnoticed.jsonfallback are as before.["/a", "/a/b"]), keep the DOM path, as do the other formats.release()frees it.New tests: skipped values are not validated, values read are (errors, json fallback: big integers, NaN, 1e400, lone surrogates), escaped and repeated keys, non-object documents, and random documents against a pure-Python JSON Pointer, with and without threads. Full suite passes on 3.14.10 and 3.14.10t.
map_manywith two fields on 7.6 GB of GH Archive events (file mapped once), big4 (Xeon Gold 6548N), 3.14.10t, with #13 and #14 applied: