Repository navigation
Conversation
- escape_into: without SSE2 or NEON, the tail code (which assumes under 16 bytes) received the whole string, and dumps(ensure_ascii=False) wrote only its first and last 8 bytes. A portable loop takes 8 bytes at a time. - Under Emscripten without pthreads, undefine _REENTRANT (set by Pyodide's pyconfig.h) so that simdjson does not start a thread in parse_many, which aborted in a noexcept function. - Tests: markers to skip the tests that start Python threads or processes under Pyodide; skip the nesting depths at which json too overflows the JavaScript stack there. - CI: build a Pyodide wheel for PyPI, and test under Pyodide.
In a Pyodide venv, the pytest script has a shebang to a shell script, which macOS cannot run as an interpreter. Native builds are unchanged. With cibuildwheel 4.3.1, the release builds Pyodide wheels for cp313 (Pyodide 0.28 and 0.29), cp314 and cp315.
- pyodide-build appends its own flags, which end with -Oz (size), after those of setup.py: the extension was compiled and linked for size. [tool.pyodide.build] in pyproject.toml gives Pyodide's flags with -O3 (the same in Pyodide 0.29, 314 and 315). In Node, dumps and dumpb are twice as fast on canada.json and 15-30% faster on citm_catalog.json; loads, loads_many 4-12%. The wheel grows from 191 kB to 215 kB. - first_escape (dumpb, non-ASCII strings) without SSE2 or NEON skips 8 bytes at a time, as escape_into does: dumpb on twitter.json +13%. - Typo in test_loads.py. Tested: 807 passed, 71 skipped under Pyodide 314.0.7; 849 passed, 29 skipped natively (macOS arm64, Python 3.14). Native builds unchanged (the new loop is in the #else of FSJ_SSE2 || FSJ_NEON).
lemire
added a commit
that referenced
this pull request
Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in place (#13). - Parallel streams move their results into the list without reference counting (#14). - fields= reads the documents with On-Demand (#15). - map_many(..., drop_none=True) leaves out the results that are None (#16). - Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16 bytes or more) and for parse_many under Pyodide (#17).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes fastsimdjson build as a Pyodide wheel (
micropip.install("fastsimdjson")in the browser), fixes two bugs that show up there, and adds Pyodide to CI.No effect on native builds
The source changes compile away on x64 and ARM:
escape_intoandfirst_escapesit in the#elsebranch of#if FSJ_SSE2 || FSJ_NEON;#undef _REENTRANTis under#if defined(__EMSCRIPTEN__) && !defined(__EMSCRIPTEN_PTHREADS__);[tool.pyodide.build]inpyproject.tomlis read only by pyodide-build.Checked:
src/fastsimdjson.cppfrommainand from this branch, compiled with the flags ofsetup.py(-std=c++17 -O3) for arm64 and x86_64, give objects whose sections are byte for byte identical (objdump -s); only a local symbol named after the source file differed (the two copies had different names). The native test suite gives the same result asmain(818 passed, 60 skipped, macOS arm64).Bugs fixed
dumps(..., ensure_ascii=False)wrote garbage for non-ASCII strings of 16 bytes or more on targets without SSE2 or NEON (WebAssembly, but also ppc64le, s390x, riscv64...).escape_intohad no main loop there: the tail code, which assumes fewer than 16 bytes, got the whole string and wrote only its first and last 8 bytes. A portable loop now takes 8 bytes at a time (load_u64/word_needs_escape, as the tail does).loads_many/parse_manyaborted on inputs of more than one batch under Pyodide. Pyodide'spyconfig.hdefines_REENTRANT, so#include <Python.h>enabledSIMDJSON_THREADS_ENABLED;document_stream::start()(noexcept) then started astd::thread, which throws without pthreads:std::terminate. The macro is undefined afterPython.hunder Emscripten without pthreads. (simdjson_arrow.cppandsimdjson_msgpack.cppdo not includePython.h, so they already had threads off.)Tests
tests/conftest.py: markersrequires_threadsandrequires_processes, which skip the tests starting Python threads,forkorsubprocessunder Pyodide only. (The library's ownthreads=arguments work: the worker pool falls back to the calling thread.)dumps, 5,000 inloads, 20,000+ in typedloads, unboundedcheck_circular=False) overflows the JavaScript stack under Pyodide, a fatal error, before Python raisesRecursionError. The standardjsonmodule does the same there, so those depths are skipped on Emscripten.Under Pyodide 314.0.7 (Python 3.14.2, Emscripten 5.0.3) in Node: 807 passed, 71 skipped, 0 failed.
CI
buildwheels.yml: abuild_pyodidejob (cibuildwheel,CIBW_PLATFORM: pyodide). With cibuildwheel 4.3.1 it builds and tests one wheel per Pyodide ABI, uploaded to PyPI with the others (PyPI accepts the PEP 783pyemscripten_*_wasm32tags):cp313-cp313-pyemscripten_2025_0_wasm32(Pyodide 0.28, 0.29)cp314-cp314-pyemscripten_2026_0_wasm32(Pyodide 314)cp315-cp315-pyemscripten_2026_5_wasm32tests.yml: a Pyodide job that builds the wheels and runs the tests in Node (this exercises the portable code paths, which no other job does).pyproject.toml:[tool.pyodide.build]compiles and links with-O3. pyodide-build appends its own flags after those ofsetup.py, and they end with-Oz(size): without this section, the wheel was built for size. The section repeats Pyodide's flags (the same in Pyodide 0.29, 314 and 315) with-O3in place of-Oz, since$(...)variables are not expanded there. The wheel grows from 191 kB to 215 kB.pyproject.toml:[tool.cibuildwheel.pyodide]runs the tests withpython -m pytest(in a Pyodide venv, thepytestscript has a shebang to a shell script, which macOS cannot run). The native wheel jobs are unchanged.The Pyodide cibuildwheel job was run locally (macOS arm64): the three wheels build and pass the tests (807 passed, 71 skipped each). It has not run on GitHub Actions yet: this PR's checks will be the first run.
Speed under Pyodide
MB/s, files from simdjson's data set, Pyodide 314.0.7 in Node 24 against native CPython 3.14, Apple Silicon (best of 80 runs). simdjson and simdutf have no WebAssembly SIMD kernel, so they run their portable code.
json.loadsfastsimdjson.loadsjson.dumpsfastsimdjson.dumpsfastsimdjson.dumpbUnder Pyodide, fastsimdjson reads JSON 2 to 3.5 times as fast as
json, and writes it 3 to 8 times as fast.-O3instead of-Ozmadedumpsanddumpbtwice as fast on canada.json (floats) and 15 to 30% faster on citm_catalog.json;loadsandloads_many, 4 to 12%.first_escape8 bytes at a time madedumpb13% faster on twitter.json. WebAssembly SIMD (-msimd128) was tried: about 10% slower, and not every engine supports it, so it is not used. Inloads, simdjson's portable parser now takes about a third of the time.