Skip to content

Build and test for Pyodide (WebAssembly) - #17

Merged
lemire merged 3 commits into
mainfrom
pyodide
Nov 14, 2026
Merged

lemire merged 3 commits into
mainfrom
pyodide

Conversation

@lemire

@lemire lemire commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Makes fastsimdjson build as a Pyodide wheel (micropip.install("fastsimdjson") in the browser), fixes two bugs that show up there, and adds Pyodide to CI.

No effect on native builds

The source changes compile away on x64 and ARM:

  • the new loops in escape_into and first_escape sit in the #else branch of #if FSJ_SSE2 || FSJ_NEON;
  • the #undef _REENTRANT is under #if defined(__EMSCRIPTEN__) && !defined(__EMSCRIPTEN_PTHREADS__);
  • [tool.pyodide.build] in pyproject.toml is read only by pyodide-build.

Checked: src/fastsimdjson.cpp from main and from this branch, compiled with the flags of setup.py (-std=c++17 -O3) for arm64 and x86_64, give objects whose sections are byte for byte identical (objdump -s); only a local symbol named after the source file differed (the two copies had different names). The native test suite gives the same result as main (818 passed, 60 skipped, macOS arm64).

Bugs fixed

  1. dumps(..., ensure_ascii=False) wrote garbage for non-ASCII strings of 16 bytes or more on targets without SSE2 or NEON (WebAssembly, but also ppc64le, s390x, riscv64...). escape_into had no main loop there: the tail code, which assumes fewer than 16 bytes, got the whole string and wrote only its first and last 8 bytes. A portable loop now takes 8 bytes at a time (load_u64 / word_needs_escape, as the tail does).
  2. loads_many / parse_many aborted on inputs of more than one batch under Pyodide. Pyodide's pyconfig.h defines _REENTRANT, so #include <Python.h> enabled SIMDJSON_THREADS_ENABLED; document_stream::start() (noexcept) then started a std::thread, which throws without pthreads: std::terminate. The macro is undefined after Python.h under Emscripten without pthreads. (simdjson_arrow.cpp and simdjson_msgpack.cpp do not include Python.h, so they already had threads off.)

Tests

  • tests/conftest.py: markers requires_threads and requires_processes, which skip the tests starting Python threads, fork or subprocess under Pyodide only. (The library's own threads= arguments work: the worker pool falls back to the calling thread.)
  • Very deep nesting (10,000+ levels in dumps, 5,000 in loads, 20,000+ in typed loads, unbounded check_circular=False) overflows the JavaScript stack under Pyodide, a fatal error, before Python raises RecursionError. The standard json module does the same there, so those depths are skipped on Emscripten.

Under Pyodide 314.0.7 (Python 3.14.2, Emscripten 5.0.3) in Node: 807 passed, 71 skipped, 0 failed.

CI

  • buildwheels.yml: a build_pyodide job (cibuildwheel, CIBW_PLATFORM: pyodide). With cibuildwheel 4.3.1 it builds and tests one wheel per Pyodide ABI, uploaded to PyPI with the others (PyPI accepts the PEP 783 pyemscripten_*_wasm32 tags):
    • cp313-cp313-pyemscripten_2025_0_wasm32 (Pyodide 0.28, 0.29)
    • cp314-cp314-pyemscripten_2026_0_wasm32 (Pyodide 314)
    • cp315-cp315-pyemscripten_2026_5_wasm32
  • tests.yml: a Pyodide job that builds the wheels and runs the tests in Node (this exercises the portable code paths, which no other job does).
  • pyproject.toml: [tool.pyodide.build] compiles and links with -O3. pyodide-build appends its own flags after those of setup.py, and they end with -Oz (size): without this section, the wheel was built for size. The section repeats Pyodide's flags (the same in Pyodide 0.29, 314 and 315) with -O3 in place of -Oz, since $(...) variables are not expanded there. The wheel grows from 191 kB to 215 kB.
  • pyproject.toml: [tool.cibuildwheel.pyodide] runs the tests with python -m pytest (in a Pyodide venv, the pytest script has a shebang to a shell script, which macOS cannot run). The native wheel jobs are unchanged.

The Pyodide cibuildwheel job was run locally (macOS arm64): the three wheels build and pass the tests (807 passed, 71 skipped each). It has not run on GitHub Actions yet: this PR's checks will be the first run.

Speed under Pyodide

MB/s, files from simdjson's data set, Pyodide 314.0.7 in Node 24 against native CPython 3.14, Apple Silicon (best of 80 runs). simdjson and simdutf have no WebAssembly SIMD kernel, so they run their portable code.

twitter.json citm_catalog.json canada.json
Pyodide native Pyodide native Pyodide native
json.loads 302 321 344 426 97 150
fastsimdjson.loads 586 1607 739 1577 339 685
json.dumps 355 673 180 342 54 92
fastsimdjson.dumps 1215 2041 566 938 407 914
fastsimdjson.dumpb 2587 5178 1338 2417 456 2025

Under Pyodide, fastsimdjson reads JSON 2 to 3.5 times as fast as json, and writes it 3 to 8 times as fast.

-O3 instead of -Oz made dumps and dumpb twice as fast on canada.json (floats) and 15 to 30% faster on citm_catalog.json; loads and loads_many, 4 to 12%. first_escape 8 bytes at a time made dumpb 13% faster on twitter.json. WebAssembly SIMD (-msimd128) was tried: about 10% slower, and not every engine supports it, so it is not used. In loads, simdjson's portable parser now takes about a third of the time.

- escape_into: without SSE2 or NEON, the tail code (which assumes under
  16 bytes) received the whole string, and dumps(ensure_ascii=False) wrote
  only its first and last 8 bytes. A portable loop takes 8 bytes at a time.
- Under Emscripten without pthreads, undefine _REENTRANT (set by Pyodide's
  pyconfig.h) so that simdjson does not start a thread in parse_many, which
  aborted in a noexcept function.
- Tests: markers to skip the tests that start Python threads or processes
  under Pyodide; skip the nesting depths at which json too overflows the
  JavaScript stack there.
- CI: build a Pyodide wheel for PyPI, and test under Pyodide.
In a Pyodide venv, the pytest script has a shebang to a shell script,
which macOS cannot run as an interpreter. Native builds are unchanged.
With cibuildwheel 4.3.1, the release builds Pyodide wheels for cp313
(Pyodide 0.28 and 0.29), cp314 and cp315.
- pyodide-build appends its own flags, which end with -Oz (size), after
  those of setup.py: the extension was compiled and linked for size.
  [tool.pyodide.build] in pyproject.toml gives Pyodide's flags with -O3
  (the same in Pyodide 0.29, 314 and 315). In Node, dumps and dumpb are
  twice as fast on canada.json and 15-30% faster on citm_catalog.json;
  loads, loads_many 4-12%. The wheel grows from 191 kB to 215 kB.
- first_escape (dumpb, non-ASCII strings) without SSE2 or NEON skips 8
  bytes at a time, as escape_into does: dumpb on twitter.json +13%.
- Typo in test_loads.py.

Tested: 807 passed, 71 skipped under Pyodide 314.0.7; 849 passed, 29
skipped natively (macOS arm64, Python 3.14). Native builds unchanged
(the new loop is in the #else of FSJ_SSE2 || FSJ_NEON).
@lemire
lemire merged commit 03c3302 into main Nov 14, 2026
36 checks passed
lemire added a commit that referenced this pull request Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in
  place (#13).
- Parallel streams move their results into the list without reference
  counting (#14).
- fields= reads the documents with On-Demand (#15).
- map_many(..., drop_none=True) leaves out the results that are None (#16).
- Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets
  without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16
  bytes or more) and for parse_many under Pyodide (#17).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant