Skip to content

HTTP 500 on every /query endpoint: pyarrow.default_serialization_context removed in pyarrow ≥12 (production flywire_fafb_public down) #229

Description

@luisan00

Summary

All /query endpoints on the production flywire_fafb_public datastack
have been returning HTTP 500 for at least 16+ hours (first
observed 2026-05-28 ~17:00 UTC, still failing 2026-05-29 09:26 UTC).
The error is the same on every request:

{"code": 500,
 "message": "module 'pyarrow' has no attribute 'default_serialization_context'"}

pyarrow.default_serialization_context was removed in pyarrow 12.0 (~2 years
ago, pyarrow #ARROW-14625).
The deployed service appears to be running a pyarrow version that no
longer has the symbol but code that still calls it.

Codex (the user-facing UI) is not affected, so this likely went
unnoticed — only the REST query API is broken.

Reproducer

Plain curl against any table, any column-selection, any limit, all
return the same 500:

curl -X POST \
  -H "Authorization: Bearer $FLYWIRE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"limit": 1}' \
  "https://prod.flywire-daf.com/materialize/api/v3/datastack/flywire_fafb_public/version/783/table/proofread_neurons/query"

The same call worked fine yesterday (downloaded 120,519,649 rows of
synapses_nt_v1 to completion via caveclient 8.0.1).

What still works

  • GET https://prod.flywire-daf.com/materialize/ → 302
  • GET /api/v3/datastack/flywire_fafb_public/versions → 200 [630, 783]
  • GET https://codex.flywire.ai/ → 200

What doesn't

Endpoint Result
POST /api/v3/.../table/<t>/query 500 pyarrow.default_serialization_context
POST /api/v2/.../table/<t>/query 500 'ipc_compress'
POST /api/v3/.../table/<t>/simple_query 404
?arrow_format=False ignored, same 500
return_pyarrow: false in body 500 schema error: "Unknown field"

All four tables I've tried fail the same way:
proofread_neurons, synapses_nt_v1, fly_synapses_neuropil_v6,
hierarchical_neuron_annotations.

Client side

  • caveclient 8.0.1
  • Python 3.14.5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions