Roadmap: Python limited API / stable ABI support (CPython 3.12+)

[henryiii]

Status: draft for maintainer discussion. No code has changed yet.

Updated after the precompiled-mode split (#6131-#6138): non-template definitions now
live in -inl.h files with declarations in the matching .h, and an opt-in
PYBIND11_PRECOMPILED build compiles them once into a static library. File references
below name the -inl.h files where the code moved.

Goal

Let users build pybind11 extensions with Py_LIMITED_API=0x030C0000 and ship one
.abi3 wheel for all CPython 3.12+ versions, the way nanobind's STABLE_ABI option
does. A few features may be disabled under the stable ABI. The plan is structured as
many small PRs, most of which are pure cleanups mergeable now and tested by the
existing CI matrix.

Why 3.12, and key facts (verified against CPython 3.14 headers)

• PyType_FromMetaclass (the core requirement) is new in 3.12. nanobind also pins
  Py_LIMITED_API=0x030C0000 (nanobind-config.cmake:440).
• METH_FASTCALL has been in the limited API since 3.10
  (methodobject.h:114-116). PyCFunction_NewEx, PyObject_Vectorcall, and
  PyVectorcall_NARGS are all available at 0x030C0000, so the existing function-call
  machinery ports without a slow fallback.
• PyMemberDef (needed for __weaklistoffset__ / __dictoffset__ /
  __vectorcalloffset__) entered the limited API in 3.12.
• Py_TPFLAGS_MANAGED_DICT is not in the limited API even in 3.14 (object.h,
  inside #ifndef Py_LIMITED_API). Under the stable ABI, py::dynamic_attr() must
  use the trailing-__dict__ layout (basicsize = sizeof(instance) + sizeof(PyObject*) plus a __dictoffset__ member) — the layout we already ship as
  PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET (detail/class-inl.h), and the
  same choice nanobind made (nb_type.cpp:1501-1513).
• Py_tss_t is opaque under the limited API. thread_specific_storage
  (detail/internals.h) embeds it by value, so the limited-API build needs a
  heap-allocated (PyThread_tss_alloc) variant. That changes the internals struct
  layout, which is why ABI isolation (below) must land before anything else diverges.
• pybind11 does not need nanobind's PEP 697 trick (negative basicsize +
  PyObject_GetTypeData). nanobind appends its type_data to the type object;
  pybind11 keeps detail::type_info in the internals registry maps. A plain positive
  basicsize = sizeof(instance) in the base type's PyType_Spec suffices, with
  weakref support moving from tp_weaklistoffset to a __weaklistoffset__ member.
• PyType_GetSlot works on static types since 3.10, so slots of PyType_Type /
  PyProperty_Type can be cached in per-DSO function-local statics — no internals
  ABI change needed for slot caching.
• PyType_FromMetaclass rejects metaclasses that override tp_new. Our default
  metaclass only overrides tp_call/tp_setattro/tp_getattro/tp_dealloc, so it is
  fine, but user-supplied py::metaclass(handle) with a custom tp_new cannot go
  through the spec path (keep the legacy path for those in non-limited builds).

What blocks compilation today

Hard blockers (need redesign):

1. Hand-built heap types. detail/class-inl.h allocates raw PyHeapTypeObjects and
   pokes tp_* fields for everything: make_static_property_type,
   make_default_metaclass, make_object_base_type, make_new_python_type,
   enable_dynamic_attributes, enable_buffer_protocol. All must move to
   PyType_FromMetaclass + PyType_Spec/PyType_Slot. We already have one
   spec-built type in-tree as precedent: detail/function_record_pyobject.h:75-100.
2. Private APIs. _PyType_Lookup (class-inl.h, cpp_conduit.h) and
   _PyObject_GetDictPtr (class-inl.h), plus direct base-slot calls
   (PyType_Type.tp_setattro/..., PyProperty_Type.tp_descr_get/...).
3. Thread-state internals. The default (non-simple) gil.h reads
   tstate->gilstate_counter; internals-inl.h uses _PyThreadState_UncheckedGet and
   tstate->interp; subinterpreter.h reads ->interp / ->native_thread_id.
4. Slot-pointer identity. cpp_conduit.h:24 compares
   type_obj->tp_new == pybind11_object_new; the PyPy branch two lines up already
   has the portable replacement (internals registry lookup) — also more correct
   cross-DSO today.
5. Public API leak. py::custom_type_setup's callback receives a raw
   PyHeapTypeObject* (attr.h:100-115); there is no spec-path equivalent.

Mechanical replacements (exist in the 3.12 limited API): tp_name →
PyType_GetName; PyTuple/PyList_GET_ITEM → function forms;
->tp_as_number->nb_bool → PyObject_IsTrue-style; tp_bases/tp_mro →
__bases__/__mro__; tp_doc → Py_tp_doc slot; traceback struct-walking in
error_already_set::what() → attribute-based walking / public PyFrame_*. The
buffer protocol is not a blocker: Py_buffer/PyBuffer_* are stable since 3.11
and Py_bf_getbuffer/Py_bf_releasebuffer are settable via PyType_Spec.

ABI isolation

A stable-ABI module and a regular module in one process cannot share internals (the
struct layouts differ: TSS keys, type_info extensions). Plan: a
PYBIND11_INTERNALS_SABI_TAG (empty normally, "_stable" under Py_LIMITED_API)
composed into PYBIND11_INTERNALS_ID / PYBIND11_MODULE_LOCAL_ID — but not into
PYBIND11_PLATFORM_ABI_ID, since the C++ ABI is unchanged and the conduit protocol
should keep bridging stable ↔ regular modules. nanobind draws the line in the same
place (nb_abi.h:76-103). The two worlds each get their own pybind11 universe;
cpp_conduit still connects them.

Interaction with the precompiled mode

tools/check_inl_headers.py rejects any macro in a -inl.h preprocessor condition that
is not known to be identical between the library and the modules, or encoded in
PYBIND11_PRECOMPILED_CONFIG_CHECK (detail/internals.h). Consequences:

• Py_LIMITED_API, PYBIND11_INTERNALS_SABI_TAG, and the stage 1 opt-in macro
  PYBIND11_TYPE_CREATION_VIA_SPEC each need a bit in the config-check symbol, an entry
  in GUARDED_MACROS, and a line in the docs/compiling.rst list. A mismatch between
  the library and a module then fails at link time with one readable undefined symbol.
• Every feature gated off under the stable ABI (custom_type_setup, embed.h,
  subinterpreter.h, chrono.h) needs matching conditionals on the declaration in the
  .h and the definition in the -inl.h.
• pybind11_precompile() builds one static library per build tree with the directory
  flags of the first PRECOMPILE target. A stable-ABI module and a regular module
  cannot share it. Rule: STABLE_ABI and PRECOMPILE in one build tree require
  Py_LIMITED_API set before the first PRECOMPILE target; otherwise the config guard
  reports the mismatch.
• C1 is mostly done: the hand-rolled type creation is already a set of PYBIND11_INLINE
  free functions in class-inl.h with declarations in class.h. The spec backend can be
  a sibling block behind the opt-in macro; no new abstraction is needed.
• C2 is unaffected: function-local statics in the precompiled library stay per-module,
  because each extension links its own copy of the static library.

Feature matrix under the stable ABI

Feature	Decision	Reason
GIL scoped_acquire/release (non-simple)	forced to PYBIND11_SIMPLE_GIL_MANAGEMENT	reads tstate->gilstate_counter; gil_simple.h is pure PyGILState_* and already limited-API-clean
subinterpreters	disabled	thread/interp-state field reads; nanobind parity
embed.h	disabled	non-limited init APIs; embedding + abi3 is niche
chrono.h	disabled at first; later port via Python datetime calls	PyDateTime_* capsule macros poke struct fields (nanobind ported it this way)
py::custom_type_setup	unavailable	contract is a raw PyHeapTypeObject* pre-PyType_Ready
buffer protocol	supported	spec slots since 3.11
numpy / eigen	expected to work (needs CI confirmation)	numpy's C API comes from its own capsule table, orthogonal to Py_LIMITED_API
free-threaded CPython	mutually exclusive	no stable ABI for free-threading yet (same as nanobind)
PyPy / GraalPy	error	no meaningful abi3

Rollout strategy for the type-creation rewrite

Three stages, so the spec-based path is never an untested #ifdef forest and never a
big-bang flip:

1. Spec-based type creation lands behind an opt-in macro (e.g.
   PYBIND11_TYPE_CREATION_VIA_SPEC) for CPython 3.12+ non-limited builds, with a
   dedicated CI job running the full test suite with it on.
2. Limited-API builds require it (it is the only option there).
3. After at least one release of soak, flip the default for all CPython 3.12+
   builds; the hand-rolled path remains for <3.12, PyPy/GraalPy, and user metaclasses
   with custom tp_new.

No PYBIND11_INTERNALS_VERSION bump is required for the flip itself: the internals
layout is untouched by how types are created. The cross-module requirement is that
spec-created and hand-rolled static_property_type / default_metaclass /
instance_base are interchangeable; a cross-module mixing test gates stage 3.

PR sequence

Phase B — preparatory PRs, mergeable now (no limited-API flag anywhere)

Each is small, independently revertible, and fully exercised by the existing matrix.

• B1 Clean #error for Py_LIMITED_API in detail/common.h (today a
  limited-API compile fails with pages of noise), plus reserved capability
  macros. This is also the future "switch" location. Add Py_LIMITED_API to
  PYBIND11_PRECOMPILED_CONFIG_CHECK, check_inl_headers.py, and the docs list.
• B2 ABI tag mechanism (PYBIND11_INTERNALS_SABI_TAG, see above). ~10 lines
  in internals.h, plus a config-check bit (the tag changes internals-inl.h code
  without changing PYBIND11_INTERNALS_VERSION). Must precede any limited-API
  layout divergence.
• B3 detail::get_bases() / get_mro() helpers replacing direct
  tp_bases/tp_mro reads (class-inl.h, type_caster_base-inl.h,
  pybind11-inl.h).
• B4 Type-name helpers replacing raw tp_name reads (cast.h, stl.h,
  pytypes.h/pytypes-inl.h, class-inl.h, function_record_pyobject.h). Non-limited implementation stays tp_name;
  the PyType_GetName branch comes later.
• B5 Unify type_is_managed_by_our_internals on the registry lookup, drop
  the tp_new pointer-identity check (cpp_conduit.h) — a latent cross-DSO
  correctness fix on its own.
• B6 Traceback formatting in error_already_set::what() without struct
  pokes (pytypes-inl.h): attribute-based tb_next/tb_frame walking +
  public PyFrame_GetCode/PyFrame_GetLineNumber.
• B7 Replace _PyType_Lookup with a detail::type_lookup MRO walk using
  PyType_GetDict (metaclass attribute paths, not hot).
• B8 tstate->interp → PyThreadState_GetInterpreter() in
  get_interpreter_state_unchecked (internals-inl.h).
• B9 Consolidate the PyPy/GraalPy fallback gates into a semantic macro (e.g.
  PYBIND11_HAVE_DIRECT_TYPE_SLOTS); the limited-API port then reuses
  already-CI-tested fallback paths (get_fully_qualified_tp_name,
  static-property eval fallback, meta_setattro __set__ path, bool cast).
• B10 Non-hot GET_ITEM macro cleanups + a wrapper for the hot dispatcher
  sites (pybind11.h, cast.h, stl.h).

Phase C — type-creation migration (sequential; opt-in backend + CI job)

• C1 Add the PYBIND11_TYPE_CREATION_VIA_SPEC opt-in macro (config-check bit,
  check_inl_headers.py, docs) and the dedicated CI job. The seam itself already
  exists: class-inl.h free functions declared in class.h. May fold into C3.
• C2 Cached-slot helpers via PyType_GetSlot for
  PyType_Type/PyProperty_Type slots; enable unconditionally on 3.10+ (the
  pointers are identical, so the whole matrix tests it).
• C3 Spec backend for pybind11_static_property (smallest type first).
• C4 Spec backend for the metaclass via PyType_FromMetaclass(nullptr, …)
  with Py_tp_call/Py_tp_setattro/Py_tp_getattro/Py_tp_dealloc slots.
• C5 Spec backend for the pybind11_object base: basicsize = sizeof(instance), Py_tp_new/Py_tp_init/Py_tp_dealloc, __weaklistoffset__
  member.
• C6 Spec backend for make_new_python_type: Py_tp_bases, Py_tp_doc
  (kills the manual tp_doc malloc), both dynamic_attr layouts
  (Py_TPFLAGS_MANAGED_DICT when non-limited, trailing dict + __dictoffset__
  when limited), buffer slots, is_final via omitting Py_TPFLAGS_BASETYPE,
  custom_type_setup routed to the legacy backend.
• C7 Instance alloc/free via PyType_GetSlot(type, Py_tp_alloc/Py_tp_free)
  and dict traverse/clear via a dictoffset cached in type_info (replaces
  _PyObject_GetDictPtr).

Risk register for phase C (each needs a test): user metaclasses with custom
tp_new; __qualname__ set post-creation (pydoc/repr/pickling); newly visible
__weaklistoffset__/__dictoffset__ attributes; Python-side subclassing with
__slots__ and GC cycles through subclass dicts; multiple-inheritance best-base;
cross-module mixing of spec-created and hand-rolled internals types.

Phase D — end game

• D1 Header flip: replace B1's #error with real gates — require CPython ≥
  3.12 and Py_LIMITED_API >= 0x030C0000, reject PyPy/GraalPy/free-threading,
  force simple GIL, disable embed/subinterpreter/chrono/custom_type_setup with
  actionable messages (gating both the .h declaration and the -inl.h
  definition), heap-allocated TSS keys.
• D2 CMake: pybind11_add_module(... STABLE_ABI) → python_add_library(... USE_SABI 3.12) (CMake ≥ 3.26), Py_LIMITED_API define, .abi3 suffix
  (pybind11NewTools.cmake, pybind11GuessPythonExtSuffix.cmake), exclusion
  checks per nanobind-config.cmake:390-405; classic pybind11Tools.cmake gets
  a clear FATAL_ERROR or parity. STABLE_ABI + PRECOMPILE: the shared
  pybind11::precompiled library must be built with Py_LIMITED_API too; document
  that one build tree cannot mix stable-ABI and regular precompiled modules.
• D3 setup_helpers.py: Pybind11Extension(py_limited_api=True) plumbing.
• D4 CI: build with Py_LIMITED_API on 3.12 and latest; the money test —
  build the suite once against 3.12 headers and run the same .abi3.so on
  3.13/3.14 (model on the "unstable ABI" job, upstream.yml:94-110);
  tests/env.py gains a LIMITED_API flag for skips. One job also sets
  PYBIND11_TEST_PRECOMPILE=ON under Py_LIMITED_API, and check_inl_headers.py
  runs in that configuration, so no -inl.h code reads a non-limited struct field.
• D5 Docs: how to enable, the feature matrix, ABI-isolation implications,
  performance notes (slot indirection, attribute-based fallbacks).
• Stage 3 flip: spec-based creation becomes the default for all CPython
  3.12+ builds after ≥1 release of opt-in soak.

Ordering

B1..B10: independent, land in any order now
         (B2 must precede any internals/type_info layout divergence, i.e. D1;
          B1 adds the Py_LIMITED_API config-check bit that C1 and D1 extend)
C1 → C2 → C3 → C4 → C5 → C6 → C7   (each keeps hand-rolled as default)
D1 (needs B*, C*) → D2 → D3 → D4 → D5

Roughly 10 prep PRs, 7 core PRs, 5 end-game PRs. Everything before D1 stays green on
the existing CI matrix, so the project never carries a long-lived feature branch.