docs: scope the planner codec rebind to one layer

Installing a codec rebuilds the installed planner against it, but the
rebuild reaches exactly one `ForeignQueryPlanner`. A planner that resolved
a fallback at install time keeps that fallback's codecs, and neither side
can repair it: the host has no handle past the first layer, and the planner
library cannot re-derive codecs at plan time because `FFI_QueryPlanner`
holds them by value and `Session` exposes no accessor for the host's
current ones. Tracked upstream in apache/datafusion#24762.

The examples cannot demonstrate it. Their fallback lives in the same cdylib
as its wrapper, and `From<&FFI_QueryPlanner>` short-circuits on a matching
`library_marker_id`, so a same-library hop never serializes. Measured: a
layered planner produces the same codec traffic as a flat one.

What is demonstrable is that the session's planner tracks whichever handle
wrote it last, so re-installing a planner from the original handle rebinds
the session back to that handle's codecs rather than picking up a codec
installed through a derived one. Pinned by a new test as the sequel to
`test_a_discarded_derived_context_still_rebinds_the_planner`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: timsaucer

.ai/skills/ffi-capsule-protocol/SKILL.md

@@ -160,6 +160,23 @@ reintroduce a `with_query_planner` that pretends otherwise — the only way to
 give a handle its own planner is a fresh `Arc<SessionContext>`, which is what
 Rule 6 forbids.
 
+Installing a codec rebuilds the installed planner against it, and that rebuild
+reaches exactly one layer. `FFI_QueryPlanner::new_with_ffi_codecs` unwraps one
+`ForeignQueryPlanner`; a fallback that planner resolved at install time sits in
+its library's private data with no handle on this side, and cannot re-derive
+codecs itself because `FFI_QueryPlanner` holds them by value and `Session`
+exposes no accessor for the host's current ones. So do not promise that install
+order is free — for a layered planner it is not. The examples cannot show this:
+their fallback lives in the same cdylib as its wrapper, and `datafusion-ffi`
+short-circuits a same-library hop rather than serializing. A fix has to come
+from upstream; tracked in
+[apache/datafusion#24762](https://github.com/apache/datafusion/issues/24762).
+
+The session's planner also tracks whichever handle wrote it last, so
+re-installing a planner on the original handle rebinds the session back to that
+handle's codecs. `test_reinstalling_a_planner_rebinds_the_session_to_that_handles_codecs`
+pins that; changing it should be deliberate.
+
 ## Where the truth is
 
 - `docs/source/contributor-guide/ffi.md` — the protocol, the fork caveat.

docs/source/contributor-guide/ffi.md

@@ -369,6 +369,43 @@ installed planner instead of layering another one. To chain planners, have the n
 planner wrap the capsule returned by `SessionContext.__datafusion_query_planner__()`,
 captured before the new planner is installed, and delegate to it explicitly.
 
+### Rebinding a planner's codecs is one level deep
+
+The rebuild above swaps the codecs on the installed `ForeignQueryPlanner` handle, and
+only that handle. A planner that wraps a fallback resolved that fallback when *it* was
+installed, and holds the result inside its own library's private data — behind a
+`create_physical_plan` function pointer, with no Python-side handle. A codec installed
+afterwards therefore reaches the outer planner and not the fallback, which keeps
+whichever codecs were in force when it was imported.
+
+Neither side can repair that:
+
+- **The host cannot reach it.** `FFI_QueryPlanner::new_with_ffi_codecs` unwraps exactly
+  one `ForeignQueryPlanner` layer. There is no deeper handle to unwrap — the same
+  situation as a codec embedded in a registered `FFI_CatalogProvider`.
+- **The planner library cannot re-derive it.** `FFI_QueryPlanner` holds its codecs by
+  value, and `Session` exposes no accessor for the ones the host currently has, so
+  `create_physical_plan` cannot pick them up from the session it is handed. The rebuild
+  has to be eager, and an eager rebuild only sees the top layer.
+
+A fix has to come from upstream, and is tracked in
+[apache/datafusion#24762](https://github.com/apache/datafusion/issues/24762).
+
+The stale codecs stay usable rather than dangling — they hold weak handles to the one
+`Arc<SessionContext>` that Rule 6 keeps alive — so the effect is a fallback hop
+serializing with an older codec, not a failure. It is also invisible to the examples
+here, which use one fallback in the same cdylib as its wrapper; `datafusion-ffi`
+short-circuits a same-library hop rather than serializing, so no codec runs. A fallback
+in a *different* library would serialize, and would do it with the codecs it was
+imported with.
+
+So install the codecs before a layered planner. If a codec has to go in afterwards,
+install the outer planner again *on the handle that holds the new codec* — that re-runs
+its getter, which re-imports the fallback against that handle's codecs. Re-installing on
+the original handle rebinds the session's planner back to the original handle's codecs
+instead, which is the trap
+`test_reinstalling_a_planner_rebinds_the_session_to_that_handles_codecs` pins.
+
 ## Alternative Approach
 
 Suppose you needed to expose some other features of DataFusion and you could not wait

examples/datafusion-ffi-example/README.md

@@ -43,6 +43,6 @@ ctx = ctx.with_physical_extension_codec(provider_physical_codec)
 ctx.set_query_planner(planner)
 ```
 
-Installing a codec after the planner rebuilds the planner against it, so this order is a recommendation rather than a requirement. Planner-last states the ownership flow more clearly.
+Installing a codec after the planner rebuilds the planner against it, so this order is a recommendation rather than a requirement. Planner-last states the ownership flow more clearly. The exception is a planner that wraps a fallback: the rebuild reaches the installed planner only, not the fallback inside it, so codecs-first is a requirement there. See [Rebinding a planner's codecs is one level deep](../../docs/source/contributor-guide/ffi.md#rebinding-a-planners-codecs-is-one-level-deep), which also covers why re-installing a planner rebinds the session to the codecs of whichever handle it was installed on.
 
 For the limits behind that choice — why there is one external codec owner rather than a registry, which node kinds survive the boundary, and what a derived context shares with the context it came from — see [Query Planners Across Multiple Libraries](../../docs/source/contributor-guide/ffi.md#query-planners-across-multiple-libraries) in the contributor guide.

examples/datafusion-ffi-query-planner-example/README.md

@@ -55,6 +55,6 @@ ctx.set_query_planner(MyQueryPlanner())
 
 `MyPlannerConfig` is transferred through the foreign session. `MyQueryPlanner` reads `ffi_query_planner.max_rows`, creates the plan with `DefaultPhysicalPlanner`, and adds a built-in `GlobalLimitExec`. The test changes the setting with `SET` and verifies the new row limit.
 
-The provider's codec pair is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. This planner deliberately uses only built-in physical nodes. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against it, but planner-last order is easier to audit.
+The provider's codec pair is attached to the planner when it is installed and is also used to decode the returned physical plan in `datafusion-python`. This planner deliberately uses only built-in physical nodes. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against it, but planner-last order is easier to audit. That rebuild is one level deep — a planner constructed with `fallback=` keeps the codecs its fallback was imported with — so codecs-first is a requirement rather than a preference once planners are layered. See [Rebinding a planner's codecs is one level deep](../../docs/source/contributor-guide/ffi.md#rebinding-a-planners-codecs-is-one-level-deep).
 
 For the limits behind that choice — why there is one external codec owner rather than a registry, which node kinds survive the boundary, and what a derived context shares with the context it came from — see [Query Planners Across Multiple Libraries](../../docs/source/contributor-guide/ffi.md#query-planners-across-multiple-libraries) in the contributor guide.

examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py

@@ -535,6 +535,41 @@ def test_a_discarded_derived_context_still_rebinds_the_planner():
     assert later.table_provider_encode_calls() > 0
 
 
+def test_reinstalling_a_planner_rebinds_the_session_to_that_handles_codecs():
+    """A planner is built against the codecs of the handle it is installed from.
+
+    The sequel to the test above, and the trap it sets up. Once a discarded
+    derived handle has rebound the session's planner to its codec, installing
+    the same planner again from the *original* handle rebuilds it against that
+    handle's codec instead -- which never changed. The session's planner tracks
+    whichever handle wrote it last, not the newest codec installed anywhere.
+
+    So "re-install the planner after installing a codec" only repairs anything
+    when it is done from the handle holding the new codec.
+    """
+    ctx, original_logical, _physical_codec = codec_context()
+    planner = MyQueryPlanner()
+    ctx.set_query_planner(planner)
+
+    later = MyLogicalExtensionCodec()
+    # Deliberately discarded, exactly as in the test above.
+    ctx.with_logical_extension_codec(later)
+    gc.collect()
+
+    ctx.sql('SELECT "A" FROM numbers ORDER BY "A"').collect()
+    assert later.table_provider_encode_calls() > 0
+    assert original_logical.table_provider_encode_calls() == 0
+
+    # `ctx`'s own codec field never changed, so this rebuilds the planner
+    # against `original_logical` and drops `later` from the session's planner.
+    ctx.set_query_planner(planner)
+    encodes_by_later = later.table_provider_encode_calls()
+
+    ctx.sql('SELECT "A" FROM numbers ORDER BY "A"').collect()
+    assert original_logical.table_provider_encode_calls() > 0
+    assert later.table_provider_encode_calls() == encodes_by_later
+
+
 def test_query_planner_requires_provider_codec():
     config = SessionConfig().with_extension(MyPlannerConfig(max_rows=2))
     ctx = SessionContext(config)

python/datafusion/context.py

@@ -1788,6 +1788,13 @@ def set_query_planner(self, planner: QueryPlannerExportable | _PyCapsule) -> Non
         :meth:`~SessionContext.__datafusion_query_planner__`, captured
         *before* the new planner is installed.
 
+        Install any extension codecs before a layered planner. Installing a
+        codec afterwards rebuilds the installed planner against it, but not the
+        fallback inside it, which keeps the codecs it was imported with. Note
+        also that the planner is built against the codecs of the context this
+        method is called on, so installing the same planner again on a different
+        handle rebinds the session's planner to *that* handle's codecs.
+
         Args:
             planner: Object exposing ``__datafusion_query_planner__`` (see
                 :class:`QueryPlannerExportable`) or a raw