fix: skip the planner rebind when inlining is unchanged

`with_python_udf_inlining` rebuilds the handle's codecs and rebinds the
session's planner to them, which is state shared with every other handle on
that session. Asking for the setting a context already has changes nothing,
so it should not pay that side effect: a defensive no-op toggle on `ctx`
otherwise drags the planner back onto `ctx`'s codecs and silently undoes a
codec installed through another handle.

Returning the existing codecs is observationally equivalent to the rebuild
otherwise -- it wraps the same inner codec in a fresh `Python*Codec` -- so
the guard is only visible through that side effect. The new test fails
without it with `assert 0 > 0`.

Also pins the divergence the rebind creates. The planner carries the codecs
of whichever handle installed it last; every other path on a context uses
that context's own codec field. Those can be different handles, and then
`Expr.to_bytes(ctx)` and `ctx.sql(...)` encode with different codecs on the
same `ctx`. Stated as a rule in the FFI guide rather than left implicit in
the description of the mechanism.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: timsaucer

crates/core/src/context.rs

@@ -1508,6 +1508,23 @@ impl PySessionContext {
     }
 
     pub fn with_python_udf_inlining(&self, enabled: bool) -> Self {
+        // Rebinding the session's planner is a side effect on state shared with
+        // every other handle, so do not pay it for a call that changes nothing.
+        // A defensive `with_python_udf_inlining(enabled=True)` on a context that
+        // already inlines would otherwise rebind the session's planner to this
+        // handle's codecs, and callers routinely discard the result. Returning
+        // the codecs as-is is observationally equivalent to the rebuild below,
+        // which wraps the same inner codec in a fresh `Python*Codec`.
+        if self.logical_codec.python_udf_inlining() == enabled
+            && self.physical_codec.python_udf_inlining() == enabled
+        {
+            return Self {
+                ctx: Arc::clone(&self.ctx),
+                logical_codec: Arc::clone(&self.logical_codec),
+                physical_codec: Arc::clone(&self.physical_codec),
+            };
+        }
+
         let logical_codec = Arc::new(
             PythonLogicalCodec::new(Arc::clone(self.logical_codec.inner()))
                 .with_python_udf_inlining(enabled),

docs/source/contributor-guide/ffi.md

@@ -351,7 +351,29 @@ that planner against the new codec for the same reason: there is one planner, an
 to carry the codecs currently in force. This happens on the shared session, so it takes
 effect even if the returned context is discarded — `ctx.with_python_udf_inlining(...)`
 whose result is thrown away still leaves the session's planner carrying the codecs of
-that discarded handle.
+that discarded handle. A call that changes nothing is exempt: asking for the inlining
+setting a context already has returns a handle without touching the session.
+
+The rule that falls out of this is worth stating on its own, because it is the one thing
+that surprises people:
+
+> The session's query planner carries the codecs of the handle that most recently
+> installed one. Every other path — `Expr.to_bytes(ctx)`, `ExecutionPlan.to_bytes(ctx)`,
+> registering a provider — uses the codecs of the handle you call it on.
+
+Those can be different handles, and then one session has two codecs in effect at once:
+
+```python
+ctx = ctx.with_logical_extension_codec(codec_a)
+ctx.set_query_planner(planner)
+ctx.with_logical_extension_codec(codec_b)  # discarded
+
+Expr.to_bytes(expr, ctx)   # encodes with codec_a -- ctx's own field
+ctx.sql(...).collect()     # plans with codec_b -- installed via the discarded handle
+```
+
+Chaining `ctx = ctx.with_...(...)`, as the example below does, keeps the two in step.
+`test_the_planner_and_the_handle_can_hold_different_codecs` pins the divergence.
 
 ```python
 ctx = SessionContext(config)

examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py

@@ -19,8 +19,9 @@
 
 import gc
 
+import pyarrow as pa
 import pytest
-from datafusion import SessionConfig, SessionContext, udf
+from datafusion import Expr, SessionConfig, SessionContext, col, udf
 from datafusion_ffi_example import (
     IsNullUDF,
     MyCatalogProvider,
@@ -535,6 +536,85 @@ def test_a_discarded_derived_context_still_rebinds_the_planner():
     assert later.table_provider_encode_calls() > 0
 
 
+def test_the_planner_and_the_handle_can_hold_different_codecs():
+    """One session, two codecs in effect, depending on the path taken.
+
+    The planner is session state and carries whichever codecs installed it
+    last -- here, ones that arrived through a handle that was discarded.
+    Everything else on a context uses that context's own codec field, which
+    the discarded handle never touched. So `Expr.to_bytes(ctx)` and
+    `ctx.sql(...)` encode with different codecs on the same `ctx`.
+
+    Chaining ``ctx = ctx.with_...(...)`` keeps the two in step; this pins what
+    happens when they are allowed to diverge.
+
+    Inlining has to be off for the assertion to say anything: with it on, a
+    Python UDF is encoded inline by ``PythonLogicalCodec`` and never reaches
+    the installed codec's ``try_encode_udf``.
+    """
+    config = SessionConfig().with_extension(MyPlannerConfig(max_rows=3))
+    handle_codec = MyLogicalExtensionCodec()
+    ctx = SessionContext(config).with_python_udf_inlining(enabled=False)
+    ctx = ctx.with_logical_extension_codec(handle_codec)
+    ctx = ctx.with_physical_extension_codec(MyPhysicalExtensionCodec())
+    ctx.register_table("numbers", MyTableProvider(1, 6, 1))
+    ctx.set_query_planner(MyQueryPlanner())
+
+    planner_codec = MyLogicalExtensionCodec()
+    # Discarded, but the planner keeps its codec.
+    ctx.with_logical_extension_codec(planner_codec)
+    gc.collect()
+
+    identity = udf(
+        lambda arr: arr,
+        [pa.int64()],
+        pa.int64(),
+        volatility="immutable",
+        name="identity_i64",
+    )
+    ctx.register_udf(identity)
+    Expr.to_bytes(identity(col("A")), ctx)
+
+    # Serializing through `ctx` uses `ctx`'s own codec field.
+    assert handle_codec.encode_udf_calls() > 0
+    assert planner_codec.encode_udf_calls() == 0
+
+    ctx.sql('SELECT "A" FROM numbers ORDER BY "A"').collect()
+
+    # Planning through the same `ctx` uses the codec the planner was rebound to.
+    assert planner_codec.table_provider_encode_calls() > 0
+    assert handle_codec.table_provider_encode_calls() == 0
+
+
+def test_an_unchanged_inlining_setting_leaves_the_planner_alone():
+    """A no-op toggle must not rebind the session's planner.
+
+    ``with_python_udf_inlining`` rebuilds the handle's codecs and rebinds the
+    session's planner to them. Asking for the setting a context already has
+    changes nothing, so it must not pay that side effect.
+
+    Observable only once the planner is holding some *other* handle's codec:
+    without the guard, a defensive no-op toggle on `ctx` drags the planner back
+    onto `ctx`'s codec and silently undoes the install below. The rebuilt
+    codecs otherwise wrap the same inner codec, so nothing else distinguishes
+    the two paths.
+    """
+    ctx, handle_codec, _physical_codec = codec_context()
+    ctx.set_query_planner(MyQueryPlanner())
+
+    planner_codec = MyLogicalExtensionCodec()
+    ctx.with_logical_extension_codec(planner_codec)  # discarded; planner keeps it
+    gc.collect()
+
+    # The default is on, so this asks for what `ctx` already has.
+    ctx.with_python_udf_inlining(enabled=True)
+    gc.collect()
+
+    ctx.sql('SELECT "A" FROM numbers ORDER BY "A"').collect()
+    assert planner_codec.table_provider_encode_calls() > 0
+    assert handle_codec.table_provider_encode_calls() == 0
+
+
 def test_reinstalling_a_planner_rebinds_the_session_to_that_handles_codecs():
     """A planner is built against the codecs of the handle it is installed from.