Submitter SDK on Python 3.13 fails against Python 3.12 endpoint (WorkerLost on default serializer)

[rajeeja]

Summary

When a globus-compute-sdk submitter is on a different Python minor version
than the endpoint's worker (e.g. SDK on 3.13, worker on 3.12), tasks fail
with WorkerLost for any non-trivial payload. The failure mode is
inconsistent and hard to diagnose:

• Tasks that round-trip simple stdlib types (small dicts of strings) often
  succeed even across the mismatch.
• Tasks that close over inline functions submitted via Executor.submit()
  with the default serializer fail with WorkerLost on the worker, with a
  message suggesting AllCodeStrategies as the workaround.
• Switching to AllCodeStrategies resolves the function-serialization
  side but not the argument/return value pickle protocol differences. The
  underlying Python pickle protocol diverges in places between minor
  versions (stdlib type internals, some collection types), and there's no
  user-facing way today to know which payloads will or won't survive.

The net effect is that a single endpoint can only reliably serve submitters
on its own Python minor version. For a research-software MCP server like
uxarray-mcp where the
endpoint sits on an HPC cluster (Slurm/PBS-managed conda env, often Python
3.12 because that's what the site's stack ships) and submitters are on
researcher laptops (currently Python 3.13 on fresh macOS installs), this
forces every user to pin their laptop install to match — or get cryptic
runtime errors much later in their workflow.

Concrete reproduction

Endpoint (chrysalis at LCRC, conda env uxarray-yac on Python 3.12.13):

display_name: chrysalis
engine:
  type: GlobusComputeEngine
  provider:
    type: SlurmProvider
    # ...

Submitter (macOS, Python 3.13.8, globus-compute-sdk==4.12.0):

from globus_compute_sdk import Executor
from globus_compute_sdk.serialize import AllCodeStrategies, ComputeSerializer

def _ls(path):
    import os
    return sorted(os.listdir(path))

ex = Executor(
    endpoint_id="<chrysalis-uuid>",
    serializer=ComputeSerializer(strategy_code=AllCodeStrategies()),
)
fut = ex.submit(_ls, "/lcrc/group/e3sm/.../mesh")
print(fut.result(timeout=180))

Result on 3.13 → 3.12: WorkerLost: Task failure due to loss of worker N on host chr-XXXX. Same code on 3.12 → 3.12 returns the directory listing
in ~5s.

The warning the SDK already prints on submission is accurate:

Environment differences detected between local SDK and endpoint workers:
        SDK: Python 3.13.8/Dill 0.3.9
    Workers: Python 3.12.13/Dill 0.3.9
This may cause serialization issues.

— but only "may" is doing a lot of work there. In practice it's "will,
for anything beyond trivial payloads."

Prior art on this tracker

• Can not use globus_compute_sdk with Python 3.14 or greater #2039 — SDK fails on Python 3.14+
• Globus Compute hosted services vs source build of non-released Python 3.12.6+ #1999 — Behavior diff between hosted services and source-built 3.12.6+
• dill 0.3.5.1 vs python 3.11 vs globus-compute-sdk 2.2.1 #1197 — dill 0.3.5.1 / Python 3.11 / SDK 2.2.1 incompatibility

These all point at the same underlying problem: there's no published
support matrix for "which submitter Python X.Y is compatible with which
endpoint Python A.B", and the failure modes are inconsistent enough that
end users learn the constraint by hitting it in production.

Asks (any one of these helps; in priority order)

1. A documented compatibility matrix for submitter Python ↔ worker
   Python, even if the answer is "must match minor version." Today the
   docs say "may cause serialization issues" without committing to what's
   tested or supported.
2. A pre-flight compatibility check on Executor() construction that
   either raises or warns loudly (not as a regular UserWarning that's
   easy to filter) when versions don't match. The information is already
   in the SDK (see Change version mismatch message, add logic to ignore Python micro # if desired #1601's text change); turn it into a structured field
   downstream tools can act on.
3. Long-term: a serialization strategy that's actually portable across
   minor versions — e.g., a documented "safe payload subset" (JSON +
   numpy arrays via a stable representation) that's guaranteed to round-
   trip regardless of submitter/worker Python. AllCodeStrategies solves
   the function half; we need the data half.

Downstream impact

We pin our package to a specific Python minor version in our docs to work
around this, because we can't tell users to follow a different version
recipe than every HPC site they connect to. Each version-pin in a
research-software package is a maintenance burden that ripples to every
user; an upstream fix or matrix would let us declare `requires-python`
honestly and let users mix.

Happy to provide more logs, dill/pickle traces, or run additional repro on
chrysalis if useful.

— filed from work on https://github.com/UXARRAY/uxarray-mcp-server

[rajeeja]

Downstream pin shipped: UXARRAY/uxarray-mcp-server#61. The repo now constrains `requires-python = ">=3.12,<3.13"` with an inline comment pointing back here, so when this issue moves we can lift the pin in one place. Happy to test any candidate fix against the live chrysalis (Slurm, 3.12) and NCAR Casper (PBS, 3.12) endpoints we operate.

[khk-globus]

Hi Rajeeja,

Unfortunately, this is a well-known gap that is endemic to serialization of in-memory objects, and is not a small problem to address. We do document this detail, and we have suggestions for how to work within this reality,¹²³ but this is a difficult problem that we are still working on.

One approach that may (there's that word, pulling a heavy load!) help with running of the function, is the .register_source_code() method. This was introduced in v3.16.0. In short, submit raw text that will be compiled on the remote side. (This is in contrast to the .submit() call, which will serialize the in-memory byte-codes from the SDK side, to be deserialized and run on the remote side.)

For example, consider this unrelated-to-your-script file:

$ cat my_mod.py
import os  # no longer inside the function

def peek_at_host(*args, **kwargs) -> dict[str, t.Any]:
    return {
      "function_args": args,
      "function_kwargs": kwargs,
      "core_count": os.cpu_count(),
      "physical_core_count": psutil.cpu_count(logical=False),
      "cores_available": os.sched_getaffinity(0),
      "virtmem": psutil.virtual_memory(),
      "uname": os.uname(),
    }

Using it in your script might look like:

from globus_compute_sdk import Executor

script_text = open("my_mod.py").read()
entry_point = "peek_at_host"
script_desc = "Not required, but always recommended: a description"

# not necessary, and a function is not actually required -- any callable should
# work -- but using a simple "typo-protection" approach here for example purposes
assert f"def {entry_point}" in script_text

with Executor(...) as ex:
    func_id = ex.register_source_code(script_text, entry_point, description=script_desc)
    future = ex.submit_to_registered_function(func_id)
    print(future.result())

There's still the issue of the result serialization, as that will have been serialized by the remote Python instance and will be deserialized by the local Python. Please look to the links I provided in this comment's footnotes for options.

As for your specific asks ... I'll simply state that "there are some details there," and documentation is the most easily approachable.

Footnotes

1. Executor User Guide, Submit-Side Serialization Strategies ↩

2. Executor User Guide, Result Serialization Strategies ↩

3. Endpoint User Guide, Restricting Serialization Strategies ↩