Skip to content

slurm: fix session-stack lifetime and other plm/ras correctness bugs - #2615

Merged
rhc54 merged 7 commits into
openpmix:masterfrom
Petter-Programs:ras-slurm-fixes
Aug 5, 2026
Merged

slurm: fix session-stack lifetime and other plm/ras correctness bugs#2615
rhc54 merged 7 commits into
openpmix:masterfrom
Petter-Programs:ras-slurm-fixes

Conversation

@Petter-Programs

@Petter-Programs Petter-Programs commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #2546, plus a handful of related correctness issues found while
working through ras/slurm and plm/slurm.

  • plm/slurm: the "using the whole allocation?" check compared
    map->num_new_daemons against session->nodes->size — a pointer
    array's capacity, not its populated count — so it emitted an
    explicit --nodes/--nodelist more often than intended.
  • ras/slurm: prte_slurm_session_stack held a bare, unretained
    prte_session_t *, relying on the stack always popping an item
    before releasing its session. DVM finalize's blanket session release
    doesn't go through this stack, so a tracked session could be freed
    out from under it. The stack now retains its own reference on push
    and releases it via the stack item's destructor on every removal.
  • ras/slurm: two spots (add_reused_nodes_to_session,
    add_pending_req) fed pmix_pointer_array_add's return value — an
    index, not a pmix_status_t — into prte_pmix_convert_status(),
    producing a meaningless error code on failure.
  • ras/slurm: guard remove_nodes_by_count against ever computing
    zero nodes to remove in the "shrink and keep the job alive" branch;
    should be unreachable given the surrounding invariant, but the
    alternative (a NULL target list, or worse, an infinite shrink loop)
    is worse than failing loudly.
  • ras/slurm: fixed a stale comment on kill_job's err_msg
    behavior.
  • test/ras: updated the slurm module contract test for the
    session-stack change (below).
  • examples/slurm: split the dvm command out of the reachability
    probe (previously run on every state query, so a probe failure was
    difficult to distinguish from an allocation failure) and dropped the
    README's srun wrapper around prterun, which is non-standard and
    could fail with all slots occupied.

release_allocation is now unimplemented everywhere

The session-stack retain change removes ras/slurm's
.release_allocation vtable entry — the reactive hook that decided
whether to scancel by checking if a session's item was still on the
stack. With the stack now holding a real reference, session_des can
never fire while an item is still linked, so the hook was structurally
dead; the scancel decision moves to being made explicitly at each
removal site instead.

ras/slurm was the only component that ever set
.release_allocation — no other ras component (hosts, pbs,
lsf, gridengine, flux, pmix, bootstrap, simulator,
testrm) implements it. Removing our one call site means the generic
plumbing around it — the release_allocation vtable slot itself,
prte_ras_base_release_allocation(), and its unconditional call from
session_des() on every session teardown, HNP and daemons alike — is
now dead code framework-wide. Left in place for this PR (only the
now-invalid unit test assertion was updated); flagging it as a
follow-up cleanup candidate rather than pulling it out here.

Testing

  • Clean --enable-debug build, no warnings.
  • make check (test/unit/ras/test_ras) passes.
  • Session-stack retain/release change live-tested on a 3-node Slurm
    cluster (extend, shrink, and DVM shutdown mid-extension).

Credits: AccelCom @ Barcelona Supercomputing Center

The "are we using the whole allocation?" check compared
map->num_new_daemons against session->nodes->size, but size is the
pointer array's allocated slot count, not the number of nodes actually
in the session, so it emitted an explicit --nodes/--nodelist more often
than intended; count only the populated entries instead. Drop the
matching "known wart" note now that the undercount it described is
fixed.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
prte_slurm_session_stack stored a bare, unretained prte_session_t *, on
the assumption that ras/slurm's own rollback/shrink paths would always
pop an item before releasing its session - but DVM finalize's blanket
release of every session in prte_sessions doesn't go through this stack,
so a session could be freed while still tracked, leaving a dangling
entry. Retain on push instead, and let the stack item's own destructor
release it, so every removal site just pops and releases the item; the
reactive release_allocation hook, which decided whether to scancel by
checking if an item was still on the stack, is now replaced by an
explicit drain at component finalize, since a session's last reference
can no longer be dropped from outside the stack. That deferral also
means a session's final release can now outlive prte_sessions itself,
so guard session_des against the array already being torn down.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
pmix_pointer_array_add returns the assigned index, not a pmix_status_t,
so feeding a negative result to prte_pmix_convert_status() produced a
meaningless error code. add_reused_nodes_to_session and
prte_ras_slurm_add_pending_req both did this; follow the correct model
already used in assign_new_session and just treat a negative return as
PRTE_ERR_OUT_OF_RESOURCE.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
nodes_to_rem_from_session should always be >= 1, since
find_releasable_session only returns sessions with removable_count > 0;
guard against that invariant breaking instead of risking a NULL target
list or an infinite shrink loop.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
The comment claimed only the first line of output is copied into
err_msg, but the code actually copies the full output up to the
buffer size.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
release_allocation was removed from ras/slurm's vtable when the session
stack started retaining its own reference (see the session-stack retain
commit), but this test still asserted it was set; assert it's unset
instead.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
Reachability probing was a common failure point but ran folded into
every state query; split it into an opt-in dvm command so a probe
failure is unambiguous. Also drop the README's srun wrapper around
prterun - it's non-standard and could fail with all slots occupied.

Credits: AccelCom @ Barcelona Supercomputing Center

Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
@rhc54
rhc54 merged commit 1dbb742 into openpmix:master Aug 5, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

plm/slurm: whole-allocation test compares a daemon count against a pointer-array capacity

2 participants