slurm: fix session-stack lifetime and other plm/ras correctness bugs - #2615
Merged
Conversation
The "are we using the whole allocation?" check compared map->num_new_daemons against session->nodes->size, but size is the pointer array's allocated slot count, not the number of nodes actually in the session, so it emitted an explicit --nodes/--nodelist more often than intended; count only the populated entries instead. Drop the matching "known wart" note now that the undercount it described is fixed. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
prte_slurm_session_stack stored a bare, unretained prte_session_t *, on the assumption that ras/slurm's own rollback/shrink paths would always pop an item before releasing its session - but DVM finalize's blanket release of every session in prte_sessions doesn't go through this stack, so a session could be freed while still tracked, leaving a dangling entry. Retain on push instead, and let the stack item's own destructor release it, so every removal site just pops and releases the item; the reactive release_allocation hook, which decided whether to scancel by checking if an item was still on the stack, is now replaced by an explicit drain at component finalize, since a session's last reference can no longer be dropped from outside the stack. That deferral also means a session's final release can now outlive prte_sessions itself, so guard session_des against the array already being torn down. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
pmix_pointer_array_add returns the assigned index, not a pmix_status_t, so feeding a negative result to prte_pmix_convert_status() produced a meaningless error code. add_reused_nodes_to_session and prte_ras_slurm_add_pending_req both did this; follow the correct model already used in assign_new_session and just treat a negative return as PRTE_ERR_OUT_OF_RESOURCE. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
nodes_to_rem_from_session should always be >= 1, since find_releasable_session only returns sessions with removable_count > 0; guard against that invariant breaking instead of risking a NULL target list or an infinite shrink loop. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
The comment claimed only the first line of output is copied into err_msg, but the code actually copies the full output up to the buffer size. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
release_allocation was removed from ras/slurm's vtable when the session stack started retaining its own reference (see the session-stack retain commit), but this test still asserted it was set; assert it's unset instead. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
Reachability probing was a common failure point but ran folded into every state query; split it into an opt-in dvm command so a probe failure is unambiguous. Also drop the README's srun wrapper around prterun - it's non-standard and could fail with all slots occupied. Credits: AccelCom @ Barcelona Supercomputing Center Signed-off-by: Petter Sandås <petter.sandas@bsc.es>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2546, plus a handful of related correctness issues found while
working through
ras/slurmandplm/slurm.map->num_new_daemonsagainstsession->nodes->size— a pointerarray's capacity, not its populated count — so it emitted an
explicit
--nodes/--nodelistmore often than intended.prte_slurm_session_stackheld a bare, unretainedprte_session_t *, relying on the stack always popping an itembefore releasing its session. DVM finalize's blanket session release
doesn't go through this stack, so a tracked session could be freed
out from under it. The stack now retains its own reference on push
and releases it via the stack item's destructor on every removal.
add_reused_nodes_to_session,add_pending_req) fedpmix_pointer_array_add's return value — anindex, not a
pmix_status_t— intoprte_pmix_convert_status(),producing a meaningless error code on failure.
remove_nodes_by_countagainst ever computingzero nodes to remove in the "shrink and keep the job alive" branch;
should be unreachable given the surrounding invariant, but the
alternative (a NULL target list, or worse, an infinite shrink loop)
is worse than failing loudly.
kill_job'serr_msgbehavior.
session-stack change (below).
dvmcommand out of the reachabilityprobe (previously run on every state query, so a probe failure was
difficult to distinguish from an allocation failure) and dropped the
README's
srunwrapper aroundprterun, which is non-standard andcould fail with all slots occupied.
release_allocationis now unimplemented everywhereThe session-stack retain change removes
ras/slurm's.release_allocationvtable entry — the reactive hook that decidedwhether to
scancelby checking if a session's item was still on thestack. With the stack now holding a real reference,
session_descannever fire while an item is still linked, so the hook was structurally
dead; the
scanceldecision moves to being made explicitly at eachremoval site instead.
ras/slurmwas the only component that ever set.release_allocation— no otherrascomponent (hosts,pbs,lsf,gridengine,flux,pmix,bootstrap,simulator,testrm) implements it. Removing our one call site means the genericplumbing around it — the
release_allocationvtable slot itself,prte_ras_base_release_allocation(), and its unconditional call fromsession_des()on every session teardown, HNP and daemons alike — isnow dead code framework-wide. Left in place for this PR (only the
now-invalid unit test assertion was updated); flagging it as a
follow-up cleanup candidate rather than pulling it out here.
Testing
--enable-debugbuild, no warnings.make check(test/unit/ras/test_ras) passes.cluster (extend, shrink, and DVM shutdown mid-extension).
Credits: AccelCom @ Barcelona Supercomputing Center