Make the iof's XON/XOFF actually stop the producer

[rhc54]

The consumer side of openpmix/openpmix#4083, which has merged. Makes the iof's XON/XOFF actually stop the producer.

The problem

The iof has carried the shape of stdin back-pressure for years and none of it did anything. A daemon whose stdin sink passed PRTE_IOF_MAX_INPUT_BUFFERS told the HNP so, the HNP logged the message and returned, and the sink went on queueing — bounded only by iof_base_output_limit, which defaults to INT_MAX. #2610 documented that rather than fixing it, because the fix was not ours to make: stdin originates in a PMIx server's read of its own stdin or in a tool's PMIx_IOF_push, and neither end honored a refusal, so returning an error would have dropped bytes rather than slowed anybody.

PMIx now has PMIx_server_IOF_flow_control, which suspends the read at the source — the bytes stay in the producer's own input stream and the OS applies the back-pressure, so nothing is buffered on behalf of a suspended stream and nothing is lost.

Wiring both halves

A daemon's XON/XOFF, which prte_iof_hnp_recv already had to screen ahead of everything else, becomes a wildcard flow-control call. Wildcard because the message says only that this daemon is behind and never which producer filled it, so every process feeding us stdin is suspended — the conservative reading, since any of them may be responsible.

The HNP's own local procs never involve the RML at all. push_stdin returns PRTE_ERR_OUT_OF_RESOURCE when a local sink crosses the same threshold, and pmix_server_stdin_push discarded it. It now turns that into PMIX_ERR_IOF_XOFF on the push_stdin completion, which PMIx reads as "I have the data, suspend the stream" and which is deliberately not a failure — the tool is not told anything is wrong, because nothing is.

The pairing obligation

PMIx has no status meaning "resume", so an XOFF we never release leaves a producer suspended. prte_mca_iof_hnp_component.xoff is the latch, and release_flow_control() runs from every path a backed-up sink can leave that state by: the check: and finish: arms of the write handler, and hnp_close/hnp_complete, which release a stdin sink directly and never reach the write handler at all. That last pair is the case that matters — a sink is backed up precisely when its proc stopped reading, which is the proc a teardown is most likely to be retiring. The helper is a no-op when nothing is outstanding, so it is safe to call from anywhere, which is what lets it be called from everywhere it must be.

Worth being precise about what an unpaired XOFF costs, because it bounds how bad this can get: PMIx keeps no sticky suspension state. pmix_iof_flow_control pushes to the peers that exist at the moment of the call, and the only durable state is the xoff flag on the read event inside the producer. The producer is prun, which is per-job, so the suspension dies with it and a later prun starts reading normally. The worst case is therefore a hung job, not a hung DVM — the next job is unaffected, and the first sink to drain clears the stale latch. That is a property of PMIx's present design rather than something it owes us, so the release paths above are still the contract.

Several procs taking stdin at different rates will oscillate: one draining proc turns the producers back on while another is still behind, and that one asserts XOFF again on its next write. That is intended — the alternative to oscillating is stalling, and nothing is dropped either way. The prted module's CHECK block has carried the same caveat in its own comment since ORTE.

Older PMIx

All of it is gated on PRTE_PMIX_IOF_FLOW_CONTROL, from a new PRTE_CHECK_PMIX_CAP([IOF_FLOW_CONTROL]). Against a PMIx without the capability every path compiles away and the previous behavior returns exactly: the message is consumed quietly and the sink queues. Verified by building both ways.

Testing

The dockerswarm suite gains a pass that counts the two halves in the HNP's log rather than grepping for them, because an implementation that asserts fifty times and releases once is indistinguishable from a correct one under a presence test — and is the hang. It fails on a mismatch, on flow control never engaging at all (which would mean the reader was too fast and the case proved nothing), and on any byte lost across the suspensions.

Verified against a live DVM two ways:

• Throughput — 1 MB of stdin into a reader taking 2 KB every 3 ms produced 37 XOFF assertions and 37 XON releases, exactly paired, with all 1048576 bytes delivered and the checksum intact.
• Teardown — a job whose proc stops reading and then exits while its sink is still backed up released both of its XOFFs, and the two jobs run after it on the same persistent DVM each took their full 256 KB of stdin byte-identically.

Plus a warning-free --enable-debug build (both with and without the capability) and make check clean, rebased onto current master.