Skip to content

Sweep add-ons: seeded repeats, local max_parallel, W&B grouping - #106

Open
aaronsmulktis wants to merge 9 commits into
mainfrom
aaronsmulktis/sweep-addons
Open

aaronsmulktis wants to merge 9 commits into
mainfrom
aaronsmulktis/sweep-addons

Conversation

@aaronsmulktis

@aaronsmulktis aaronsmulktis commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Changes:

  1. launch_parallel_agents.py updated for local and on-SLURM usage; scripts/sweep_slurm.sh AKA conduct_slurm.sh removed.
  2. agent.hostname=auto finds the node serving each agent's model among your running SLURM jobs (squeue --me, then /v1/models on agent.port), for mode=local and mode=slurm_cluster alike.
  3. parallel_tasks.repeats=N runs every job N times with the seed fixed, to measure run-to-run variance. W&B run names get -r<N>.
  4. max_parallel=N runs local jobs N at a time, each in its own spawned process writing to <job>/run.log, with a summary at the end. Default 1 keeps the current sequential behaviour; inside an srun allocation it replaces the wrapper script.
  5. With use_wandb=True, a sweep's runs share a W&B group named after its run-<id> directory, with the agent as job_type.
  6. Per-job port windows wrap every 500 jobs, so jobs ≥ 606 keep a window of their own instead of a random port.
  7. Docs: docs/agents.md gives the SLURM commands (one job per run, or one allocation) and a dummy-agent check of each.
  8. Ctrl-C stops a local sweep: queued runs never start, running ones stop their web apps and exit (exit 130; Ctrl-C again kills them now). A process per run replaces ProcessPoolExecutor, whose shutdown ran every queued job first
  9. One Ctrl-C under uv run arrives twice (uv forwards it); the echo is ignored, not taken as "kill now"
  10. Fix: the launcher's 30s startup wait was outside launch()'s try/finally, so interrupting it leaked the web app (single runs too); stopping the web app now ignores Ctrl-C (bounded by SIGKILL)
  11. README and docs/index.md explain agents=, repeats, max_parallel, SLURM with agent.hostname=auto, W&B grouping and stopping a sweep

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 29, 2026
@aaronsmulktis aaronsmulktis self-assigned this Oct 29, 2026
@aaronsmulktis
aaronsmulktis marked this pull request as ready for review October 29, 2026 20:44
Base automatically changed from marksibrahim/parallel-task-agent-sweeps to main October 30, 2026 15:14
aaronsmulktis and others added 2 commits October 30, 2026 17:25
…nt.hostname=auto, no sweep_slurm.sh

agent.hostname=auto finds the node serving each agent's model among the
user's running SLURM jobs (squeue --me, then /v1/models on agent.port), once
per model, for mode=local and mode=slurm_cluster alike. That was the part of
scripts/sweep_slurm.sh the launcher lacked; the rest is mode=local
max_parallel=N inside an srun allocation, so the script is removed. The
agents docs give both SLURM commands and a dummy-agent check of each.
@aaronsmulktis aaronsmulktis changed the title Sweep add-ons: seeded repeats, local max_parallel, W&B grouping, sweep_slurm.sh (on #105) Sweep add-ons: seeded repeats, local max_parallel, W&B grouping Nov 17, 2026

@marksibrahim marksibrahim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, thanks for extending the parallel launcher with these nice features.

  1. Could you make sure the documentation explains the new functionality?
  2. Launching parallel jobs locally breaks the ability of Ctlr-C to stop a sweep. Consider catching KeyboardInterrupt (or BaseException) and call pool.shutdown(cancel_futures=True), or cancel the pending futures.

run_concurrently: a process per job instead of a ProcessPoolExecutor, whose
shutdown ran every queued job before exiting and whose pre-queued job could
not be cancelled. Ctrl-C now starts no further job and SIGTERMs the running
ones, which unwind through their finally (stopping their web apps); a second
Ctrl-C kills them, and uv's echo of the first is not mistaken for one.

AgentLauncher: the 30 s startup wait moves inside launch()'s try/finally, so
an interrupt during it no longer leaks the web app, and stopping the web app
ignores Ctrl-C (it is bounded by SIGKILL).

README and docs/index.md explain agents=, repeats, max_parallel, SLURM with
agent.hostname=auto, W&B grouping and stopping a sweep.
@aaronsmulktis

aaronsmulktis commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Looks good overall, thanks for extending the parallel launcher with these nice features.

  1. Could you make sure the documentation explains the new functionality?
  2. Launching parallel jobs locally breaks the ability of Ctlr-C to stop a sweep. Consider catching KeyboardInterrupt (or BaseException) and call pool.shutdown(cancel_futures=True), or cancel the pending futures.

ProcessPoolExecutor ditched for individual processes with cleaner Cntrl+C termination, safer also when one worker dies/fails the queue of others continues, and the docs updated for the new parallelism ergonomics.

The sweep's handler covers SIGTERM as well as SIGINT, so `kill <pid>`
stops queued and running jobs the way Ctrl-C does (exit 128 + signal), in
both the pooled and the one-at-a-time paths. Off the main thread, where no
handler can be installed, jobs take Ctrl-C themselves and exit 130, which
stops the sweep. A job ignores signals after the first, so the parent's
SIGTERM cannot cut short a cleanup a Ctrl-C started.
@marksibrahim

marksibrahim commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

@aaronsmulktis One other concern post the recent CTRL-C fix is that the web app may keep running: The web app runs in its own session (start_new_session=True), so the kill doesn't reach it. It keeps serving on the job's port until someone kills it by hand. In other words, we want to stop the web apps without depending on the job shutting down cleanly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants