Repository navigation
Sweep add-ons: seeded repeats, local max_parallel, W&B grouping - #106
aaronsmulktis wants to merge 9 commits into
Conversation
…ep their own windows
…d process and run.log per job
…launch_parallel_agents sweep
… run dirs and the logs_dir warning)
…nt.hostname=auto, no sweep_slurm.sh agent.hostname=auto finds the node serving each agent's model among the user's running SLURM jobs (squeue --me, then /v1/models on agent.port), once per model, for mode=local and mode=slurm_cluster alike. That was the part of scripts/sweep_slurm.sh the launcher lacked; the rest is mode=local max_parallel=N inside an srun allocation, so the script is removed. The agents docs give both SLURM commands and a dummy-agent check of each.
marksibrahim
left a comment
There was a problem hiding this comment.
Looks good overall, thanks for extending the parallel launcher with these nice features.
- Could you make sure the documentation explains the new functionality?
- Launching parallel jobs locally breaks the ability of Ctlr-C to stop a sweep. Consider catching KeyboardInterrupt (or BaseException) and call pool.shutdown(cancel_futures=True), or cancel the pending futures.
run_concurrently: a process per job instead of a ProcessPoolExecutor, whose shutdown ran every queued job before exiting and whose pre-queued job could not be cancelled. Ctrl-C now starts no further job and SIGTERMs the running ones, which unwind through their finally (stopping their web apps); a second Ctrl-C kills them, and uv's echo of the first is not mistaken for one. AgentLauncher: the 30 s startup wait moves inside launch()'s try/finally, so an interrupt during it no longer leaks the web app, and stopping the web app ignores Ctrl-C (it is bounded by SIGKILL). README and docs/index.md explain agents=, repeats, max_parallel, SLURM with agent.hostname=auto, W&B grouping and stopping a sweep.
|
The sweep's handler covers SIGTERM as well as SIGINT, so `kill <pid>` stops queued and running jobs the way Ctrl-C does (exit 128 + signal), in both the pooled and the one-at-a-time paths. Off the main thread, where no handler can be installed, jobs take Ctrl-C themselves and exit 130, which stops the sweep. A job ignores signals after the first, so the parent's SIGTERM cannot cut short a cleanup a Ctrl-C started.
|
@aaronsmulktis One other concern post the recent CTRL-C fix is that the web app may keep running: The web app runs in its own session (start_new_session=True), so the kill doesn't reach it. It keeps serving on the job's port until someone kills it by hand. In other words, we want to stop the web apps without depending on the job shutting down cleanly. |
Changes:
launch_parallel_agents.pyupdated for local and on-SLURM usage;scripts/sweep_slurm.shAKAconduct_slurm.shremoved.agent.hostname=autofinds the node serving each agent's model among your running SLURM jobs (squeue --me, then/v1/modelsonagent.port), formode=localandmode=slurm_clusteralike.parallel_tasks.repeats=Nruns every job N times with the seed fixed, to measure run-to-run variance. W&B run names get-r<N>.max_parallel=Nruns local jobs N at a time, each in its own spawned process writing to<job>/run.log, with a summary at the end. Default 1 keeps the current sequential behaviour; inside ansrunallocation it replaces the wrapper script.use_wandb=True, a sweep's runs share a W&B group named after itsrun-<id>directory, with the agent asjob_type.docs/agents.mdgives the SLURM commands (one job per run, or one allocation) and a dummy-agent check of each.ProcessPoolExecutor, whose shutdown ran every queued job firstuv runarrives twice (uv forwards it); the echo is ignored, not taken as "kill now"launch()'s try/finally, so interrupting it leaked the web app (single runs too); stopping the web app now ignores Ctrl-C (bounded bySIGKILL)docs/index.mdexplainagents=, repeats,max_parallel, SLURM withagent.hostname=auto, W&B grouping and stopping a sweep