Bug report: Invalid account launch_slurm Jupiter

[Matteo-Salis]

What happened?

Hi,
I'm starting to work on Jupiter, and I tried to submit a slurm job with ../WeatherGenerator-private/hpc/launch-slurm.py --stage inference --from-run-id fbclk6da --config ./config/inference/inference_era_o96_cerra_config.yml --account=e-ext-2025e01-128 and I get this error:
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

Actually, I have access to Jupiter through the "e-ext-2025e01-128" project. However, it seems that "weatherai" account is used by default for the cleanup job, and the --account=e-ext-2025e01-128 is not taken into account for it.

In my .bashrc file I have: jutil env activate -p e-ext-2025e01-128

What are the steps to reproduce the bug?

No response

Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.

No response

[simone99n]

Could you run the following command in your Jupiter terminal?

sacctmgr show assoc where user=$USER format=account,partition

It should show you the account/partition combinations available to your user, and the output should be:

$ sacctmgr show assoc where user=$USER format=account,partition
   Account  Partition 
---------- ---------- 
e-ext-202+    booster 
 weatherai    booster 

[Matteo-Salis]

My output is:

$ sacctmgr show assoc where user=$USER format=account,partition
Account  Partition 
---------- ---------- 
e-ext-202+    booster 

It seems that within the weatherai project, I do not have access to the booster partition. Indeed, on the web page "https://judoor.fz-juelich.de/projects/weatherai", I see: "Available, but no access granted: JUPITER_BOOSTER ". I requested access to it.

In any case, the launch_slurm.py script seems to have the weatherai project hard-coded, and it is not possible to specify another account using the --account argument.
The only way to change it is by modifying the field secret.post_train.slurm_account: in "./WeatherGenerator-private/hpc/jupiter/config/paths.yml"

[simone99n]

For training, inference, and evaluation jobs, the account is selected in the following order of preference: CLI or pipeline slurm_args > BUDGET_ACCOUNTS environment variable > default ""

For cleanup jobs, the account is hard-coded in secret.post_train.slurm_account: I don’t recall if there was a specific reason why the cleanup job account was left hard-coded, but I can easily change it!

CC @clessig @tjhunter @grassesi