keep-workflows-enabled: Pathogen workflows not running as scheduled

[joverlee521]

The workflows for hmpv, measles, and oropouche have not run as scheduled since Dec 8, which coincides with the merge of #151 + the post-merge run of keep-workflows-enabled. There were no errors in the keep-workflows-enabled run and the pathogen workflows themselves are still active.

[joverlee521]

I wonder if there's an issue with nextstrain-bot being the "owner" of too many workflows so I manually disabled/enabled the workflows under my GH user and will see if they run as scheduled tomorrow.

[joverlee521]

The workflows ran as scheduled today

• hmpv
• measles
• oropouche

Manually triggered the keep-workflows-enabled workflow to see what happens tomorrow when they're switched back to the nextstrain-bot user.

[joverlee521]

Sigh, today there were different pathogens that did not run as scheduled: norovirus and yellow-fever.

From GH Action docs (emphasis mine):

The schedule event can be delayed during periods of high loads of GitHub Actions workflow runs. High load times include the start of every hour. If the load is sufficiently high enough, some queued jobs may be dropped. To decrease the chance of delay, schedule your workflow to run at a different time of the hour.

I think the only way to reliably run the scheduled workflows is to spread out the schedules.

[joverlee521]

Confirmed today that norovirus and yellow-fever did not run again. I'm not sure what determines which jobs are dropped, but they seem to stay consistent after a run of keep-workflows-enabled.

The 5 pathogens that have been affected so far are one of the 17 workflows scheduled to run at 9AM (GH search). They were previously spread out among at least 3 different "owners" so not as much of an issue. I'm going to update 8 of the workflows to run at 9:15 and then re-run keep-workflows-enabled to see if that is enough to resolve the issue.

[joverlee521]

Oh weird. The norovirus ingest ran to day as scheduled, but the "owner" of the workflow was @j23414. @j23414 did you disable/enable the workflow manually to trigger it?

[j23414]

Weird, I did not disable/enable the workflow manually to trigger it