Stuck merge queue postmortem

[parshimers]

Hi folks,
In case you weren't aware, for the past 36 or so hours, no merges were going into the main repo (or any other repo). The reason for this, is because we require every change to go through the merge queue on GitHub before being merged to main. The merge queue timeout is set to 45 minutes. That means if the Actions required for a PR to be merged from the queue aren't started before that, the PR will be ejected from the merge queue.

When you combine that limit with the runner starvation in ASF, it manifests into the situation we had, where no change can be merged because it is not assigned a runner in time, and is simpy ejected. This is compounded by the fact that nobody is permitted to bypass the merge queue. It's a very deep hole that we were in. Humbedooh in Infra was able to (hopefully) teleport us out of it by bypassing the merge queue to commit 2407d79ee7a94ac92c65c720254566c25f69dd8a on our behalf. This workaround simply raises the time limit. Again we received community help on this part, Andrew Lamb of DataFusion saw the situation we were in and pointed out they had done this to help their quandry. 

Now we come to the part where we need to decide on what to do to stop this from happening. Clearly what happened is not acceptable. There are situations where code needs to be urgently merged (e.g. CVEs, etc.) and having this happen during one of those fires would be a disaster. Therefore we need a "break glass" kind of relief valve.

I think there are 2 things we can do to stop this situation from happening. One, is to merge Yicong's change to allow the self-hosted runners to run most of our jobs. That will be the primary fix. The second, is to allow some subset of folks in the project the privilege to bypass the merge queue or other restrictions, in an emergency. I wasn't sure if this was possible, but after discussions with Humbedooh of ASF Infra it seems like it is.

If we are all agreed on both points, I would like to start a discussion within the PPMC on private@ as to who should be granted this authority for emergency situations, like we experienced. 

Looking forward to hearing everyone's thoughts.

[aglinxinyuan]

We may need to switch to a self-hosted runner, as increasing the timeout may not help if the runner is too busy. For example, this PR has already been waiting in the queue for four hours.

[chenlica]

I support the first idea, if it's easy to set up. If there is no downside for the second option, I am OK with it as well.

[Yicong-Huang]

I think both points are valid.

• We need self-hosted runners, at least for backup when github runners are not available. But one thing: self runner cannot run macos CI (it just does not have the image), which means if we make macOS CI required, it will likely to be blocked by github default runners still.
• I'd like to propose give release managers the emergency merge privilege. I knew this was technically possible but I failed with previous attempts with @xuang7 on her v1.2 release manager assignment. I tried to include her github account but she seems not able to perform more actions. If @parshimers or someone can help figure out how to do it technically, that'd be nice. We can discuss about the candidates later, but he/she must be a committer at least.

[parshimers]

I will look into how to allow certain folks to bypass the privilege, I think I know how after reading the .asf.yaml processor source code a bit. I will follow up with a proposal on who should be allowed to do so on private@

[xuang7]

I agree that we need an emergency path. With the merge queue as the only way to merge changes and nobody able to bypass it, a period of runner unavailability could block the entire project. Having no fallback doesn't seem ideal.

Some background on the earlier attempt Yicong mentioned: During my v1.2 release manager assignment, #5354 and #5413 added my account as a bypass actor in the Default Branch Protection ruleset. I could create branches as usual, but I still couldn't delete a protected release branch. While this doesn't necessarily mean the bypass configuration wasn't applied, it does raise some questions about how the permissions actually work. Before deciding who should have bypass privileges, I think it would be worth asking an admin to confirm that bypass_actors in .asf.yaml are actually being applied, and then testing the bypass with a harmless PR.

[Ma77Ball]

I strongly agree that we should have someone who can bypass the merge queue in emergencies.

I think Self-hosted runners are a tricky situation. One thing I can't wrap my head around is the ASF policy on running on secure, trusted hardware. It seems past efforts to do this were in tandem with the ASF. @parshimers, any thoughts on the procedure or best practice when approaching something like this?

[parshimers]

I myself am a bit puzzled about it. I think basically we are already at the point, somehow, where the API key necessary to add the runners to this repo was given (I would assume to @Yicong-Huang) and each of the runners was started with it.
The part I don't fully grasp is that as you mentioned, there are certain restrictions on the runners. I assume this is because they can have credentials on them that are scoped wider than our repo. However even in that case it seems weird to me, it isn't as if running it in AWS or similar is really that much of an improvement overall. You're still on your own w.r.t keeping things secure.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173082816&spaceKey=INFRA&title=GitHub%2Bself-hosted%2Brunners has a lot of good tidbits.