Expectations for first-time Superset pull request authors in the age of AI

[sfirke]

Background

This is prompted by a few PRs I've seen recently after tagging an issue "good first issue" and by these two blog posts:

• The Zig project's take on Contributor Poker and AI
• Simon Willson's writeup of that post

I am not proposing a ban on AI-generated code from new contributors. For one thing, the majority of top contributors to Superset seem to be using AI to write and review code and even to comment on GitHub issues. I write my code contributions with AI. But I think there is a phenomenon to discuss.

Recent Superset experiences

The blog posts note that we (Superset committers) can probably move the project faster if we write PRs ourselves rather than review and give feedback on PRs from first-time authors. The value proposition for us is that we're growing potential long-term contributors who will over time give more to the project than they require in review.

That's why I tag issues "good first issue" instead of just fixing them myself, to get new people involved (at a time cost to myself as I work with the new author).

It raises the question of what we expect from first-time PR authors. I have recently seen PRs that consist only of:

• An AI-generated code output that has not been manually tested
• AI-generated comments in the PR

When a human generates the AI code they can still add value: manually testing the output via a local Superset deployment, working through edge cases, or adding human comments to an issue or PR. These authors did none of these.

I have twice now asked for screenshots and the contributors admitted they don't have a local instance of Superset running. To me that's unacceptable, to send code that you can't or won't even deploy yourself (except for maybe editing the docs, which was one of the cases). If setting up a dev environment is too big a hurdle then you should find another project to work on.

What to do about it

I would love to have some statement for potential new contributors that sets a certain bar. It's not about AI but it is only relevant in this era -- before AI you knew they had a certain amount invested already.

For starters I would say the expectations include:

• You have checked for existing open PRs that do the same thing
• You have deployed the changes in your own Superset environment and tested the new behavior directly as a user. Include screenshots if relevant
• You understand all of the code you are submitting
• You get pre-commit checks passing before a committer reviews
• If you are using AI, also use that AI to conduct an adversarial review of your code and address anything it finds
• You write in your own voice in the PR, at least a few sentences. If you then want to include AI text, that can follow, with a clear disclaimer like "Here's the AI summary of the changes, I reviewed this and agree with it:"

If your PR does not indicate you've done these things, it may be tagged as lacks-human-authorship and closed

Apache Airflow asks PR authors if they used AI to generate the code. At this point I assume any new code is AI generated so I actually don't think we need that.

Agree? Disagree? Other thoughts?

[rusackas]

I'm definitely not against more rules to "thin the noise". I opened an Elastic Search PR (still waiting for their review), but they have you sign a CLA, and the linting enforces good patterns. I think if we had checks that could qualify a PR by adding/removing a "ready-for-review" tag (based on conflicts, resolved bot comments, passing CI, etc) I'd be into it.

elastic/elasticsearch#141480