docs(pilot-runs): 3 strong upstream PR drafts ready for team greenlight - #51
Open
Zhaiyuqing2003 wants to merge 3 commits into
Open
docs(pilot-runs): 3 strong upstream PR drafts ready for team greenlight#51Zhaiyuqing2003 wants to merge 3 commits into
Zhaiyuqing2003 wants to merge 3 commits into
Conversation
Four polished PR drafts for proposed upstream contributions to public agent skill repos. Each draft is ready to copy-paste into the actual upstream repo after a final review. The actual `git push` to a fork + `gh pr create` is left to the operator. ## Drafts | # | Skill | Target repo | Evidence | |---|---|---|---| | 1 | web-design-guidelines (rules doc) | vercel-labs/web-interface-guidelines | Strong: v1.2.1 measured 0.92→1.00 (+0.08, 18 trials × 3 frontier models) | | 3 | agent-browser Pre-flight | vercel-labs/agent-browser | Soft: v1.0 baseline 0.97; observed 1/9 gemini fell back to curl; preventive 11-line addition | | 4 | supabase-postgres-best-practices | supabase/agent-skills | Soft: per-case update-without-where 77.8% identified; convention-perfect monitor-* reference file | | 5 | shadcn-ui code review checklist | google-labs-code/stitch-skills | Strong: v1.3 orchestrator measured 0.667→0.889 (+0.222, gpt-5 frontier matrix) | Plus `upstream-pr-conventions.md` documenting per-repo title format, body convention, CI gates, license, CLA requirements, and merge-style gotchas for the 4 target upstream repos. ## How to submit each PR 1. Fork the upstream repo (or use an existing fork). 2. Clone the fork locally outside this repo. 3. Make the changes described in the draft on a new branch. 4. Run any local checks the convention doc calls for (e.g. `pnpm test:sanity` for supabase, Google CLA signing for stitch-skills). 5. Commit + push. 6. Open the PR with the title/body from the draft. 7. Link the resulting URL back to the draft for traceability. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor
There was a problem hiding this comment.
Pull request overview
Adds documentation to capture four upstream PR drafts (for external skill repositories) plus an internal conventions guide to help operators submit those drafts consistently.
Changes:
- Added
docs/pilot-runs/upstream-pr-drafts/with a README and 4 draft PR writeups (#1, #3, #4, #5). - Added
docs/pilot-runs/upstream-pr-conventions.mdsummarizing per-repo PR norms (titles, CI, CLA, etc.). - Documented operator “how to submit” steps inside each draft.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/pilot-runs/upstream-pr-drafts/README.md | Index + operator process for the upstream PR drafts |
| docs/pilot-runs/upstream-pr-drafts/1-vercel-labs-web-interface-guidelines.md | Draft PR writeup for web-interface-guidelines checklist addition |
| docs/pilot-runs/upstream-pr-drafts/3-vercel-labs-agent-browser-pre-flight.md | Draft PR writeup for agent-browser pre-flight guidance |
| docs/pilot-runs/upstream-pr-drafts/4-supabase-agent-skills-two-pass.md | Draft PR writeup for a new supabase SQL review reference file |
| docs/pilot-runs/upstream-pr-drafts/5-google-labs-code-stitch-skills-shadcn-ui.md | Draft PR writeup for shadcn-ui checklist + placement rule guidance |
| docs/pilot-runs/upstream-pr-conventions.md | Internal conventions reference for submitting PRs to the target upstream repos |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+8
to
+14
| ## Drafts (current canonical set) | ||
|
|
||
| | # | Skill | Target repo | Evidence strength | Draft | | ||
| |---|---|---|---|---| | ||
| | 1 | web-design-guidelines (rules doc) | `vercel-labs/web-interface-guidelines` | **Strong.** v1.2.1 measured 0.92→1.00 across 18 trials × 3 frontier models. 22-line additive change. | [draft](./1-vercel-labs-web-interface-guidelines.md) | | ||
| | 3 | agent-browser (Pre-flight) | `vercel-labs/agent-browser` | **Soft.** v1.0 baseline 0.97; observed 1/9 Gemini trial fell back to `curl`. Deeper-eval v1.2.1 pilot was attempted but timed out at the 90-min wrapper cap mid-baseline (50/54 trials done, no Phase 5 commit). 11-line additive Pre-flight section. | [draft](./3-vercel-labs-agent-browser-pre-flight.md) | | ||
| | 4 | supabase-postgres-best-practices | `supabase/agent-skills` | **Soft.** v2 baseline 0.97 overall; per-case shows update-without-where at 77.8% (the failure pattern the reference targets). Auto-pilot's exit-on-≥0.95-overall logic missed the per-case signal (v1.3 design addresses this). Single additive reference file under existing `monitor-` prefix. | [draft](./4-supabase-agent-skills-two-pass.md) | |
Comment on lines
+16
to
+18
| The wrapper-skill PR target (`vercel-labs/agent-skills/skills/web-design-guidelines/SKILL.md`) | ||
| was dropped — see `superseded/README.md`. The SKILL.md is a thin | ||
| discovery-stub adapter; all value lives in `command.md` (PR #1). |
Comment on lines
+27
to
+29
| `pnpm test:sanity` for supabase — but note: sanity test does NOT | ||
| validate per-reference frontmatter; convention is enforced by | ||
| maintainer review). |
Comment on lines
+91
to
+92
| naming convention with valid frontmatter (`title`, `impact`, `tags`). | ||
| Run `pnpm test:sanity` locally before submitting. |
Comment on lines
+58
to
+61
| The full proposed file is committed in our repo at: | ||
|
|
||
| - [`examples/workbench/shadcn-ui/proposed-upstream-changes/google-labs-code-stitch-skills/after-SKILL.md`](../../../examples/workbench/shadcn-ui/proposed-upstream-changes/google-labs-code-stitch-skills/after-SKILL.md) | ||
|
|
Comment on lines
+49
to
+53
| The full proposed `after-SKILL.md` is checked into our repo at: | ||
|
|
||
| - [`examples/workbench/agent-browser/proposed-upstream-changes/vercel-labs-agent-browser/after-SKILL.md`](../../../examples/workbench/agent-browser/proposed-upstream-changes/vercel-labs-agent-browser/after-SKILL.md) | ||
| (note: the auto-pilot's proposal points at `skills/agent-browser/SKILL.md`; for the | ||
| upstream PR we re-target to `skill-data/core/SKILL.md` per AGENTS.md) |
Comment on lines
+68
to
+75
| 3. **No dashboard/MDX page update needed?** Per AGENTS.md, "Any skill | ||
| improvement PR must touch `skill-data/core/SKILL.md` and its | ||
| `references/` files, plus README and docs MDX pages." This change | ||
| is so minor (a single ## section) that it likely doesn't need the | ||
| README or MDX updates — but worth checking with the maintainer | ||
| (`ctate`) in the PR description if you want zero-friction merge. | ||
| Alternative: also add a one-line bullet to README's "Tips" or | ||
| equivalent that says "verify install with `which agent-browser`". |
Comment on lines
+16
to
+23
| "two-pass review" concept. The auto-pilot read the upstream context | ||
| file (`tools/auto-improve-contexts/supabase-postgres-best-practices.md`, | ||
| encoded from gh-CLI research of CONTRIBUTING.md, `_template.md`, | ||
| `_contributing.md`, `_sections.md`, plus the last 10 merged PRs) and | ||
| produced a file that conforms exactly to the existing 28-reference | ||
| convention: 4-field frontmatter, `monitor-` prefix, single rule, | ||
| `**Incorrect**`/`**Correct**` SQL blocks, trailing `Reference:` link, | ||
| ~50 lines. |
|
|
||
| - v1.2.1 auto-pilot run: branch `eval/auto-pilot/web-design-guidelines`, | ||
| commit `df7149e`, status `success`, baseline 0.92, final 1.00 | ||
| - Context file: `tools/auto-improve-contexts/vercel-web-interface-guidelines.md` |
Comment on lines
+203
to
+206
| - Context file (research subagent output): | ||
| `skills/auto-improve-orchestrator/references/contexts/google-labs-code-shadcn-ui.md` | ||
| - Eval workbench: `examples/workbench/shadcn-ui/` (2 cases: | ||
| `review-usercard`, `review-statusbadge`) |
Adds #6 firebase-hosting-basics (the first full end-to-end v1.3 orchestrator demo with Phase 3.5 eval-iteration). Measured uplift 0.89 → 1.00 (+0.11) on frontier matrix; orchestrator added 2 harder cases via add-harder direction before applying Recipe C. Removes #3 agent-browser and #4 supabase from the canonical set — both ended with null/soft evidence (#3 timed out after frontier-matrix re-fire showed uplift-too-small; #4's per-case finding was identified but no measured uplift). Keeping them for internal reference only, not for team submission. They remain in PR #49 (the older bloated PR) for traceability. Updates README to reflect 3-strong canonical set with Google CLA note for #5 and #6 (both Google-org repos: stitch-skills and firebase). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fact-check of firebase/agent-skills upstream confirmed the orchestrator's research subagent context was accurate (Apache 2.0, Google CLA, main vs next branch convention, frontmatter spec, CI). One addition from CONTRIBUTING.md not in the original context: an eval pipeline at firebase/firebase-tools/scripts/agent-evals where contributors are recommended to add test cases for their skills. Optional follow-up PR — adding as a caveat for the operator to mention in the PR description. Also explicit branch-target confirmation: incremental → main per CONTRIBUTING.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Zhaiyuqing2003
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three polished PR drafts for proposed upstream contributions to public agent skill repos. Each draft is a self-contained submission packet — describes the exact diff to apply, the PR title + body to use, caveats, and the shell commands the operator runs to actually create the upstream PR.
All three have measured uplift on the frontier model matrix (claude-sonnet-4.6, openai/gpt-5, google/gemini-2.5-pro × 3 trials per case).
Drafts
vercel-labs/web-interface-guidelinesgoogle-labs-code/stitch-skillsfirebase/agent-skillsPlus
upstream-pr-conventions.md— per-repo title format, body convention, CI gates, license, CLA, branch-target (main vs next), and merge-style gotchas.What's NOT here (deliberately)
Two earlier drafts (#3 agent-browser, #4 supabase) reached null / soft findings after the v1.3 orchestrator dispatch:
uplift-too-small(0.667→0.667); the per-case-min floor was gpt-5 missing the core-skill-load step on Tier-0 cases — model-capability issue, not skill-content addressable.Both kept internal-only at
docs/pilot-runs/upstream-pr-drafts/(older PR #49). NOT submitted for team review here to avoid wasted attention on null findings.Reviewer checklist
Greenlight on which of the 3 to submit upstream:
main(incremental change), NOTnext(which is for new skills).Background on the methodology
These drafts are outputs of the auto-improve-skill pipeline (v1.2.1 + v1.3). See PR #50 for the v1.3 orchestrator architecture and
docs/auto-improve-skill-v1.3-spec.mdin that PR for the design rationale.Briefly: each orchestrator subagent is dispatched against one upstream skill, runs a 5-phase pipeline (research → baseline → eval-iterate → skill-iterate → package), and produces a
proposed-upstream-changes/directory with before/after files + an honest evidence-tagged status (success|uplift-too-small|skill-genuinely-good|blocked-by-skill-shape).Test plan
## Evidencetable + diff## Operator steps to submitsection in each chosen draft to create the actual upstream PR🤖 Generated with Claude Code