Copilot Agent Prompt Clustering Analysis - 2025-12-22

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts.

Summary

Analysis Period: Last 30 days

Total Tasks Analyzed: 2,328

Clusters Identified: 10

Overall Success Rate: 72.9% (1,697 merged)

Data Source: PRs #2097-#7207

Using K-means clustering with TF-IDF vectorization, we identified 10 distinct clusters of task patterns in copilot agent prompts. The analysis reveals clear patterns in task types, complexity, and success rates.

Key Insights

Top Performing Clusters

1. Cluster 10 (Version/CLI tasks): 79.0% success rate

   • Keywords: version, cli, changes, release, update
   • 124 tasks, avg 3.1 commits
   • Best practices: Clear, focused updates to dependencies and versioning

2. Cluster 9 (Update/Command tasks): 78.0% success rate

   • Keywords: update, command, docs, issue, file
   • 168 tasks, avg 3.6 commits
   • Includes documentation and configuration updates

3. Cluster 2 (Package/Workflow tasks): 77.8% success rate

   • Keywords: pkg, pkg workflow, workflow, validation, functions
   • 279 tasks, avg 3.5 commits
   • Well-scoped package-level refactoring

Most Common Task Types

Cluster 1 (Add/Fix): 28.9% of all tasks (673 tasks)

• Keywords: add, fix, tests, make, github
• 73.4% success rate
• Broad category covering bug fixes and new features
• Average 3.6 commits, 20.6 files changed

Complexity Analysis

Most Iterative (requiring most refinement):

• Cluster 3 (Safe Output): 4.8 commits average, 33.3 files
• Cluster 4 (MCP Server): 4.6 commits average, 26.7 files

These complex tasks involve architectural changes and may benefit from more detailed specifications.

Full Analysis Report

General Insights

• Highest Success Rate: Cluster 10 (79.0%) - Keywords: version, cli, changes
• Most Common Task Type: Cluster 1 (673 tasks, 28.9%) - Keywords: add, fix, tests
• Most Iterative Tasks: Cluster 3 (avg 4.8 commits)
• Average Files Changed: 19.5 files per task

Cluster Analysis

Cluster 1: Add, Fix

• Size: 673 tasks (28.9% of total)
• Success Rate: 73.4% (494 merged)
• Average Commits: 3.6
• Average Files Changed: 20.6
• Top Keywords: add, fix, tests, make, github
• Characteristics: Bug fixes and error handling; New feature implementation; Testing-related tasks
• Example PRs: Configure network firewall, edit, and bash tools for scheduled Copilot workflows #2472, Fix logs command counter to include cached runs #5500, Fix broken integration tests by stripping YAML comment header before assertions #4890

Sample Task: Add more docs on security...

Cluster 5: Workflows, Gh

• Size: 296 tasks (12.7% of total)
• Success Rate: 71.6% (212 merged)
• Average Commits: 3.1
• Average Files Changed: 13.9
• Top Keywords: workflows, gh, aw, md, gh aw
• Example PRs: Add .env file protection to .gitignore #3847, Add inline definitions for technical terms on first use #4500, chore: restructure docs following modern DX best practices #3526

Cluster 2: Pkg, Pkg Workflow

• Size: 279 tasks (12.0% of total)
• Success Rate: 77.8% (217 merged)
• Average Commits: 3.5
• Average Files Changed: 13.0
• Top Keywords: pkg, pkg workflow, workflow, validation, functions
• Example PRs: Migrate workflow commands (run, status, logs, audit) to RunE #5329, Consolidate scattered validation functions into dedicated validation files #3581, Upgrade google/jsonschema-go to v0.4.0 #6842

Cluster 7: Agentic, Agentic Workflow

• Size: 237 tasks (10.2% of total)
• Success Rate: 77.2% (183 merged)
• Average Commits: 3.6
• Average Files Changed: 12.7
• Top Keywords: agentic, agentic workflow, workflow, workflows, update
• Characteristics: Code updates and refactoring
• Example PRs: Expand zizmor workflow to run actionlint and poutine, rename to static-analysis-report #3043, Update DailyOps with example of adding comment to a discussion #4843, Refactor changeset workflow to scheduled batch processing with cache-based deduplication #3757

Cluster 9: Update, Command

• Size: 168 tasks (7.2% of total)
• Success Rate: 78.0% (131 merged)
• Average Commits: 3.6
• Average Files Changed: 18.1
• Top Keywords: update, command, docs, issue, file
• Characteristics: Bug fixes and error handling; Code updates and refactoring; Documentation tasks
• Example PRs: Use network config domains as fallback for sanitization when not manually configured #2397, Consolidate permission parsing logic and add support for "all: read" syntax #2119, Remove matrix strategy from CI build job #3146

Cluster 6: Agent, Github

• Size: 164 tasks (7.0% of total)
• Success Rate: 71.3% (117 merged)
• Average Commits: 3.8
• Average Files Changed: 23.6
• Top Keywords: agent, github, copilot, step, logs
• Example PRs: Add jsweep daily workflow for automated JavaScript cleanup #6854, Update dev.md to demonstrate custom_agent and custom_instructions with expression rendering #5611, Update terminology from "prompt command" to "agent" #5534

Cluster 8: Comments, Issue_Title

• Size: 137 tasks (5.9% of total)
• Success Rate: 62.0% (85 merged)
• Average Commits: 3.7
• Average Files Changed: 20.7
• Top Keywords: comments, issue_title, issue, code, section
• Characteristics: Bug fixes and error handling
• Example PRs: Refactor: Extract duplicate log parser formatting to shared module #4514, Update Claude Code CLI to v2.0.71 and disable Chrome integration #6699, Refactor update entity JS wrappers using factory pattern #7092

Cluster 3: Safe, Safe Output

• Size: 126 tasks (5.4% of total)
• Success Rate: 63.5% (80 merged)
• Average Commits: 4.8
• Average Files Changed: 33.3
• Top Keywords: safe, safe output, output, add, create
• Characteristics: New feature implementation
• Example PRs: fix: Use simple GraphQL mutation for backward compatibility when no Copilot options provided #5576, Improve MCP safe output tool descriptions for better AI tool selection #4729, Add "p" element to safe HTML allow list in text sanitization #4862

Cluster 4: Mcp, Server

• Size: 124 tasks (5.3% of total)
• Success Rate: 64.5% (80 merged)
• Average Commits: 4.6
• Average Files Changed: 26.7
• Top Keywords: mcp, server, mcp server, github, tool
• Example PRs: Refactor safe outputs MCP server into modular utilities to avoid step length limits #5477, Make MCP configuration default in init command, add --no-mcp flag #7001, Add 'add' tool to mcp-server command #3447

Cluster 10: Version, Cli

• Size: 124 tasks (5.3% of total)
• Success Rate: 79.0% (98 merged)
• Average Commits: 3.1
• Average Files Changed: 28.3
• Top Keywords: version, cli, changes, release, update
• Characteristics: Code updates and refactoring
• Example PRs: Fix Playwright version confusion between MCP package and browser image #4205, Update github.com/Masterminds/semver/v3 from v3.3.0 to v3.4.0 #4469, Update CLI versions: Claude Code 2.0.70, Codex 0.73.0 #6618

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Commits	Avg Files	Top Keywords
1	673	73.4%	3.6	20.6	add, fix, tests
5	296	71.6%	3.1	13.9	workflows, gh, aw
2	279	77.8%	3.5	13.0	pkg, pkg workflow, workflow
7	237	77.2%	3.6	12.7	agentic, agentic workflow, workflow
9	168	78.0%	3.6	18.1	update, command, docs
6	164	71.3%	3.8	23.6	agent, github, copilot
8	137	62.0%	3.7	20.7	comments, issue_title, issue
3	126	63.5%	4.8	33.3	safe, safe output, output
4	124	64.5%	4.6	26.7	mcp, server, mcp server
10	124	79.0%	3.1	28.3	version, cli, changes

Sample of Analyzed PRs

PR #	Title	Cluster	Outcome	Files	Commits	Keywords
#4976	Add security documentation guides	1	❌ Closed	5	3	add, fix
#4609	Fix PR branch checkout step failing with...	1	❌ Closed	90	4	add, fix
#3847	Add .env file protection to .gitignore	5	✅ Merged	1	2	workflows, gh
#4500	Add inline definitions for technical ter...	5	✅ Merged	4	2	workflows, gh
#5329	Migrate workflow commands (run, status, ...	2	✅ Merged	6	3	pkg, pkg workflow
#3581	Consolidate scattered validation functio...	2	✅ Merged	5	4	pkg, pkg workflow
#3043	Expand zizmor workflow to run actionlint...	7	✅ Merged	9	4	agentic, agentic workflow
#4843	Update DailyOps with example of adding c...	7	✅ Merged	1	2	agentic, agentic workflow
#2397	Use network config domains as fallback f...	9	✅ Merged	53	5	update, command
#2119	Consolidate permission parsing logic and...	9	✅ Merged	5	4	update, command
#6854	Add jsweep daily workflow for automated ...	6	✅ Merged	2	4	agent, github
#5466	Reject classic PATs in pre-agent job tok...	6	❌ Closed	62	5	agent, github
#4514	Refactor: Extract duplicate log parser f...	8	✅ Merged	84	3	comments, issue_title
#7031	Refactor: eliminate sanitization pipelin...	8	❌ Closed	3	3	comments, issue_title
#5477	Refactor safe outputs MCP server into mo...	4	✅ Merged	101	5	mcp, server
#7001	Make MCP configuration default in init c...	4	✅ Merged	4	4	mcp, server
#4205	Fix Playwright version confusion between...	10	✅ Merged	16	6	version, cli
#4469	Update github.com/Masterminds/semver/v3 ...	10	✅ Merged	4	3	version, cli

Key Findings

1. Clear Specification Wins: Clusters with highest success rates (Version/CLI, Update/Command) feature well-defined, focused tasks with clear objectives.

2. Complexity Matters: Tasks requiring 4.5+ commits (Safe Output, MCP Server) involve architectural changes and show lower success rates (63-65%). These benefit from:

   • More detailed upfront specifications
   • Breaking into smaller subtasks
   • Including architectural guidance

3. Documentation Excellence: Documentation tasks show 78% success rate, making them ideal candidates for agent automation.

4. Middle Ground Challenges: Cluster 8 (Comments/Issue handling) shows lowest success rate (62%), suggesting refactoring tasks with scattered changes are more challenging.

Recommendations

Based on this clustering analysis:

1. Optimize for High-Success Patterns:

   • Version/dependency updates: 79% success
   • Focused configuration changes: 78% success
   • Package-level refactoring: 77.8% success
   • Consider structuring prompts similar to these successful patterns

2. Break Down Complex Tasks:

   • Safe output and MCP server tasks average 4.6-4.8 commits
   • Lower success rates (63-65%)
   • Recommendation: Split into planning phase + implementation subtasks

3. Leverage Documentation Tasks:

   • 78% success rate, straightforward execution
   • Good candidates for automation
   • Consider batch processing doc updates

4. Improve Scattered Refactoring:

   • Cluster 8 (comment/issue handling): 62% success
   • Challenge: Changes span multiple files without clear structure
   • Recommendation: Provide architectural context or limit scope

Methodology

• Dataset: 2,328 copilot agent tasks (PRs Add minimal path format syntax reference to imports documentation #2097-[WIP] Fix agent-performance-analyzer to use local GitHub MCP #7207)
• Technique: K-means clustering with TF-IDF vectorization
• Features: N-grams (1-3), 150 max features, English stop words removed
• Optimal Clusters: 10 (determined via silhouette score: 0.062)
• Success Metric: PR merged vs closed/open

---

Analysis Artifacts:

• Full report: /tmp/gh-aw/agent/clustering-report.md
• Structured data: /tmp/gh-aw/agent/clustering-data.json
• Visualizations: /tmp/gh-aw/python/charts/clustering-analysis.png

Next Steps: Consider running weekly to track trends in task patterns and success rates over time.

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.