Test expirimental branch: analyst

[dustinvannoy-db]

Evalusate skills in experimental branch (no MCP server) compared to main

• databricks-aibi-dashboards
• databricks-dbsql
• databricks-genie
• databricks-metric-views

[jacksandom]

databricks-genie — A/B comparison: main vs. experimental

Compared main (MCP-based) vs. origin/experimental (CLI-based) using SkillForge's A/B pipeline. Ground truth: 7 shared test cases at .test/skills/databricks-genie/ (create space, list spaces, ask genie, inspect schema, get space details, delete space, follow-up conversation).

Per-side MCP isolation enforced via per-repo .mcp.json:

• A side: ai-dev-kit/.mcp.json → only databricks MCP server, no plugin tools. Confirmed at runtime: MCP preflight: ... servers=['databricks'].
• B side: ai-dev-kit-experimental/.mcp.json = {"mcpServers": {}} → no MCP tools, CLI-only. Confirmed at runtime: MCP preflight: ... servers=[].

Attached: report.html for each side (per-side L1/L3/L5 breakdown with pass/fail checks).

Genie Report A.html
Genie Report B.html

Headline

Experimental edges out main on agent task execution. L3 doc audit is statistical parity (0.77 vs 0.76); L5 output goes to experimental by 8 points (0.65 vs 0.73). Recent skill polish on the experimental branch (commit 385a6de — Polish skill clarity: parallel SQL probes, Genie ID rules) closed several specific operational issues that previously cost experimental tool calls.

Per-level scores

Level	Main (MCP)	Experimental (CLI)
L1 (unit — syntax / link validity)	0.97	0.97
L3 (static doc audit)	0.77	0.76
L5 (output — agent task completion)	0.65	0.73

What changed since the previous A/B

The experimental branch's recent commit 385a6de directly addressed three operational issues we surfaced from earlier runs:

• serialized_space ID uniqueness rule documented: previously the API rejected payloads with "Duplicate instruction ID '...': first seen in ..., duplicated in ..." because IDs collided across text_instructions / example_question_sqls / sample_questions. Now documented in Field Format Requirements.
• Sort-order rules documented: data_sources.tables must be sorted by identifier; example_question_sqls and text_instructions must be sorted by id. Previously surfaced as Invalid export proto: ... must be sorted by id errors mid-task.
• Parallel SQL probes via Statement Execution API: experimental's Step 1 now shows a fan-out pattern that runs schema / cardinality / distribution probes concurrently instead of serially. The skill estimates 5 probes in 17s wall time, vs. several minutes before.

Each of these had cost B's agent multiple recovery tool calls per task. With them documented, the agent steers around them and L5 climbs from ~0.41 in the prior run to 0.73 here.

Judge verdict

Judge step crashed with a Pydantic validation error — Sonnet 4.6 returned per-dimension fields (a_score_note, b_score_note) that the ComparisonVerdict schema forbids:

Comparison failed: 8 validation errors for ComparisonVerdict
dimensions.0.a_score_note: Extra inputs are not permitted [type=extra_forbidden]
dimensions.0.b_score_note: Extra inputs are not permitted [type=extra_forbidden]
... etc

This is a known SkillForge bug — the schema is too rigid against LLM output drift. Same shape of failure as the previous run (different field names, same root cause). Per-dimension judge scores not available for this run as a result.

L3 audit — dimension breakdown

Dimension	Main	Experimental	Winner
self_contained	7	8	experimental +1
no_conflicts	8	9	experimental +1
security	9	8	main +1
llm_navigable_structure	9	9	tie
actionable_instructions	9	9	tie
scoped_clearly	9	8	main +1
error_handling	8	8	tie
no_hallucination_triggers	8	7	main +1

Experimental wins:

• self_contained 8/10: documents the full lifecycle inline — schema discovery, space creation, export/import, improvement, conversation API, troubleshooting — with no unexplained external references.
• no_conflicts 9/10: serialization convention is consistently applied end-to-end (file on disk is always parsed JSON; every use site uses the same jq -c '.' pattern).

Main wins:

• security 9/10: minor edge for more explicit permission-guidance framing.
• scoped_clearly 9/10: edge for more explicit negative-scoping language.
• no_hallucination_triggers 8/10: experimental references databricks experimental aitools tools discover-schema and databricks experimental aitools tools query — flagged as experimental CLI commands that may not exist in all Databricks CLI versions. Fix for experimental: pin the required CLI version in prerequisites, or mark these as known-good maintained dependencies.

L5 — behavioural observations

Both sides exercised the same 7-prompt suite. Per-side MCP isolation was clean: A consistently used mcp__databricks__manage_genie, mcp__databricks__execute_sql, mcp__databricks__get_table_stats_and_schema etc.; B used Bash + databricks CLI throughout.

Both sides hit the same workspace data issue on the create-space task (my_catalog doesn't exist). Recovery patterns differed — main pivoted via MCP catalog discovery, experimental attempted catalog creation via CLI and migrated to a real catalog (jss_sandbox_catalog). Experimental's path was longer (more tool calls per task on average) but landed on a real persisted artifact.

How much should you care about the L3 split?

The L3 differences are ±1 point per dimension on either side — essentially parity. The skill text itself is at comparable quality on both branches. The L5 edge to experimental is the operationally relevant signal, and the recent commit explains why.

Other data

• Run duration: main ~14m, experimental ~20m. Experimental is ~40% slower because CLI requires more tool-call round-trips, partially offset by the new parallel-probe pattern.

Caveats

• expected_facts, expected_patterns, and guidelines are empty for each of the 7 test cases in this ground truth, so per-case static scoring doesn't discriminate between sides. The L3 doc-audit and L5 numeric scores are the real signals.
• L5 numeric scores are noisy run-to-run because agent behaviour is stochastic. The 8-point gap here is consistent with the qualitative signal (experimental's polish reduced recovery tool calls), but treat it as a lower bound on the true effect rather than a precise measurement.
• Judge verdict unavailable due to SkillForge schema bug — reading the per-side scores + the commit context is the best read on this run.

[jacksandom]

databricks-metric-views — A/B comparison: main vs. experimental

Compared main (MCP-based) vs. origin/experimental (CLI-based) using SkillForge's A/B pipeline. Ground truth: 24 shared test cases at .test/skills/databricks-metric-views/eval/ covering create/query/alter/grant/drop plus conceptual questions and migration scenarios.

Note on ground truth: dropped the 2 original test cases that explicitly prompt for MCP tools (metric-views_create_mcp_003, metric-views_query_mcp_004) since those are guaranteed-fail on experimental and would add noise. Also stripped trace_expectations blocks (required/banned tool lists) that would have systematically penalised the CLI path. expected_facts, expected_patterns, and guidelines kept intact — those are SQL-syntax based and tool-agnostic.

Attached: report.html per side.

Metric Views Report A.html
Metric Views Report B.html

Headline

This is a near-tie. Judge explicitly returned confidence 0.35 and called the outcomes "largely parallel." Both skills scored identically on L1 (0.95) and L3 (0.86); experimental is 4 points behind on L5 composite (0.55 vs 0.59), which the judge attributed to small differences in tool idiom, not correctness.

Per-level scores

Level	Main (MCP)	Experimental (CLI)
L1 (unit)	0.95	0.95
L3 (static doc audit)	0.86	0.86
L5 (output — agent task completion)	0.59	0.55

L3 — essentially identical, both 0.86

Unlike genie, where experimental was 7 points behind main on L3, metric-views has basically parity. Per-dimension scores:

Dimension	Main	Experimental
self_contained	9	9
no_conflicts	7	7
security	9	9
llm_navigable_structure	9	9
actionable_instructions	9	9
scoped_clearly	9	9
error_handling	7	7
no_hallucination_triggers	8	8

Shared issue (same score both sides): the no_conflicts 7/10 score flags an identical YAML version inconsistency in both skills.

"Notable inconsistency around YAML version requirements. The main SKILL.md states version: 1.1 for DBR 17.2+ throughout, but [sections about window measures / later content] reference version differently."

Fix (applies to both main and experimental): standardise on one YAML version and update every reference in SKILL.md and yaml-reference.md.

Shared issue — error_handling 7/10: "Common Issues table covers 6 scenarios." Expanding to match the depth of, say, genie's main troubleshooting section (7+ scenarios) would push both to 8/10.

No experimental-specific doc issues to call out — the rewrite landed cleanly.

L5 — behavioural reality, per task

Both sides executed real work on the execution-heavy tasks (create/alter/query). Main made 156 tool calls across 24 tasks; experimental made 231. Experimental takes more calls because CLI requires more round trips per operation.

Breakdown of where experimental fell short:

Category	Count	Notes
Tasks both executed successfully	~15	Create, query, alter, grant, drop, filtered/windowed/joined metric views, messy-phrasing test, conversational phrasing
Tasks both correctly answered without tool calls	7	Conceptual Qs: "Why does SELECT * not work?", "Should I use a metric view or regular view?", "How do I enable materialization?", etc.
Tasks main executed but experimental skipped	2	Task 17: "conversational support tickets" creation; Task 20: "convert existing view to metric view" (main inspected the source table; experimental produced documentation only)

The 2 cases where experimental skipped execution likely explain the 4-point L5 gap.

Judge verdict

Winner A with confidence 0.35 — the judge explicitly signalled low confidence, calling the outcomes "largely parallel":

"Given the very small composite score delta and largely parallel outcomes, confidence in A's edge is low. The primary differentiator is A's use of the purpose-built MCP tooling (richer schema stats, managed metric view action) versus B's CLI workaround."

Per-dimension scores:

Dimension	Main	Experimental	Δ
correctness	8	7	+1 main
user_experience	7	7	tie
safety	9	9	tie
code_quality	8	7	+1 main

No regression flags.

Specific actionable findings for experimental

1. discover-schema returns less rich stats than get_table_stats_and_schema. The MCP tool returns min/max/avg/unique_count/value_counts; the CLI equivalent returns sample rows and null counts only. If column distributions matter for dimension/measure selection, the CLI path gives the agent less context to work with. Possible fix: document a fallback pattern using execute_sql to compute approx distributions when needed, e.g. SELECT approx_count_distinct(col), min(col), max(col) FROM ....

2. DDL heredoc ergonomics. Experimental's approach requires the agent to emit databricks experimental aitools tools query with heredoc-escaped dollar signs (AS \$\$ ... \$\$). This works but is less idiomatic than MCP's structured manage_metric_views(action="create", ...). In the tasks where experimental skipped execution (17, 20), the agent may have balked at the heredoc construction. Possible fix: document a template snippet with correctly-escaped $$ syntax the agent can follow, or show how to pass SQL via a -f file.sql pattern.

3. Two tasks fell through: "convert existing view to metric view" and "conversational support tickets" — experimental produced documentation-style responses; main executed the transformation. These are both substantive asks, not conceptual questions, so skipping execution is a real gap. Worth tracing which skill section the agent was reading and whether it suggests a "document this workflow" interpretation.

How much should you care?

Unlike genie where main had a clear L3 doc-quality advantage, metric-views is a genuine near-tie. Experimental's doc caught up with main's, the CLI path works for 15+ substantive tasks, and the judge confidence (0.35) confirms the two approaches are largely equivalent in practice. The 4-point L5 gap comes down to 2 skipped tasks and minor code-quality idiom differences — fixable, not fundamental.

Other data

• Run duration: main 6,102s (~1h42m), experimental 6,118s (~1h42m). Essentially identical — the ~30% CLI overhead we saw in genie doesn't appear here because most metric-view tasks are SQL-heavy rather than management-call-heavy, so both tool surfaces require similar per-task work.
• 24 tasks × 2 sides × 2 variants = 96 agent sessions. This is ~3× the size of genie's eval, so L5 numbers here are more stable.

Caveats

• Per-side reports overwrite each other — the attached report.html is whichever side wrote last. For the full per-side breakdown, see MLflow traces or the verdict JSON.
• 9 of 24 tasks have zero tool calls on both sides (conceptual Qs) — for those, L5 signal is whether the textual answer is correct, which the judge handles qualitatively.

[jacksandom]

databricks-aibi-dashboards — A/B comparison: main vs. experimental

Compared main (MCP-based) vs. origin/experimental (CLI-based) using SkillForge's A/B pipeline. Ground truth: 4 shared test cases at .test/skills/databricks-aibi-dashboards/ (basic dashboard, global filter, page-level filter, line-chart time series).

Attached: report.html for each side (per-side L1/L3/L5 breakdown with pass/fail checks).

Dashboards Report A.html
Dashboards Report B.html

Headline

Main wins with judge confidence 0.70. L3 (doc-quality audit) is essentially parity (0.83 vs 0.82), but L5 shows a real behavioural gap on the dashboard-creation step: main consistently reaches a create_or_update_dashboard call with a complete serialized payload and publishes; experimental explores the data extensively but the judge observed a visible transcript where the dashboard creation step was missing.

Per-level scores

Level	Main (MCP)	Experimental (CLI)
L1 (unit — syntax / link validity)	0.84	0.84
L3 (static doc audit)	0.83	0.82
L5 (output — agent task completion)	0.75	0.67

L5 — what each agent actually did, per task

Task	Main (MCP calls)	Experimental (Bash calls)
Basic NYC taxi dashboard	11	23
Sales dashboard with global region filter	16	21
Two-page dashboard with page-level filter	14	54
Daily trip-trend line chart	7	18
Total	48	116

Experimental made 2.4× more tool calls across the 4 tasks. Task 3 (multi-page filter) was especially expensive on experimental (48 Bash + 4 TodoWrite + 2 Write), vs 14 MCP calls on main. The CLI path requires many more round-trips to discover and assemble the same outcome.

Judge verdict

SkillForge's comparison judge returned winner A, confidence 0.70:

Dimension	Main	Experimental	Winner
correctness	8	5	main +3
user_experience	8	5	main +3
safety	8	8	TIE
code_quality	8	7	main +1

Rationale excerpt:

"Run A used the Databricks MCP tools and successfully reached a final manage_dashboard(create_or_update, publish=true) call, producing a published dashboard with KPI counters, daily/hourly/weekday bar charts, distance buckets, and top pickup zips. The response_text confirms the dashboard was 'created and published'. Run B took a CLI-based approach and explored the schema and metrics more broadly, but the truncated response_text ends mid-sentence ('High cardinality zips (128/205) — I') with no evidence in the shown tool calls that the dashboard was actually created/published — the 23 tool calls shown are all Bash-based data exploration queries, not a dashboard creation step."

Regression flags raised by the judge: dashboard-not-created-in-B, truncated-response-in-B.

L3 audit — dimension breakdown

Dimension	Main	Experimental	Winner
self_contained	9	9	tie
no_conflicts	8	9	experimental +1
security	9	9	tie
llm_navigable_structure	10	10	tie
actionable_instructions	10	10	tie
scoped_clearly	9	8	main +1
error_handling	9	9	tie
no_hallucination_triggers	9	8	main +1

L3 is essentially a wash — 5 ties, experimental wins 1 dimension, main wins 2. The two skills are at comparable documentation quality.

Experimental wins:

• no_conflicts 9/10: "Instructions are remarkably consistent throughout. The version requirements (counter=2, table=2, charts=3, filters=2, combo/choropleth=1) are stated identically in the main file and widget specifications."

Main wins:

• scoped_clearly 9/10: edge for clearer boundaries with related skills (Unity Catalog, metric views).
• no_hallucination_triggers 9/10: experimental references databricks experimental aitools tools commands which are flagged as experimental and may not be available in all environments. Fix for experimental: pin the required CLI version in prerequisites, or mark these as known-good maintained dependencies.

L5 — the operational gap

The main signal is behavioural, not structural. On the basic NYC taxi dashboard task the judge saw:

• Main: 11 MCP calls ending with manage_dashboard(action='create_or_update', publish=true, display_name='NYC Taxi Trip Statistics', serialized_dashboard={...}). Dashboard published. Agent narration: "Your NYC Taxi Trip Statistics dashboard has been created and published!"
• Experimental: 23 Bash calls of data exploration (warehouse list, schema discovery, SQL testing), no visible create-dashboard CLI call. Response cuts off mid-sentence in the middle of analysis.

The judge was explicit that B's truncation means it's possible subsequent tool calls (beyond the 23 shown) completed the dashboard — confidence is 0.70 rather than higher for that reason. But on the visible evidence, the experimental CLI path takes substantially longer to get to the dashboard-creation step and sometimes doesn't reach it within the session budget.

How much should you care about the L3 parity?

L3 parity is reassuring: the experimental rewrite hasn't degraded documentation quality. Both skills are well-structured and auditor-clean. The L5 gap is the real question, and it points at a workflow-completion issue on experimental, not a documentation issue.

Other data

• Run duration: main 2,365s (~39 min), experimental 2,557s (~43 min). Similar, despite experimental's 2.4× tool-call count — individual Bash calls are fast.
• Composite scores (informational): main 0.808, experimental 0.777.

Caveats

• 4 test cases is a modest sample. L5 numeric scores would be noisier at this sample size, but the qualitative judge finding ("dashboard creation step missing in B") is stable and specific.
• The judge sees a 12k-char-per-section truncation of each transcript. B's alleged failure to create the dashboard may be a truncation artefact, not an actual runtime failure — but even if that's the case, the signal is that experimental uses enough tool calls that the dashboard-creation step falls outside the visible window, which is operationally the same problem.
• expected_facts, expected_patterns, and guidelines for each case were rewritten to be tool-agnostic (e.g., "should deploy the dashboard to the workspace" rather than "should call mcp__databricks__create_or_update_dashboard") so static scoring doesn't penalise either tool surface. L5 numbers above reflect that rewrite.

[jacksandom]

databricks-dbsql — no meaningful A/B to run

The entire diff between main and origin/experimental for this skill is a single line in the Best Practices section:

- **Use MCP tools** (`execute_sql`, `execute_sql_multi`) to test and validate all SQL before deploying
+ **Test SQL via CLI** (`databricks experimental aitools tools query`) or notebooks before deploying

No script changes, no doc restructuring, no new sections, no new content anywhere else. A full SkillForge A/B on this skill would spawn ~28 agent sessions and run for 30-60 minutes to produce a verdict that's essentially predetermined: the two skills are functionally identical on every task except that one line of guidance. Skipping.