feat(databricks-skills): add databricks-mlflow-ml skill for classic ML

[dgokeeffe]

Why

The existing MLflow-related skills leave a gap for classic ML practitioners:

Skill	Scope	Covers classic ML UC registration?
databricks-mlflow-evaluation	GenAI agent evaluation (mlflow.genai.evaluate, scorers, judges)	❌ Different audience
databricks-model-serving	Real-time serving endpoints	❌ Covers serving, not training/registration
databricks-unity-catalog	Tables, volumes, system tables	❌ Data primitives, not model registry
databricks-mlflow-ml (this PR)	Classic ML training + UC registration + batch inference	✅

A data scientist training a forecasting model, registering it to Unity Catalog, and scoring predictions in a notebook or Lakeflow pipeline has no skill to trigger on. This PR fills that gap.

What's in the skill

SKILL.md — workflow index (Train → Register → Score, Retrain + Promote A/B, Debugging), quick-start, runtime compatibility note, and trigger description.

7 reference files:

• GOTCHAS.md — 14 common mistakes with symptoms + fixes
• CRITICAL-interfaces.md — exact API signatures + the models:/catalog.schema.model@alias URI format
• patterns-experiment-setup.md — UC volume artifact_location (required in UC-enforced workspaces)
• patterns-training.md — logging with signature + input_example, sklearn.Pipeline wrapping, autologging
• patterns-uc-registration.md — three-level names, @champion/@challenger aliases, verification via DESCRIBE MODEL, A/B promotion
• patterns-batch-inference.md — notebook pyfunc.load_model (Tier 1), Lakeflow SDP pyfunc.spark_udf (Tier 2), champion-vs-challenger validation, explicit warning against ai_query on custom UC models
• user-journeys.md — 7 end-to-end workflows including debugging scenarios

Key gotchas this skill teaches that other guides miss

1. UC volume artifact_location on experiment creation — DBFS root is rejected in UC-enforced workspaces. Every log_model call fails with opaque errors until artifact_location points at a UC volume.
2. mlflow.set_registry_uri('databricks-uc') — without this, register_model silently routes to the legacy workspace registry. The Add initial skills for Databricks development #1 "my model isn't showing up in Catalog Explorer" support question.
3. ai_query on custom UC models — doesn't work. Requires a serving endpoint. Correct primitive is mlflow.pyfunc.load_model (notebook) or mlflow.pyfunc.spark_udf (Lakeflow).
4. @champion / @challenger aliases — replace deprecated transition_model_version_stage() stages. The legacy API still exists but is a no-op on UC-registered models (no error, no effect).
5. mlflow.pyfunc.spark_udf in Lakeflow SDP — must be constructed at module scope, not inside @dp.materialized_view. Otherwise deserialization repeats on every pipeline evaluation.
6. pip install 'mlflow[databricks]' — required for UC registration outside Databricks clusters. Plain pip install mlflow omits the cloud-storage SDKs (azure-core / boto3 / google.cloud) MLflow needs to stage UC artifacts. Clusters ship the extras pre-installed.

Testing

Field-tested end-to-end against a live Databricks workspace:

• Feature table seeded, trained a GradientBoostingRegressor
• Registered to UC with @champion alias — verified in Catalog Explorer UI
• Loaded via mlflow.pyfunc.load_model — predictions within ~2% of actuals
• Two additional gotchas surfaced during the test (mlflow[databricks] install + artifact_path deprecation) and added to GOTCHAS.md

Runtime verified: MLflow 3.11 on Lakeflow SDP serverless compute v5 (current default). Patterns compatible with MLflow 2.16+ — pairs on older classic DBRs still get correct behaviour. 2.x/3.x divergences called out in GOTCHAS.md (e.g., artifact_path → name=).

Structure parity

File layout matches databricks-mlflow-evaluation (same SKILL.md + references/ + GOTCHAS.md + CRITICAL-interfaces.md + patterns-*.md convention). Installable via the existing install_skills.sh:

./install_skills.sh databricks-mlflow-ml

Not in scope

• Model Serving endpoints (databricks-model-serving covers that)
• GenAI agent evaluation (databricks-mlflow-evaluation covers that)
• Generic UC primitives like volumes and tables (databricks-unity-catalog covers those)

Deliberately narrow — classic ML + UC registration + batch inference only.

Origin

Built to fill a gap encountered during the Coles Vibe Workshop (airgapped Databricks field-engineer hackathon). DS pairs needed UC-scoped MLflow guidance that wasn't covered by any existing skill. Content battle-tested in the workshop before being contributed upstream.

[dustinvannoy-db]

Do the mlflow official skills we install not over this gap? cc: @jacksandom

[dgokeeffe]

@dustinvannoy-db I checked the mlflow/skills repo (what install_genie_code_skills.py pulls from). All 8 skills are GenAI/LLM-tracing scoped: agent-evaluation, mlflow-onboarding, instrumenting-with-mlflow-tracing, analyze-mlflow-trace, analyze-mlflow-chat-session, querying-mlflow-metrics, retrieving-mlflow-traces, searching-mlflow-docs. Not one touches Unity Catalog, set_registry_uri('databricks-uc'), @champion/@challenger aliases, or pyfunc.spark_udf.

The UC-specific stuff is what this PR covers: UC-enforced workspaces rejecting DBFS artifact roots, the legacy stage transition API silently no-oping on UC models, ai_query not working on custom UC models. That belongs here rather than upstream — it's Databricks config, not MLflow API.