New Python model skills

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Change Log
 
+## 2.2.1
+
+### Agent skills
+
+- **New `dj-migrate-notebook-to-pymodel`, `dj-verify-pymodel-parity`, and `dj-document-pymodels` skills.** Migrate a legacy Jupyter notebook into a python model with migration plan, generate Trino SQL to verify a python model's output table matches a legacy table, and generate/refresh a topic-level README for a `python_models/<group>/<topic>/` folder — rounding out the python model lifecycle alongside `dj-create-python-model` and `dj-review-python-model`.
+
 ## 2.2.0
 
 ### Create Source UX Improvements

docs/AGENT_SKILLS.md

@@ -2,7 +2,7 @@
 
 DJ ships **AI agent skills** — packaged instructions that guide AI coding assistants (Claude Code, Cursor, GitHub Copilot, Cline, Windsurf, and others) through common DJ (Data JSON) Framework tasks: creating and refactoring models, registering sources, authoring Lightdash dashboards, running dbt and Trino commands, diagnosing slow Trino queries, resolving merge conflicts, committing your work, and more.
 
-This page explains what the skills are, how to turn them on, and catalogs the 17 skills DJ provides today.
+This page explains what the skills are, how to turn them on, and catalogs the 20 skills DJ provides today.
 
 ## What are DJ Agent Skills?
 
@@ -38,11 +38,11 @@ Point your AI coding tool at the workspace and the skills become available. Most
 - **Progressive disclosure.** A skill loads its `SKILL.md` first and pulls in `references/` or runs `scripts/` only when needed, keeping the assistant focused.
 - **Single source of truth.** Skills edit only the JSON sources of truth — `.model.json`, `.source.json`, `.python.json` — and never hand-edit the generated `.sql` / `.yml` / `.python.py`, which DJ regenerates via JSON Sync.
 - **You stay in control of DJ commands.** Skills can't run VS Code commands themselves; they'll ask you to run things like **`DJ: Sync to SQL and YML`** or **`DJ: Refresh Projects`** at the right moment.
-- **Some skills are read-only.** `dj-review-python-model`, `dj-govern-model`, and `dj-trino-analyzer` produce reports and change nothing.
+- **Some skills are read-only.** `dj-review-python-model`, `dj-govern-model`, `dj-trino-analyzer`, and `dj-verify-pymodel-parity` produce reports and change nothing.
 
 ## The skills
 
-DJ provides 17 skills, grouped below by what they help you do.
+DJ provides 20 skills, grouped below by what they help you do.
 
 ### Setup & configuration
 
@@ -95,7 +95,31 @@ Scaffolds a `.python.json` for a pre-dbt Python ETL pipeline that extracts data
 - **Example prompt:** _"Review this Python model for production readiness."_
 - **Bundled reference:** `review-checklist.md` — pass/fail examples and edge cases for every check.
 
-### Lightdash & AI hints
+#### `dj-migrate-notebook-to-pymodel`
+
+Migrates a legacy Jupyter notebook (`.ipynb`) into a python model. Classifies each cell into extract/transform/load/exploratory, flags hardcoded secrets and non-deterministic code, applies the SQL-first decision tree to every pandas transform, and produces a migration plan report for you to approve before it hands off to `dj-create-python-model` to scaffold the actual `.python.json`.
+
+- **Use when:** you want to migrate, port, or convert an existing notebook into a python model.
+- **Example prompt:** _"Migrate this notebook into a python model."_
+- **Bundled reference:** `notebook-pattern-mapping.md` — common notebook idioms mapped to their DJ/Trino equivalents.
+
+#### `dj-verify-pymodel-parity`
+
+**Read-only** (w.r.t. model files) generator of Trino SQL that proves a python model's output table matches a legacy/reference table — schema diff, per-partition row-count parity, tolerance-based aggregate parity, and row-level diffs via full outer join or checksum. Hands off actual execution to `dj-run-trino`.
+
+- **Use when:** you want to verify, check, or prove parity between an old table and a newly built or migrated python model's output table.
+- **Example prompt:** _"Verify this new table matches the old one for yesterday's partition."_
+- **Bundled reference:** `parity-recipes.md` — copy-paste SQL templates for each check type.
+
+#### `dj-document-pymodels`
+
+Generates or refreshes a topic-level `README.md` under `dags/python_models/<group>/<topic>/`, documenting every python model in that topic — a model table, per-model data flow and business-logic notes, upstream/downstream cross-references, and gotchas. Shows a diff before writing and preserves hand-written prose wrapped in `<!-- keep -->` markers across regenerations.
+
+- **Use when:** you want to document, write docs for, or generate/refresh a README for a python model topic or group.
+- **Example prompt:** _"Document the models in this topic."_
+- **Bundled reference:** `topic-readme-template.md` — the README skeleton and the preserve/regenerate convention.
+
+### Lightdash BI & AI hints
 
 #### `dj-create-lightdash-yaml`
 

package.json

@@ -10,7 +10,7 @@
     "type": "git",
     "url": "https://github.com/Workday/dj.git"
   },
-  "version": "2.2.0",
+  "version": "2.2.1",
   "workspaces": [
     "web"
   ],

templates/skills/dj-create-python-model/_SKILL.md

@@ -29,6 +29,9 @@ Use this skill when the user mentions: python model, ETL, data ingestion, API fe
 - `.source.json` files (registering a raw table as a source) → `dj-create-source`
 - Lightdash YAML → `dj-edit-lightdash-yaml`
 - Refactoring existing models → `dj-review-and-refactor-model`
+- Migrating a legacy Jupyter notebook into a new python model → `dj-migrate-notebook-to-pymodel` (it hands off to this skill's workflow once the migration plan is approved)
+- Verifying a python model's output table against a legacy/reference table → `dj-verify-pymodel-parity`
+- Documenting a topic's python models in a README → `dj-document-pymodels`
 
 ## Interactive gathering workflow
 

templates/skills/dj-document-pymodels/_SKILL.md

@@ -0,0 +1,59 @@
+---
+name: dj-document-pymodels
+description: >-
+  Generate or refresh a topic-level README.md documenting every python model
+  under a dags/python_models/<group>/<topic>/ folder — model table, data flow,
+  business-logic notes, upstream/downstream references, and gotchas. Preserves
+  hand-written prose across regenerations. Use when the user wants to
+  document, write docs for, or generate a README for a python model topic or
+  group. Not for creating or editing the python models themselves (->
+  dj-create-python-model), dbt SQL model docs under docs/models/ (that's a
+  separate existing convention), or auditing a model's production readiness
+  (-> dj-review-python-model).
+compatibility: DJ (Data JSON) Framework extension workspace with dags/python_models/
+metadata:
+  dj-skill: '1.0'
+---
+
+# Document Python Model Topic
+
+**Goal:** generate or refresh a human-readable `README.md` for a `dags/python_models/<group>/<topic>/` folder, documenting every python model in that topic — what it does, how data flows through it, and what a future maintainer needs to know. Invoked on demand; not an automated sync feature.
+
+**Scope:** this skill documents the `python_models/<group>/<topic>/` tree specifically. It does not generate or touch `docs/models/*.md` — that's the existing, separate convention for dbt SQL models.
+
+## When this skill applies
+
+Use this skill when the user mentions: document this topic, write a README for this python model group, generate docs for these python models, or update the topic documentation after adding/changing models.
+
+**Out of scope** — delegate to sibling skills:
+
+- Creating or editing the `.python.json` files themselves → **`dj-create-python-model`**
+- Migrating a notebook into a new python model (document it once it exists) → **`dj-migrate-notebook-to-pymodel`**
+- Auditing a model's production readiness → **`dj-review-python-model`**
+- Verifying a model's output data → **`dj-verify-pymodel-parity`**
+
+## Workflow
+
+- [ ] **1. Resolve scope.** Ask which topic folder to document if not already clear: `dags/python_models/<group>/<topic>/`. A "group" may span multiple topics — confirm whether the user wants one topic's README or every topic under a group (one README per topic either way).
+- [ ] **2. Scan all `.python.json` files** in the target topic folder. For each, read: `name`, `group`, `topic`, `description`, `dags`, `output` (database/schema/table, write_mode, partition_by), `depends_on`, and any markdown header cells in `cells` for narrative content the author already wrote.
+- [ ] **3. Cross-reference lineage.** For upstream references, check each model's `depends_on` and any `.source.json` files it reads from (via SQL `FROM`/`JOIN` in its cells). For downstream references, search other `.python.json` / `.model.json` files in the project for `depends_on` entries or `source` references pointing at this model's output table.
+- [ ] **4. Check for an existing README.** If `dags/python_models/<group>/<topic>/README.md` already exists, read it fully first.
+  - If it has no `<!-- keep -->` markers, treat the whole file as regeneratable but show the user a diff before overwriting — do not silently replace hand-written prose the author may not have marked.
+  - If it has `<!-- keep -->...<!-- /keep -->` blocks, preserve their contents verbatim and only regenerate everything outside them (primarily the model table, which is mechanically derived from the JSON files and should always reflect current state).
+- [ ] **5. Draft the README** using the template in [references/topic-readme-template.md](references/topic-readme-template.md): topic purpose paragraph, model table, per-model sections (data flow, business-logic notes, upstream/downstream, gotchas).
+  - **Topic purpose:** ask the user for a one-paragraph summary if it can't be reasonably inferred from the models' `description` fields — do not invent a purpose from thin air.
+  - **Gotchas:** only include what's actually evidenced in the code/JSON (e.g., a rate-limit comment, a manual credential-rotation note, a known data-quality caveat mentioned in a markdown cell) — do not fabricate operational knowledge that isn't there. If nothing is evidenced, omit the subsection for that model rather than inventing filler.
+- [ ] **6. Show the diff and confirm before writing.** Present the draft (or, for a refresh, a diff against the existing file) and get the user's go-ahead before writing to disk.
+- [ ] **7. Write the README** to `dags/python_models/<group>/<topic>/README.md` only after confirmation.
+
+## Hard rules (DO NOT)
+
+- **DO NOT** silently overwrite an existing README without showing a diff first.
+- **DO NOT** discard content inside `<!-- keep -->...<!-- /keep -->` markers when regenerating.
+- **DO NOT** invent business-logic notes, gotchas, or a topic purpose that isn't evidenced in the model JSON, code, or user input — omit the subsection instead of fabricating content.
+- **DO NOT** edit any `.python.json` file — this skill only reads them.
+- **DO NOT** write to `docs/models/*.md` — that's a separate, existing convention for dbt SQL models.
+
+## Reference
+
+For the full README skeleton (section-by-section) and the `<!-- keep -->` preserve/regenerate convention, see [references/topic-readme-template.md](references/topic-readme-template.md).

templates/skills/dj-document-pymodels/references/topic-readme-template.md

@@ -0,0 +1,65 @@
+# Topic README template
+
+Skeleton for `dags/python_models/<group>/<topic>/README.md`, generated/refreshed by `dj-document-pymodels`.
+
+## Preserve vs. regenerate convention
+
+Wrap any hand-written prose you want to survive future regenerations in a `<!-- keep -->` / `<!-- /keep -->` block:
+
+```markdown
+<!-- keep -->
+This section was written by a human and will not be touched by future
+regenerations of this README.
+<!-- /keep -->
+```
+
+Everything **outside** `<!-- keep -->` blocks is mechanically regenerated from the current `.python.json` files each time the skill runs — most importantly the model table (§2 below), which should always reflect current state rather than go stale. A README with no `<!-- keep -->` markers at all is treated as fully regeneratable, but the skill still shows a diff before overwriting it.
+
+## Skeleton
+
+```markdown
+# <topic> (<group>)
+
+<!-- keep -->
+<One-paragraph topic purpose — what this group of models exists to do,
+who/what consumes the output, and why it's organized as its own topic.>
+<!-- /keep -->
+
+## Models
+
+| Model | Description | DAG(s) | Output table | Write mode | Partition | Depends on |
+|-------|-------------|--------|---------------|------------|-----------|------------|
+| `python__<group>__<topic>__<name>` | <one-line description> | <dag list or "none"> | `<database>.<schema>.<table>` | <write_mode> | <partition_by columns> | <upstream models or "none"> |
+
+## `<model_name>`
+
+### Data flow
+
+<Source → extract → transform → output, one or two sentences. E.g., "Fetches
+paginated results from the Backstage API, stages raw JSON into a Trino temp
+table, then transforms and writes to `glue_development.opus_python_source.catalogs`.">
+
+### Business logic notes
+
+<Non-obvious joins, dedup rules, special-case handling worth calling out —
+only include what's evidenced in the code or JSON, omit if none.>
+
+### Upstream / downstream
+
+- **Reads from:** <source tables / other python models this depends on>
+- **Read by:** <other python models or dbt sources that consume this model's output table>
+
+### Gotchas
+
+<Only include what's actually evidenced — a rate limit noted in a markdown
+cell, a manual credential-rotation step, a known data-quality caveat. Omit
+this subsection entirely for a model if nothing is evidenced.>
+
+<!-- repeat the "## `<model_name>`" section for every model in the topic -->
+```
+
+## Notes
+
+- The model table (§ "Models") is always mechanically derived from the `.python.json` files' `name`/`description`/`dags`/`output`/`depends_on` fields — never hand-edit it directly; edit the source JSON and re-run the skill instead.
+- Per-model sections that have no evidenced content for a subsection (e.g., no gotchas found) should omit that subsection rather than show an empty heading.
+- If the topic folder has only one model, still use the same per-model section structure — consistency matters more than brevity for a topic that later grows more models.