diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f65c5b..66730e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Change Log +## 2.2.1 + +### Agent skills + +- **New `dj-migrate-notebook-to-pymodel`, `dj-verify-pymodel-parity`, and `dj-document-pymodels` skills.** Migrate a legacy Jupyter notebook into a python model with migration plan, generate Trino SQL to verify a python model's output table matches a legacy table, and generate/refresh a topic-level README for a `python_models///` folder — rounding out the python model lifecycle alongside `dj-create-python-model` and `dj-review-python-model`. + ## 2.2.0 ### Create Source UX Improvements diff --git a/docs/AGENT_SKILLS.md b/docs/AGENT_SKILLS.md index 2dd7c55..aa0bc3d 100644 --- a/docs/AGENT_SKILLS.md +++ b/docs/AGENT_SKILLS.md @@ -2,7 +2,7 @@ DJ ships **AI agent skills** — packaged instructions that guide AI coding assistants (Claude Code, Cursor, GitHub Copilot, Cline, Windsurf, and others) through common DJ (Data JSON) Framework tasks: creating and refactoring models, registering sources, authoring Lightdash dashboards, running dbt and Trino commands, diagnosing slow Trino queries, resolving merge conflicts, committing your work, and more. -This page explains what the skills are, how to turn them on, and catalogs the 17 skills DJ provides today. +This page explains what the skills are, how to turn them on, and catalogs the 20 skills DJ provides today. ## What are DJ Agent Skills? @@ -38,11 +38,11 @@ Point your AI coding tool at the workspace and the skills become available. Most - **Progressive disclosure.** A skill loads its `SKILL.md` first and pulls in `references/` or runs `scripts/` only when needed, keeping the assistant focused. - **Single source of truth.** Skills edit only the JSON sources of truth — `.model.json`, `.source.json`, `.python.json` — and never hand-edit the generated `.sql` / `.yml` / `.python.py`, which DJ regenerates via JSON Sync. - **You stay in control of DJ commands.** Skills can't run VS Code commands themselves; they'll ask you to run things like **`DJ: Sync to SQL and YML`** or **`DJ: Refresh Projects`** at the right moment. -- **Some skills are read-only.** `dj-review-python-model`, `dj-govern-model`, and `dj-trino-analyzer` produce reports and change nothing. +- **Some skills are read-only.** `dj-review-python-model`, `dj-govern-model`, `dj-trino-analyzer`, and `dj-verify-pymodel-parity` produce reports and change nothing. ## The skills -DJ provides 17 skills, grouped below by what they help you do. +DJ provides 20 skills, grouped below by what they help you do. ### Setup & configuration @@ -95,7 +95,31 @@ Scaffolds a `.python.json` for a pre-dbt Python ETL pipeline that extracts data - **Example prompt:** _"Review this Python model for production readiness."_ - **Bundled reference:** `review-checklist.md` — pass/fail examples and edge cases for every check. -### Lightdash & AI hints +#### `dj-migrate-notebook-to-pymodel` + +Migrates a legacy Jupyter notebook (`.ipynb`) into a python model. Classifies each cell into extract/transform/load/exploratory, flags hardcoded secrets and non-deterministic code, applies the SQL-first decision tree to every pandas transform, and produces a migration plan report for you to approve before it hands off to `dj-create-python-model` to scaffold the actual `.python.json`. + +- **Use when:** you want to migrate, port, or convert an existing notebook into a python model. +- **Example prompt:** _"Migrate this notebook into a python model."_ +- **Bundled reference:** `notebook-pattern-mapping.md` — common notebook idioms mapped to their DJ/Trino equivalents. + +#### `dj-verify-pymodel-parity` + +**Read-only** (w.r.t. model files) generator of Trino SQL that proves a python model's output table matches a legacy/reference table — schema diff, per-partition row-count parity, tolerance-based aggregate parity, and row-level diffs via full outer join or checksum. Hands off actual execution to `dj-run-trino`. + +- **Use when:** you want to verify, check, or prove parity between an old table and a newly built or migrated python model's output table. +- **Example prompt:** _"Verify this new table matches the old one for yesterday's partition."_ +- **Bundled reference:** `parity-recipes.md` — copy-paste SQL templates for each check type. + +#### `dj-document-pymodels` + +Generates or refreshes a topic-level `README.md` under `dags/python_models///`, documenting every python model in that topic — a model table, per-model data flow and business-logic notes, upstream/downstream cross-references, and gotchas. Shows a diff before writing and preserves hand-written prose wrapped in `` markers across regenerations. + +- **Use when:** you want to document, write docs for, or generate/refresh a README for a python model topic or group. +- **Example prompt:** _"Document the models in this topic."_ +- **Bundled reference:** `topic-readme-template.md` — the README skeleton and the preserve/regenerate convention. + +### Lightdash BI & AI hints #### `dj-create-lightdash-yaml` diff --git a/package-lock.json b/package-lock.json index 05dffda..384a1dd 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "dj", - "version": "2.2.0", + "version": "2.2.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "dj", - "version": "2.2.0", + "version": "2.2.1", "license": "Apache-2.0", "workspaces": [ "web" diff --git a/package.json b/package.json index aa81f9c..276a305 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,7 @@ "type": "git", "url": "https://github.com/Workday/dj.git" }, - "version": "2.2.0", + "version": "2.2.1", "workspaces": [ "web" ], diff --git a/templates/skills/dj-create-python-model/_SKILL.md b/templates/skills/dj-create-python-model/_SKILL.md index b8d86a9..0af4033 100644 --- a/templates/skills/dj-create-python-model/_SKILL.md +++ b/templates/skills/dj-create-python-model/_SKILL.md @@ -29,6 +29,9 @@ Use this skill when the user mentions: python model, ETL, data ingestion, API fe - `.source.json` files (registering a raw table as a source) → `dj-create-source` - Lightdash YAML → `dj-edit-lightdash-yaml` - Refactoring existing models → `dj-review-and-refactor-model` +- Migrating a legacy Jupyter notebook into a new python model → `dj-migrate-notebook-to-pymodel` (it hands off to this skill's workflow once the migration plan is approved) +- Verifying a python model's output table against a legacy/reference table → `dj-verify-pymodel-parity` +- Documenting a topic's python models in a README → `dj-document-pymodels` ## Interactive gathering workflow diff --git a/templates/skills/dj-document-pymodels/_SKILL.md b/templates/skills/dj-document-pymodels/_SKILL.md new file mode 100644 index 0000000..1bfd9b6 --- /dev/null +++ b/templates/skills/dj-document-pymodels/_SKILL.md @@ -0,0 +1,59 @@ +--- +name: dj-document-pymodels +description: >- + Generate or refresh a topic-level README.md documenting every python model + under a dags/python_models/// folder — model table, data flow, + business-logic notes, upstream/downstream references, and gotchas. Preserves + hand-written prose across regenerations. Use when the user wants to + document, write docs for, or generate a README for a python model topic or + group. Not for creating or editing the python models themselves (-> + dj-create-python-model), dbt SQL model docs under docs/models/ (that's a + separate existing convention), or auditing a model's production readiness + (-> dj-review-python-model). +compatibility: DJ (Data JSON) Framework extension workspace with dags/python_models/ +metadata: + dj-skill: '1.0' +--- + +# Document Python Model Topic + +**Goal:** generate or refresh a human-readable `README.md` for a `dags/python_models///` folder, documenting every python model in that topic — what it does, how data flows through it, and what a future maintainer needs to know. Invoked on demand; not an automated sync feature. + +**Scope:** this skill documents the `python_models///` tree specifically. It does not generate or touch `docs/models/*.md` — that's the existing, separate convention for dbt SQL models. + +## When this skill applies + +Use this skill when the user mentions: document this topic, write a README for this python model group, generate docs for these python models, or update the topic documentation after adding/changing models. + +**Out of scope** — delegate to sibling skills: + +- Creating or editing the `.python.json` files themselves → **`dj-create-python-model`** +- Migrating a notebook into a new python model (document it once it exists) → **`dj-migrate-notebook-to-pymodel`** +- Auditing a model's production readiness → **`dj-review-python-model`** +- Verifying a model's output data → **`dj-verify-pymodel-parity`** + +## Workflow + +- [ ] **1. Resolve scope.** Ask which topic folder to document if not already clear: `dags/python_models///`. A "group" may span multiple topics — confirm whether the user wants one topic's README or every topic under a group (one README per topic either way). +- [ ] **2. Scan all `.python.json` files** in the target topic folder. For each, read: `name`, `group`, `topic`, `description`, `dags`, `output` (database/schema/table, write_mode, partition_by), `depends_on`, and any markdown header cells in `cells` for narrative content the author already wrote. +- [ ] **3. Cross-reference lineage.** For upstream references, check each model's `depends_on` and any `.source.json` files it reads from (via SQL `FROM`/`JOIN` in its cells). For downstream references, search other `.python.json` / `.model.json` files in the project for `depends_on` entries or `source` references pointing at this model's output table. +- [ ] **4. Check for an existing README.** If `dags/python_models///README.md` already exists, read it fully first. + - If it has no `` markers, treat the whole file as regeneratable but show the user a diff before overwriting — do not silently replace hand-written prose the author may not have marked. + - If it has `...` blocks, preserve their contents verbatim and only regenerate everything outside them (primarily the model table, which is mechanically derived from the JSON files and should always reflect current state). +- [ ] **5. Draft the README** using the template in [references/topic-readme-template.md](references/topic-readme-template.md): topic purpose paragraph, model table, per-model sections (data flow, business-logic notes, upstream/downstream, gotchas). + - **Topic purpose:** ask the user for a one-paragraph summary if it can't be reasonably inferred from the models' `description` fields — do not invent a purpose from thin air. + - **Gotchas:** only include what's actually evidenced in the code/JSON (e.g., a rate-limit comment, a manual credential-rotation note, a known data-quality caveat mentioned in a markdown cell) — do not fabricate operational knowledge that isn't there. If nothing is evidenced, omit the subsection for that model rather than inventing filler. +- [ ] **6. Show the diff and confirm before writing.** Present the draft (or, for a refresh, a diff against the existing file) and get the user's go-ahead before writing to disk. +- [ ] **7. Write the README** to `dags/python_models///README.md` only after confirmation. + +## Hard rules (DO NOT) + +- **DO NOT** silently overwrite an existing README without showing a diff first. +- **DO NOT** discard content inside `...` markers when regenerating. +- **DO NOT** invent business-logic notes, gotchas, or a topic purpose that isn't evidenced in the model JSON, code, or user input — omit the subsection instead of fabricating content. +- **DO NOT** edit any `.python.json` file — this skill only reads them. +- **DO NOT** write to `docs/models/*.md` — that's a separate, existing convention for dbt SQL models. + +## Reference + +For the full README skeleton (section-by-section) and the `` preserve/regenerate convention, see [references/topic-readme-template.md](references/topic-readme-template.md). diff --git a/templates/skills/dj-document-pymodels/references/topic-readme-template.md b/templates/skills/dj-document-pymodels/references/topic-readme-template.md new file mode 100644 index 0000000..e8c1958 --- /dev/null +++ b/templates/skills/dj-document-pymodels/references/topic-readme-template.md @@ -0,0 +1,65 @@ +# Topic README template + +Skeleton for `dags/python_models///README.md`, generated/refreshed by `dj-document-pymodels`. + +## Preserve vs. regenerate convention + +Wrap any hand-written prose you want to survive future regenerations in a `` / `` block: + +```markdown + +This section was written by a human and will not be touched by future +regenerations of this README. + +``` + +Everything **outside** `` blocks is mechanically regenerated from the current `.python.json` files each time the skill runs — most importantly the model table (§2 below), which should always reflect current state rather than go stale. A README with no `` markers at all is treated as fully regeneratable, but the skill still shows a diff before overwriting it. + +## Skeleton + +```markdown +# () + + + + + +## Models + +| Model | Description | DAG(s) | Output table | Write mode | Partition | Depends on | +|-------|-------------|--------|---------------|------------|-----------|------------| +| `python______` | | | `..` | | | | + +## `` + +### Data flow + + + +### Business logic notes + + + +### Upstream / downstream + +- **Reads from:** +- **Read by:** + +### Gotchas + + + + +``` + +## Notes + +- The model table (§ "Models") is always mechanically derived from the `.python.json` files' `name`/`description`/`dags`/`output`/`depends_on` fields — never hand-edit it directly; edit the source JSON and re-run the skill instead. +- Per-model sections that have no evidenced content for a subsection (e.g., no gotchas found) should omit that subsection rather than show an empty heading. +- If the topic folder has only one model, still use the same per-model section structure — consistency matters more than brevity for a topic that later grows more models. diff --git a/templates/skills/dj-migrate-notebook-to-pymodel/_SKILL.md b/templates/skills/dj-migrate-notebook-to-pymodel/_SKILL.md new file mode 100644 index 0000000..d8124df --- /dev/null +++ b/templates/skills/dj-migrate-notebook-to-pymodel/_SKILL.md @@ -0,0 +1,132 @@ +--- +name: dj-migrate-notebook-to-pymodel +description: >- + Migrate a legacy Jupyter notebook (.ipynb) into a DJ python model. Reads the + notebook, classifies cells into extract/transform/load/exploratory, flags + hardcoded secrets and non-deterministic code, applies the SQL-first decision + tree, and produces a migration plan report for the user to approve before + any .python.json is scaffolded. Use when the user wants to migrate, port, or + convert an existing notebook/.ipynb into a python model. Not for scaffolding + a python model from scratch (-> dj-create-python-model), verifying old vs. + new table parity after migration (-> dj-verify-pymodel-parity), or auditing + an existing python model (-> dj-review-python-model). +compatibility: DJ (Data JSON) Framework extension workspace with .dj/schemas/ and dags/python_models/ +metadata: + dj-skill: '1.0' +--- + +# Migrate Notebook to DJ Python Model + +**Goal:** turn a legacy Jupyter notebook (`.ipynb`) that runs an ETL job into a DJ python model — a JSON-defined pipeline that extracts data from external sources and loads it into Iceberg tables for downstream dbt models to consume. This is a **two-phase** skill: Phase A analyzes the notebook and produces a migration plan for the user to review; Phase B (only after approval) scaffolds the actual `.python.json`. + +**SQL-first principle carries over from the target framework:** wherever a notebook cell's pandas transform can be expressed as Trino SQL, the migration plan proposes SQL instead of carrying pandas logic forward. Python is for orchestration and external ingestion; Trino is for transformation and storage. + +## When this skill applies + +Use this skill when the user mentions: migrate a notebook, convert a notebook, port a notebook, turn this `.ipynb` into a python model, or asks to bring a legacy/manual notebook pipeline into the framework. + +**Out of scope** — delegate to sibling skills: + +- Building the final `.python.json` once the plan is approved → **`dj-create-python-model`** (Phase B below hands off to it) +- Verifying the migrated model's output table matches the old notebook's output table → **`dj-verify-pymodel-parity`** +- Auditing a python model for production readiness after migration → **`dj-review-python-model`** + +## Phase A — Analyze (read-only, always runs first) + +- [ ] **1. Read the notebook.** Parse the `.ipynb` JSON. Walk `cells` in order, keeping `cell_type` (`code`/`markdown`), `source`, and any `outputs`. +- [ ] **2. Strip notebook-only output.** Never read or echo back the contents of a cell's `outputs` array in the migration report — it may contain stale data, PII, or credentials from a prior run. Analyze `source` only. +- [ ] **3. Classify every code cell** into one stage: + - **Extract** — reads from an external source (`requests.get`, a DB/Trino client, `pd.read_csv`, `boto3` S3 calls). + - **Transform** — pandas/DataFrame manipulation (filter, merge, groupby, reshape, type conversion). + - **Load** — writes the result somewhere (`df.to_sql`, `df.to_parquet`, `df.to_csv` to a shared location, a DB `INSERT`). + - **Exploratory/drop** — plotting (`matplotlib`, `seaborn`, `plotly`), `display()`/`print()`-only cells, interactive widgets, ad-hoc `df.head()`/`df.describe()` checks, commented-out scratch code. These are dropped from the migration — record each with a one-line reason. +- [ ] **4. Flag magic commands.** `%%time`, `!pip install ...`, `%matplotlib inline`, `%%bash`, etc. have no notebook-model equivalent — list them as dropped, and if a `!pip install` reveals a runtime dependency, carry that package name forward into the plan's dependency list instead of the magic itself. +- [ ] **5. Flag hardcoded secrets.** Scan every cell for API keys, tokens, passwords, or connection strings (literal strings assigned to variables named/matching `key`, `token`, `secret`, `password`, `conn(ection)?_string`, or high-entropy literals passed to auth headers/params). **Never carry a found secret into the plan or the eventual model** — call it out explicitly and state it must move to environment variables or the project's secret manager before the model can run. +- [ ] **6. Identify source type and destination.** From the extract/load cells, determine the source type (REST API, DB/Trino, CSV/file, S3) and the destination (what table or path is written, and in what mode — append vs. overwrite/replace). +- [ ] **7. Apply the SQL-first decision tree** (from `dj-create-python-model`) to every transform cell: + + | If the pandas op is... | Propose | + | --- | --- | + | Filtering / boolean masking | Trino `WHERE` | + | Column rename / select subset | Trino column aliasing / `SELECT` list | + | `.astype(...)` type conversion | Trino `CAST(... AS type)` | + | `.drop_duplicates()` | Trino `ROW_NUMBER()` dedup | + | `.groupby().agg(...)` | Trino `GROUP BY` | + | `pd.merge` / `.join()` | Trino `JOIN` | + | Nested JSON flattening (`pd.json_normalize`, dict/list unpacking) | Stays pandas (SQL can't easily express this) | + | API pagination / response parsing | Stays pandas | + | ML preprocessing (`sklearn`, embeddings, etc.) | Stays pandas | + + Record each transform cell's verdict (→ SQL or stays pandas) with the proposed SQL/pandas equivalent. + +- [ ] **8. Flag non-determinism / order dependence.** Look for: cells that reference variables only defined by running an earlier cell out of its written order, global mutable state mutated across cells, `datetime.now()`/`random`/`np.random` without a fixed seed or an explicit `context["ds"]`-derived date. Each is a migration risk — the DJ model must be safely re-runnable for a given `ds`. +- [ ] **9. Render the migration plan report** (template below) and stop — do not proceed to Phase B until the user approves it or asks for changes. + +### Migration plan report template + +```text +## Notebook Migration Plan: + +### Proposed identity (confirm with user) +| Field | Proposed | Notes | +|-------|----------|-------| +| name | | ^[a-z][a-z0-9_]*$ | +| group | | | +| topic | | | + +### Source & destination +- Source type: +- Extract pattern: +- Destination:
, write mode: + +### Cell classification +| Cell # | Stage | Summary | Disposition | +|--------|-------|---------|-------------| +| 1 | markdown | ... | keep as narrative | +| 2 | extract | fetches X via requests.get | migrate to extract() | +| 3 | exploratory | df.head() sanity check | drop — exploratory only | +| ... | | | | + +### Dropped cells (with reason) +- Cell #: + +### Magic commands found +- `` in cell # — dropped; + +### Pandas → SQL mapping +| Cell # | Pandas operation | Proposal | +|--------|-------------------|----------| +| 4 | `df.groupby("id").sum()` | Trino `GROUP BY id` | +| 5 | `pd.json_normalize(resp["items"])` | stays pandas (nested JSON) | + +### Flagged secrets — MUST resolve before migration +- Cell #: `` looks like a hardcoded . Move to env var / secret manager; do not carry into the new model. + +### Non-determinism / order-dependence risks +- Cell #: + +### Open questions for the user +- +``` + +- [ ] **10. Wait for the user to approve the plan, or apply requested edits and re-render.** Do not scaffold anything until this happens. + +## Phase B — Build (only after the user approves the plan) + +- [ ] **11. Hand off to `dj-create-python-model`.** Walk that skill's interactive gathering workflow (identity, DAG assignment, source type, transformation needs, output configuration, dependencies), but pre-fill every answer from the approved Phase A plan instead of asking the user again — only ask what the plan left open (see "Open questions"). +- [ ] **12. Use `dj-create-python-model`'s ETL cell structure and `_trino_io` helpers** for the generated `cells` array. Do not invent a different structure — this guarantees the output is schema-correct and consistent with hand-authored models. +- [ ] **13. After the `.python.json` is written**, tell the user the notebook can be retired once the model is verified — suggest running **`dj-verify-pymodel-parity`** against the old notebook's output table and the new model's output table before deleting the notebook. + +## Hard rules (DO NOT) + +- **DO NOT** scaffold or write any `.python.json` before the user has approved the Phase A plan. +- **DO NOT** echo a notebook cell's `outputs` array contents into the migration report — analyze `source` only. +- **DO NOT** carry a hardcoded secret found in the notebook into the new model, the plan report, or any generated code — flag it and stop short of reproducing it. +- **DO NOT** default risky ambiguous behavior (e.g., unclear write mode, unclear idempotency) silently — list it under "Open questions" and ask. +- **DO NOT** invent a different `.python.json` structure from what `dj-create-python-model` defines — Phase B must produce output that skill would also produce. + +## Reference + +For common notebook idiom → DJ/Trino equivalent mappings, see [references/notebook-pattern-mapping.md](references/notebook-pattern-mapping.md). + +For the full python model conventions this skill hands off to (ETL cell structure, `_trino_io` DML helpers, output config defaults, write-mode selection), see the **`dj-create-python-model`** skill — read it before Phase B. diff --git a/templates/skills/dj-migrate-notebook-to-pymodel/references/notebook-pattern-mapping.md b/templates/skills/dj-migrate-notebook-to-pymodel/references/notebook-pattern-mapping.md new file mode 100644 index 0000000..87697bc --- /dev/null +++ b/templates/skills/dj-migrate-notebook-to-pymodel/references/notebook-pattern-mapping.md @@ -0,0 +1,57 @@ +# Notebook pattern → DJ/Trino equivalent mapping + +Common Jupyter notebook idioms and the DJ python model / Trino equivalent to migrate them to. Use this alongside the SQL-first decision tree in `dj-create-python-model`. + +## Extract stage + +| Notebook idiom | Migrated equivalent | +| --- | --- | +| `requests.get(url, headers=...)` + `pd.json_normalize(resp.json())` | `extract(context)` — same HTTP call, stage the flattened rows into a Trino temp/staging table via `_trino_io` | +| `requests.get` with manual `while` pagination loop | Keep the pagination loop in `extract()`; accumulate all pages before staging | +| `pyodbc`/`sqlalchemy` query against another DB | `extract(context)` using the same driver; write results to a Trino staging table rather than holding the full result in a DataFrame for the rest of the notebook | +| `pd.read_csv("local/path.csv")` | `extract(context)` reading the same CSV (from S3/blob if it was a shared drive path), staged into Trino | +| `boto3.client("s3").get_object(...)` | `extract(context)` using `boto3`, staged into Trino | + +## Transform stage + +| Notebook idiom | Migrated equivalent | +| --- | --- | +| `df[df["col"] > 0]` | Trino `WHERE col > 0` | +| `df.rename(columns={"a": "b"})` | Trino `SELECT a AS b` | +| `df["col"].astype(int)` | Trino `CAST(col AS INTEGER)` | +| `df.drop_duplicates(subset=["id"])` | Trino `ROW_NUMBER() OVER (PARTITION BY id ORDER BY ...) = 1` filter | +| `df.groupby("id")["amount"].sum()` | Trino `SELECT id, SUM(amount) FROM ... GROUP BY id` | +| `pd.merge(df1, df2, on="id")` | Trino `JOIN ... ON id` | +| `df.sort_values("date")` | Trino `ORDER BY date` (usually unnecessary before a partitioned write) | +| `pd.json_normalize(nested_col)` on a column of dicts/lists | Stays pandas — SQL cannot easily flatten arbitrary nested JSON structures cell-by-cell | +| Custom Python string parsing / regex not expressible in SQL `REGEXP_*` | Stays pandas | +| `sklearn`/embedding/ML preprocessing | Stays pandas | + +## Load stage + +| Notebook idiom | Migrated equivalent | +| --- | --- | +| `df.to_sql(table, engine, if_exists="replace")` | `overwrite_partition(...)` or `overwrite(...)` from `_trino_io`, depending on whether it's a full refresh or one partition | +| `df.to_sql(table, engine, if_exists="append")` | `append(...)` from `_trino_io` | +| `df.to_parquet("s3://.../date=...")` | An `INSERT ... SELECT` writing to the Iceberg table's corresponding partition via `_trino_io` | +| `df.to_csv(...)` for manual inspection only | Drop — this was a debugging artifact, not part of the production load path | + +## Magic commands / notebook-only constructs (always dropped) + +| Idiom | Disposition | +| --- | --- | +| `%%time`, `%timeit` | Drop — no equivalent needed in a scheduled model | +| `!pip install ` | Drop the magic, but carry `` into the model's declared Python dependencies | +| `%matplotlib inline`, any `plt.*`/`sns.*`/`px.*` plotting cell | Drop — exploratory only | +| `display(df)`, bare `df` as last line of a cell, `df.head()`/`df.describe()`/`df.info()` | Drop — exploratory only | +| `%%bash`, `%%sh` shell-out cells | Drop unless the shell command performs a required step with no Python/SQL equivalent — flag for the user to confirm | +| Interactive widgets (`ipywidgets`, `input()` prompts) | Drop — a scheduled model cannot pause for interactive input; if the widget drove a parameter, surface it as a `context`/config value instead | + +## Non-determinism patterns to flag + +| Pattern | Why it's a risk | +| --- | --- | +| `datetime.now()` / `date.today()` without deriving from `context["ds"]` | Re-running the model for a historical `ds` would use today's date instead of the intended one | +| `random`/`np.random` without a fixed seed | Output differs across re-runs, breaking idempotency checks | +| A cell that only works if run out of written order (relies on a variable set by a cell below it, or by manual re-running) | Scheduled execution always runs cells top-to-bottom once — order-dependent logic will silently break or reference stale state | +| Global mutable state (a module-level list/dict appended to across multiple cells) | Doesn't survive being reorganized into discrete `extract`/`transform_and_load`/`cleanup` functions | diff --git a/templates/skills/dj-review-python-model/_SKILL.md b/templates/skills/dj-review-python-model/_SKILL.md index d4e0868..8dd50a9 100644 --- a/templates/skills/dj-review-python-model/_SKILL.md +++ b/templates/skills/dj-review-python-model/_SKILL.md @@ -26,6 +26,8 @@ Use this skill when the user mentions: review python model, audit python model, - Creating new Python models → `dj-create-python-model` - SQL `.model.json` review/refactoring → `dj-review-and-refactor-model` - Lightdash YAML → `dj-edit-lightdash-yaml` +- Verifying the model's output *data* against a legacy/reference table (this skill audits code, not data) → `dj-verify-pymodel-parity` +- Migrating a legacy notebook into a python model before it can be reviewed → `dj-migrate-notebook-to-pymodel` ## Workflow diff --git a/templates/skills/dj-verify-pymodel-parity/_SKILL.md b/templates/skills/dj-verify-pymodel-parity/_SKILL.md new file mode 100644 index 0000000..2cca250 --- /dev/null +++ b/templates/skills/dj-verify-pymodel-parity/_SKILL.md @@ -0,0 +1,59 @@ +--- +name: dj-verify-pymodel-parity +description: >- + Generate Trino SQL to verify that a python model's output table matches a + legacy/reference table — schema diff, per-partition row-count parity, + tolerance-based aggregate parity, and row-level diffs via full outer join or + checksum. Use when the user wants to verify, check, or prove parity between + an old table and a newly built or migrated python model's output table. Not + for creating the python model itself (-> dj-create-python-model or + dj-migrate-notebook-to-pymodel), executing the generated SQL (-> + dj-run-trino), or auditing the model's code for production readiness (-> + dj-review-python-model). +compatibility: DJ (Data JSON) Framework extension workspace with Trino access +metadata: + dj-skill: '1.0' +--- + +# Verify Python Model Table Parity + +**Goal:** generate Trino SQL that proves (or disproves) a new python model's output table matches an old/reference table, so the user can confirm a migration or rebuild produced the same data before retiring the old source. This skill is **read-only with respect to model files** — it only produces SQL and a report; it never edits `.python.json` / `.model.json`, and it never executes SQL itself. + +## When this skill applies + +Use this skill when the user mentions: verify parity, check parity, compare tables, row-by-row check, does the new table match the old one, or wants to validate a python model's output against a legacy table after a migration or rebuild. + +**Out of scope** — delegate to sibling skills: + +- Building the python model whose output is being verified → **`dj-create-python-model`** +- Migrating a legacy notebook into a python model (parity check is the natural next step after that) → **`dj-migrate-notebook-to-pymodel`** +- Actually running the generated SQL against Trino → **`dj-run-trino`** +- Auditing the model's code (not its data) for production readiness → **`dj-review-python-model`** + +## Workflow + +- [ ] **1. Gather inputs.** Ask for: + - Old/reference table: `catalog.schema.table` + - New table: `catalog.schema.table` + - Key column(s) that uniquely identify a row (for the row-level diff) + - Partition column (if any) and the range/filter to check (default: the most recent partition, widen only if asked) + - Float tolerance for aggregate comparisons (default: `1e-6`, ask if the data has known precision quirks) +- [ ] **2. Schema diff first.** Generate `SHOW COLUMNS FROM
` for both tables and compare the results before proposing any data comparison — column renames, drops, additions, or type changes should be surfaced and resolved before row-level SQL is written (a data diff against a mismatched schema produces misleading noise). +- [ ] **3. Row-count parity.** Generate a per-partition (and overall) `COUNT(*)` comparison. This is the cheapest, highest-signal check — surface any mismatched partition before running anything more expensive. +- [ ] **4. Aggregate parity.** For numeric/fact columns, generate `SUM`/`AVG`/`MIN`/`MAX`/`COUNT(DISTINCT ...)` per partition for both tables, comparing with `ABS(old - new) > ` for floating-point columns and exact equality for integers/counts. +- [ ] **5. Row-level diff.** Generate a `FULL OUTER JOIN` on the key column(s) (or `EXCEPT`/`INTERSECT` when an exact-match check is enough), restricted to the chosen partition/filter, that surfaces: rows only in old, rows only in new, and rows present in both but differing on a non-key column. For "differing on any column" without listing every column by hand, use a hash/checksum of the concatenated (cast-to-`VARCHAR`) non-key columns and compare hashes. +- [ ] **6. Apply the sampling guard.** Default every check to one partition (or date) at a time. Do not generate a full-table diff across all history unless the user explicitly asks for it — state this default and let the user widen scope. +- [ ] **7. Render the output.** Produce clearly labeled SQL sections (schema diff / row counts / aggregates / row-level diff) as a `.draft.sql` file or in-chat SQL blocks, plus a plain-language summary of what each section checks. Offer to hand execution to **`dj-run-trino`** — never run the SQL yourself. +- [ ] **8. Interpret results if the user shares them back.** If the user pastes query results, summarize pass/fail per section and suggest likely root causes for any failure (e.g., a missing dedup step, a timezone shift, a partition filter off by one day) — but do not edit any model file to fix it; that's a manual follow-up once root cause is clear. + +## Hard rules (DO NOT) + +- **DO NOT** generate anything other than read-only `SELECT` / `SHOW COLUMNS` / `DESCRIBE` SQL. No DDL/DML. +- **DO NOT** execute SQL yourself — always hand off to `dj-run-trino`, which owns confirmation and connection resolution. +- **DO NOT** target a production catalog/schema without the user's explicit confirmation of the connection — mirror `dj-run-trino`'s safety posture. +- **DO NOT** diff a full table across all history by default — always scope to one partition/filter first (see sampling guard) and state that default in the response. +- **DO NOT** edit `.python.json` or `.model.json` to "fix" a mismatch — this skill only reports. + +## Reference + +For copy-paste SQL templates (count parity, checksum-based row diff, tolerance-based aggregate diff, null-safe key joins, and handling schema drift), see [references/parity-recipes.md](references/parity-recipes.md). diff --git a/templates/skills/dj-verify-pymodel-parity/references/parity-recipes.md b/templates/skills/dj-verify-pymodel-parity/references/parity-recipes.md new file mode 100644 index 0000000..aae133f Binary files /dev/null and b/templates/skills/dj-verify-pymodel-parity/references/parity-recipes.md differ