Replace ReactFlow+ELK pipeline diagrams with Next.js+Graphviz and add CI auto-update

[anth-volk]

Summary

Migrate the pipeline diagram app from Vite + ReactFlow + ELK to Next.js + WASM Graphviz with DOT files as source of truth, and add a CI workflow that automatically keeps diagrams in sync with code changes.

Current state

The pipeline diagrams are hand-coded React components (src/stages/*.jsx) that define nodes and edges in JavaScript. Updating them requires manually editing JSX, and they frequently drift out of sync with the actual Python pipeline code.

Proposed changes

1. Migrate to Next.js + Graphviz DOT files

• Replace Vite + ReactFlow + ELK with Next.js + @hpcc-js/wasm-graphviz
• Use plain-text DOT files (public/dots/{us,uk}/stage_N.dot) as the diagram source of truth
• Client-side rendering with zoom/pan via CSS transforms
• Shared sidebar, legend, and layout components
• Remove ~10,500 lines of hand-coded JSX stage definitions

2. Add manifest.json as single source of truth

• Machine-readable manifest defining all stages, titles, DOT paths, and Python source file mappings
• Sidebar and page components derive their data from the manifest (no hardcoded stage lists)
• crossStageUtilities section maps shared utility files to the stages they affect

3. Add CI auto-update workflow (.github/workflows/update_pipeline_diagrams.yaml)

A 3-phase pipeline that runs on PRs changing Python source files:

• Phase 1 — Stage list evaluation: Detects unmapped Python files, uses AI to determine if the stage list itself needs updating (conservative, biased toward NO)
• Phase 2 — Per-stage DOT update: Maps changed files to affected stages via manifest globs, spawns parallel Claude Code CLI agents (up to 4) to read the code changes and update DOT diagrams
• Phase 3 — Structural validation: Validates modified DOT files with @hpcc-js/wasm-graphviz, spawns fix agents on failure (up to 3 retries), reverts to HEAD on persistent failure

Cost controls:

• --max-turns limits per phase (6 / 10 / 4)
• Max 4 stages processed per run
• --model sonnet for all agents
• Estimated ~$0.50–$1.50 per PR run

4. DOT style guide (docs/pipeline-diagrams/CLAUDE.md)

• Auto-loaded by Claude Code CLI agents when spawned from the diagrams directory
• Defines node type colors, edge type colors, HTML label format, and node ID conventions

Files added/modified

File	Description
docs/pipeline-diagrams/manifest.json	Stage definitions + source file mappings
docs/pipeline-diagrams/CLAUDE.md	DOT style guide for AI agents
docs/pipeline-diagrams/scripts/update-diagrams.mjs	3-phase orchestrator
.github/workflows/update_pipeline_diagrams.yaml	GitHub Actions workflow
docs/pipeline-diagrams/app/	Next.js app directory (pages, layout)
docs/pipeline-diagrams/components/	Diagram, Sidebar, legends
docs/pipeline-diagrams/public/dots/	US (9) + UK (9) DOT files + overviews
docs/pipeline-diagrams/scripts/convert-to-dot.mjs	One-time JSX→DOT converter

Testing

• Next.js build passes
• Orchestrator tested locally against 86 Python files changed on main (30 commits)
• Phase 1 correctly identified unmapped files and determined no stage list change needed
• Phase 2 correctly determined code changes were internal logic (no structural diagram changes)
• Phase 3 validation path verified (no DOT files modified in test run)