|
| 1 | +# Design: `expand` and `restrict` — the diagram-traversal algebra |
| 2 | + |
| 3 | +Status: proposed (targets the 2.4 line). Supersedes the separate |
| 4 | +`Diagram.cascade` / `Diagram.trace` / `Diagram.restrict` operators with a small, |
| 5 | +composable algebra. `cascade` and `trace` are retained as aliases. |
| 6 | + |
| 7 | +This document derives the design from first principles so the rules — not just |
| 8 | +the API — are the thing agreed upon. |
| 9 | + |
| 10 | +--- |
| 11 | + |
| 12 | +## 1. The one primitive: propagating a restriction across a foreign key |
| 13 | + |
| 14 | +A **restriction** on a table is a subset of its rows, written as a condition. |
| 15 | + |
| 16 | +Every foreign key `child → parent` defines a function: each child row references |
| 17 | +exactly one parent row. Propagating a restriction across that edge is a |
| 18 | +**semijoin** (restrict-by-matching), and it works in either direction: |
| 19 | + |
| 20 | +- **downstream** (parent restricted `T'` → child): `S' = S ⋉ T'` — the children |
| 21 | + whose parent is in `T'`. |
| 22 | +- **upstream** (child restricted `S'` → parent): `T' = T ⋉ S'` — the parents |
| 23 | + referenced by `S'`. |
| 24 | + |
| 25 | +Downstream and upstream are the *same* operation pointed opposite ways along the |
| 26 | +same FK. This is the whole engine; everything below is about how the edge rule |
| 27 | +degenerates, what parts add, and how you accumulate across many edges. |
| 28 | + |
| 29 | +## 2. The edge rule (R1, referential), derived by adding complications |
| 30 | + |
| 31 | +**Base case — FK is the whole primary key, not renamed, no parts.** The parent's |
| 32 | +primary key is embedded verbatim in the child's, with the same column names. A |
| 33 | +primary-key restriction on the parent is a predicate on exactly those columns, |
| 34 | +which the child also has, by the same names. So: |
| 35 | + |
| 36 | +> **Carry the restriction unchanged** — the identical predicate that selects the |
| 37 | +> parent rows already selects the matching child rows. |
| 38 | +
|
| 39 | +This is why "the same primary-key restriction rides the whole diagram": every |
| 40 | +table on the path shares those PK columns by name. |
| 41 | + |
| 42 | +**Complication A — secondary foreign key.** The parent's key lands in the child's |
| 43 | +*secondary* (non-PK) attributes. A raw predicate still selects the right child |
| 44 | +rows, but the restriction is no longer a statement about the child's *identity*, |
| 45 | +so it cannot be promoted to the child's PK and ridden further. Keep it |
| 46 | +relational: **project the restricted parent to its key and semijoin the child on |
| 47 | +the FK columns.** |
| 48 | + |
| 49 | +**Complication B — renamed foreign key.** The referencing columns have different |
| 50 | +names in the child. The parent's predicate names columns the child lacks. Fix, |
| 51 | +mechanically: **rename the restriction's columns through the FK's attribute map** |
| 52 | +before the semijoin (reverse the rename going upstream). |
| 53 | + |
| 54 | +**Unification.** These are one rule with two degenerate fast paths: |
| 55 | + |
| 56 | +> **R1 (edge rule):** propagate a restriction across an FK edge as a **semijoin** |
| 57 | +> against the restricted neighbor, projected/renamed onto the shared FK columns. |
| 58 | +> When the FK is the whole primary key and unrenamed, the projection is the |
| 59 | +> identity and the semijoin collapses to "apply the same predicate." |
| 60 | +
|
| 61 | +(R1 absorbs what were previously six separate rules: forward F1/F2/F3 and upward |
| 62 | +U1/U2/U3.) |
| 63 | + |
| 64 | +## 3. The group rule (R2, compositional) |
| 65 | + |
| 66 | +Part tables add **compositional** integrity on top of referential integrity: a |
| 67 | +master and its parts are one entity, created and deleted all-or-nothing. |
| 68 | + |
| 69 | +- **master → part** needs nothing new — a part carries `→ master` in its PK, so |
| 70 | + R1 already sweeps in all parts of a restricted master. |
| 71 | +- **part → master** is the new rule. A restriction landing on part rows satisfies |
| 72 | + referential integrity by touching just those rows, but leaves a fragment of an |
| 73 | + entity. So it must **lift existentially to the master** (the master is in if |
| 74 | + *any* of its parts is), and the master re-expands to **all** its parts. |
| 75 | + |
| 76 | +> **R2 (group rule):** a restriction touching any part of a master's group brings |
| 77 | +> the whole group — existential lift part→master, then expand master→all parts. |
| 78 | +
|
| 79 | +R2 is a closure over the master–part grouping, which is exactly why FK semijoins |
| 80 | +alone can't express it. |
| 81 | + |
| 82 | +## 4. Two operations over R1 + R2 |
| 83 | + |
| 84 | +There are exactly two irreducible ways to use the rules, and they are opposites. |
| 85 | + |
| 86 | +### `expand` — additive (grow from one seed) |
| 87 | + |
| 88 | +A **constructor**. Seed a single restricted table and grow outward by R1 + R2, |
| 89 | +accumulating reachable rows by **union** (a table is reached if reachable via any |
| 90 | +path). Directional: |
| 91 | + |
| 92 | +``` |
| 93 | +Diagram.expand(seed, direction="down" | "up" | "both") |
| 94 | +``` |
| 95 | + |
| 96 | +- `direction="down"` — descendants: the **delete blast radius**. |
| 97 | +- `direction="up"` — ancestors: the **valid query sources** a `make()` may read |
| 98 | + under the reproducibility contract. |
| 99 | +- `direction="both"` — a referentially-consistent **export region** around the |
| 100 | + seed. |
| 101 | + |
| 102 | +A single-seed additive closure is always consistent and never needs an |
| 103 | +intersection: tracing up pulls exactly the referenced ancestors, cascading down |
| 104 | +pulls exactly the dependents — all exact semijoins. |
| 105 | + |
| 106 | +**Retained aliases:** `Diagram.cascade(seed) = expand(seed, "down")` and |
| 107 | +`Diagram.trace(seed) = expand(seed, "up")`. Inside `make()`, |
| 108 | +`self.upstream = Diagram.expand(self & key, direction="up")`. |
| 109 | + |
| 110 | +### `restrict` — subtractive (progressively carve any diagram) |
| 111 | + |
| 112 | +An **instance method** on any Diagram (including one built by `expand`). Carves |
| 113 | +the diagram down by applying conditions, accumulating by **intersection**: |
| 114 | + |
| 115 | +``` |
| 116 | +diagram.restrict(*conditions, direction="down" | "up" | "both") |
| 117 | +``` |
| 118 | + |
| 119 | +Each condition propagates by R1 (in the chosen direction) and R2. The result: |
| 120 | +**every table is restricted by the conjunction of all conditions that reach it**; |
| 121 | +tables that go empty drop out. Properties: |
| 122 | + |
| 123 | +- **Progressive / chainable** — `.restrict(A).restrict(B)…`, each carves further. |
| 124 | +- **Order-independent** — the result is the conjunction of all conditions, so |
| 125 | + order doesn't matter. |
| 126 | +- **Monotone** — every step only removes; every intermediate is a valid slice. |
| 127 | + |
| 128 | +`restrict` is *not* reducible to combinations of `expand`, because it applies |
| 129 | +**multiple independent conditions** and gives each table the AND of the ones |
| 130 | +upstream of it. Example: "all data for `mouse_id=5` **and** `method_id=5`" — |
| 131 | +`Mouse` and `ProcessingMethod` are independent ancestors meeting only at a shared |
| 132 | +descendant; `expand` + combine cannot assemble it, `restrict.restrict` does. |
| 133 | + |
| 134 | +## 5. Why this is the whole story |
| 135 | + |
| 136 | +- **Intersection is not a convergence rule.** It only arises in the subtractive |
| 137 | + model with multiple independent conditions (`restrict`). The additive model |
| 138 | + (`expand`) is pure reachability — always union. |
| 139 | +- **`expand` and `restrict` compose freely.** A Diagram is a set of tables, each |
| 140 | + holding one row-set. `expand` **unions** reachable rows in (grow); `restrict` |
| 141 | + **intersects** a propagated condition in (carve). One representation, so they |
| 142 | + chain in any order — `Diagram.expand(seed, "both").restrict(cond).restrict(cond)` |
| 143 | + — and the current "cascade and restrict are mutually exclusive" wall is |
| 144 | + removed. Per-table combine is unambiguous because composition order is explicit |
| 145 | + (∪ at a grow step, ∩ at a carve step). |
| 146 | +- **Materialization is a delete-time concern, not part of traversal.** Freezing a |
| 147 | + group's keys before deleting (delete runs parts-before-masters) matters only |
| 148 | + when a traversal feeds `delete`; the read-only closures never pay for it. |
| 149 | + |
| 150 | +## Summary |
| 151 | + |
| 152 | +| | additive (grow) | subtractive (carve) | |
| 153 | +|---|---|---| |
| 154 | +| operator | `expand(seed, direction)` — constructor | `diagram.restrict(*conds, direction)` — transform | |
| 155 | +| accumulate | union | intersection | |
| 156 | +| serves | blast radius / make() sources / export region | multi-condition pipeline carving | |
| 157 | +| aliases | `cascade`=down, `trace`=up | — | |
| 158 | + |
| 159 | +One data structure, two composable transforms, two rules (R1 edge-semijoin, R2 |
| 160 | +group). `cascade`/`trace` survive as named shortcuts. |
| 161 | + |
| 162 | +## Open / follow-ups |
| 163 | + |
| 164 | +- **`restrict` direction default and the both-way carve** — confirm whether a |
| 165 | + condition on a descendant should also trim ancestors to those still referenced |
| 166 | + (`direction="both"`) for a fully-consistent export slice; pick the default. |
| 167 | +- **A3 / #1481** — `direction="up"` applies R2, so `trace` descends from an |
| 168 | + ancestor master into its parts (reproducibility-contract grounds). This flips |
| 169 | + the currently-pinned `test_trace_stops_at_master_no_part_down_collection`; |
| 170 | + update test and `trace.md` together. |
| 171 | +- **Platform behavior** (Pipeline Navigator: progressive vs batch carving UI) is |
| 172 | + a product decision; the library only guarantees the algebra above. |
| 173 | +- **Release scope** — this is a 2.4 API evolution. The 2.3.3 line keeps the |
| 174 | + landed multi-FK-path fix + alias-docstring scrub and the non-traversal items; |
| 175 | + #1496/#1501/#1481 fold into this redesign rather than shipping as standalone |
| 176 | + 2.3.3 patches. |
0 commit comments