Skip to content

Commit 3867163

Browse files
docs(design): derive the expand/restrict diagram-traversal algebra
Proposed 2.4 design that collapses Diagram.cascade / Diagram.trace / Diagram.restrict into a small composable algebra, derived from first principles: - One primitive: propagate a restriction across a foreign key by semijoin, either direction. - R1 (edge rule): the semijoin, with identity fast-path for primary, non-renamed FKs; absorbs the former F1-3 / U1-3 rules. - R2 (group rule): a restriction touching any part of a master group brings the whole group (existential lift + re-expand). - Two operations: expand(seed, direction) — additive constructor (cascade=down, trace=up retained as aliases); and diagram.restrict(*conditions, direction) — subtractive progressive carving. They compose freely on one representation (union to grow, intersection to carve), removing the current cascade-vs-restrict mutual exclusivity. Includes derivation, the mouse/method counterexample showing restrict is not reducible to expand, and open follow-ups (restrict direction default, #1481 master->parts trace descent, release scoping).
1 parent afc350f commit 3867163

1 file changed

Lines changed: 176 additions & 0 deletions

File tree

‎DESIGN-expand-restrict.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# Design: `expand` and `restrict` — the diagram-traversal algebra
2+
3+
Status: proposed (targets the 2.4 line). Supersedes the separate
4+
`Diagram.cascade` / `Diagram.trace` / `Diagram.restrict` operators with a small,
5+
composable algebra. `cascade` and `trace` are retained as aliases.
6+
7+
This document derives the design from first principles so the rules — not just
8+
the API — are the thing agreed upon.
9+
10+
---
11+
12+
## 1. The one primitive: propagating a restriction across a foreign key
13+
14+
A **restriction** on a table is a subset of its rows, written as a condition.
15+
16+
Every foreign key `child → parent` defines a function: each child row references
17+
exactly one parent row. Propagating a restriction across that edge is a
18+
**semijoin** (restrict-by-matching), and it works in either direction:
19+
20+
- **downstream** (parent restricted `T'` → child): `S' = S ⋉ T'` — the children
21+
whose parent is in `T'`.
22+
- **upstream** (child restricted `S'` → parent): `T' = T ⋉ S'` — the parents
23+
referenced by `S'`.
24+
25+
Downstream and upstream are the *same* operation pointed opposite ways along the
26+
same FK. This is the whole engine; everything below is about how the edge rule
27+
degenerates, what parts add, and how you accumulate across many edges.
28+
29+
## 2. The edge rule (R1, referential), derived by adding complications
30+
31+
**Base case — FK is the whole primary key, not renamed, no parts.** The parent's
32+
primary key is embedded verbatim in the child's, with the same column names. A
33+
primary-key restriction on the parent is a predicate on exactly those columns,
34+
which the child also has, by the same names. So:
35+
36+
> **Carry the restriction unchanged** — the identical predicate that selects the
37+
> parent rows already selects the matching child rows.
38+
39+
This is why "the same primary-key restriction rides the whole diagram": every
40+
table on the path shares those PK columns by name.
41+
42+
**Complication A — secondary foreign key.** The parent's key lands in the child's
43+
*secondary* (non-PK) attributes. A raw predicate still selects the right child
44+
rows, but the restriction is no longer a statement about the child's *identity*,
45+
so it cannot be promoted to the child's PK and ridden further. Keep it
46+
relational: **project the restricted parent to its key and semijoin the child on
47+
the FK columns.**
48+
49+
**Complication B — renamed foreign key.** The referencing columns have different
50+
names in the child. The parent's predicate names columns the child lacks. Fix,
51+
mechanically: **rename the restriction's columns through the FK's attribute map**
52+
before the semijoin (reverse the rename going upstream).
53+
54+
**Unification.** These are one rule with two degenerate fast paths:
55+
56+
> **R1 (edge rule):** propagate a restriction across an FK edge as a **semijoin**
57+
> against the restricted neighbor, projected/renamed onto the shared FK columns.
58+
> When the FK is the whole primary key and unrenamed, the projection is the
59+
> identity and the semijoin collapses to "apply the same predicate."
60+
61+
(R1 absorbs what were previously six separate rules: forward F1/F2/F3 and upward
62+
U1/U2/U3.)
63+
64+
## 3. The group rule (R2, compositional)
65+
66+
Part tables add **compositional** integrity on top of referential integrity: a
67+
master and its parts are one entity, created and deleted all-or-nothing.
68+
69+
- **master → part** needs nothing new — a part carries `→ master` in its PK, so
70+
R1 already sweeps in all parts of a restricted master.
71+
- **part → master** is the new rule. A restriction landing on part rows satisfies
72+
referential integrity by touching just those rows, but leaves a fragment of an
73+
entity. So it must **lift existentially to the master** (the master is in if
74+
*any* of its parts is), and the master re-expands to **all** its parts.
75+
76+
> **R2 (group rule):** a restriction touching any part of a master's group brings
77+
> the whole group — existential lift part→master, then expand master→all parts.
78+
79+
R2 is a closure over the master–part grouping, which is exactly why FK semijoins
80+
alone can't express it.
81+
82+
## 4. Two operations over R1 + R2
83+
84+
There are exactly two irreducible ways to use the rules, and they are opposites.
85+
86+
### `expand` — additive (grow from one seed)
87+
88+
A **constructor**. Seed a single restricted table and grow outward by R1 + R2,
89+
accumulating reachable rows by **union** (a table is reached if reachable via any
90+
path). Directional:
91+
92+
```
93+
Diagram.expand(seed, direction="down" | "up" | "both")
94+
```
95+
96+
- `direction="down"` — descendants: the **delete blast radius**.
97+
- `direction="up"` — ancestors: the **valid query sources** a `make()` may read
98+
under the reproducibility contract.
99+
- `direction="both"` — a referentially-consistent **export region** around the
100+
seed.
101+
102+
A single-seed additive closure is always consistent and never needs an
103+
intersection: tracing up pulls exactly the referenced ancestors, cascading down
104+
pulls exactly the dependents — all exact semijoins.
105+
106+
**Retained aliases:** `Diagram.cascade(seed) = expand(seed, "down")` and
107+
`Diagram.trace(seed) = expand(seed, "up")`. Inside `make()`,
108+
`self.upstream = Diagram.expand(self & key, direction="up")`.
109+
110+
### `restrict` — subtractive (progressively carve any diagram)
111+
112+
An **instance method** on any Diagram (including one built by `expand`). Carves
113+
the diagram down by applying conditions, accumulating by **intersection**:
114+
115+
```
116+
diagram.restrict(*conditions, direction="down" | "up" | "both")
117+
```
118+
119+
Each condition propagates by R1 (in the chosen direction) and R2. The result:
120+
**every table is restricted by the conjunction of all conditions that reach it**;
121+
tables that go empty drop out. Properties:
122+
123+
- **Progressive / chainable** — `.restrict(A).restrict(B)…`, each carves further.
124+
- **Order-independent** — the result is the conjunction of all conditions, so
125+
order doesn't matter.
126+
- **Monotone** — every step only removes; every intermediate is a valid slice.
127+
128+
`restrict` is *not* reducible to combinations of `expand`, because it applies
129+
**multiple independent conditions** and gives each table the AND of the ones
130+
upstream of it. Example: "all data for `mouse_id=5` **and** `method_id=5`" —
131+
`Mouse` and `ProcessingMethod` are independent ancestors meeting only at a shared
132+
descendant; `expand` + combine cannot assemble it, `restrict.restrict` does.
133+
134+
## 5. Why this is the whole story
135+
136+
- **Intersection is not a convergence rule.** It only arises in the subtractive
137+
model with multiple independent conditions (`restrict`). The additive model
138+
(`expand`) is pure reachability — always union.
139+
- **`expand` and `restrict` compose freely.** A Diagram is a set of tables, each
140+
holding one row-set. `expand` **unions** reachable rows in (grow); `restrict`
141+
**intersects** a propagated condition in (carve). One representation, so they
142+
chain in any order — `Diagram.expand(seed, "both").restrict(cond).restrict(cond)`
143+
— and the current "cascade and restrict are mutually exclusive" wall is
144+
removed. Per-table combine is unambiguous because composition order is explicit
145+
(∪ at a grow step, ∩ at a carve step).
146+
- **Materialization is a delete-time concern, not part of traversal.** Freezing a
147+
group's keys before deleting (delete runs parts-before-masters) matters only
148+
when a traversal feeds `delete`; the read-only closures never pay for it.
149+
150+
## Summary
151+
152+
| | additive (grow) | subtractive (carve) |
153+
|---|---|---|
154+
| operator | `expand(seed, direction)` — constructor | `diagram.restrict(*conds, direction)` — transform |
155+
| accumulate | union | intersection |
156+
| serves | blast radius / make() sources / export region | multi-condition pipeline carving |
157+
| aliases | `cascade`=down, `trace`=up | — |
158+
159+
One data structure, two composable transforms, two rules (R1 edge-semijoin, R2
160+
group). `cascade`/`trace` survive as named shortcuts.
161+
162+
## Open / follow-ups
163+
164+
- **`restrict` direction default and the both-way carve** — confirm whether a
165+
condition on a descendant should also trim ancestors to those still referenced
166+
(`direction="both"`) for a fully-consistent export slice; pick the default.
167+
- **A3 / #1481** — `direction="up"` applies R2, so `trace` descends from an
168+
ancestor master into its parts (reproducibility-contract grounds). This flips
169+
the currently-pinned `test_trace_stops_at_master_no_part_down_collection`;
170+
update test and `trace.md` together.
171+
- **Platform behavior** (Pipeline Navigator: progressive vs batch carving UI) is
172+
a product decision; the library only guarantees the algebra above.
173+
- **Release scope** — this is a 2.4 API evolution. The 2.3.3 line keeps the
174+
landed multi-FK-path fix + alias-docstring scrub and the non-traversal items;
175+
#1496/#1501/#1481 fold into this redesign rather than shipping as standalone
176+
2.3.3 patches.

0 commit comments

Comments
 (0)