Skip to content

The newest export was on disk for six days and no file here named it - #511

Merged
syncytium2 merged 1 commit into
mainfrom
full-cohort-senktide-ttx
Sep 9, 2026
Merged

The newest export was on disk for six days and no file here named it#511
syncytium2 merged 1 commit into
mainfrom
full-cohort-senktide-ttx

Conversation

@syncytium2

Copy link
Copy Markdown
Owner

Groundwork for running the full senktide and TTX cohorts through the loop. The run itself is still going; this is everything it needed that did not exist, landed on its own so the branch does not carry code and results together.

The finding that started it

Tony asked for the full senktide and TTX data set "with excluded artifacts", and the first lookup turned up an export the repo had never heard of. interface2 shipped 2026-09-03_revised_2v_long_STEPS_EXCLUDED six days ago — artifacts removed, pre-split into SENKTIDE (29 recordings) and TTX (38) — and its own README says "for any new analysis, use this folder." No file here named it. current_export.toml, which exists because a session once could not tell which folder was current, still pointed at the 2026-08-18 export.

It is declared now as three roles beside default, not instead of it. Moving default would silently re-point every existing analysis and every parity fixture at a different corpus; that is a decision for Tony, and the INDEX row says so.

Three things the run could not honour

was now
K = 10% floored at 3 not expressible — k_from_fraction clamps at 1 and leaves the clamp to the caller, and nobody had written the caller --k-floor on assess, assess_archive.py and derive_spec.py; every result records whether the floor bound there
a spec from a percentage derive_spec selects one absolute-K column, which drops every recording whose K landed elsewhere --k-percent aggregates across recordings at each one's own count
group facets one page, all detectors stacked, no groups --per-detector --facet group

The floor is not cosmetic here. The APV+CNQX+GZ pilot filed this same gap and got away with it because that cohort's field sizes made 10% and floor-3 the same K on every recording. On this corpus the floor binds on 26 of 84 recordings, so the population is governed by two K rules at once — which is why a bound floor is recorded per recording rather than absorbed into the number.

And selecting a column would have been quiet. 10% floored at 3 resolves to K=3 on 55 recordings, 4 on 15, 5 on 12 and 6 on 2. The median column alone holds 55 of 84; the other 29 would have left the spec with nothing said about it.

k_source reads percentage_unreviewed, never mahice. MAHICE was skipped on instruction, so this K has the shape of one a person would set and no person set it, and a consumer has to be able to tell those apart.

Two silent defects in code that was already shipping

The shared PNG renderer clipped at a literal 1120.0 — the old viewport as a constant — so widening the viewport widened the window and clipped back anyway. A four-facet page rendered as three facets and a sliced fourth, with no error and a file that looks finished unless somebody opens it. Same shape as the height bug documented two comments above it in the same function.

make_detector_table.py reads rates() by tuple position across a module boundary. Adding the group field turned it into a TypeError — the good outcome; a shifted numeric field would have been wrong numbers and no complaint. Fixed, with the reason written at the call site.

Checks

Full suite green — 2270 passed, 46 skipped. Sapper clear, check_quotes clear. New tests pin the floor (that it applies after the fraction meets the recording, that a bound floor is recorded rather than absorbed, that it is refused under an absolute K) and the facets (group carried unchanged, unlabelled facet drawn last, inks stable across a run's pages).

🤖 Generated with Claude Code

Tony: "i fear our ability to document progress is faulty." The first lookup of
the run he asked for proved him right. interface2 shipped
`2026-09-03_revised_2v_long_STEPS_EXCLUDED` on 2026-09-03, with the field-step
artifacts removed and pre-split into SENKTIDE (29) and TTX (38) cohorts. Its own
README says "for any new analysis, use this folder." Nothing in this repo
mentioned it, and `current_export.toml` — the file that exists BECAUSE a session
could not tell which folder was current — still named the 2026-08-18 export.

It is declared now, as three roles beside `default` rather than instead of it:
moving `default` re-points every existing analysis and every parity fixture at a
different corpus, which is a decision and not a housekeeping edit. The role notes
carry what the folder's README paid to establish — that a whole-field brightness
step recruits 74% (fast) and 82% (slow) of ROIs at 8.5x and 22.7x each slice's own
rate, in both streams, which is the signature a coordination detector exists to
find; that the pre-artifact folder is kept as a paired control; and that `io.py`
still drops `on_field_step` and `field_step_id` silently, which costs nothing in
an excluded folder and would cost the flagged companion everything.

Three things the run could not honour until they existed.

K = 10% FLOORED AT 3 was not expressible. `k_from_fraction` clamps at 1 and says
in its own docstring that the clamp belongs to the caller; nobody had written the
caller. The APV+CNQX+GZ pilot recorded this as a gap and got away with it because
that cohort's field sizes made 10% and floor-3 the same K on every recording.
Here they are not: the floor BINDS on 26 of 84 recordings, so the corpus is
governed by two K rules at once. `--k-floor` now exists on `bugarach assess`,
`assess_archive.py` and `derive_spec.py`, every Assessment records whether the
floor bound there, and it is refused under an absolute K, where it would be a
second opinion about a number somebody already chose.

DERIVE_SPEC COULD ONLY WORK AT ONE ABSOLUTE K. That is right for an assessment
holding every recording at every K, and wrong for one produced at a single
percentage: each recording appears once, at its own count, so selecting a column
would have silently dropped most of the corpus — 10% floored at 3 lands on
K=3,4,5,6 here and the median column holds 55 of 84. `--k-percent` aggregates
across recordings at each one's own count, which is what "one percentage, fair
across the folder" means. It stamps `k_source: percentage_unreviewed`, never
`mahice`: it has the shape of a K a person would set and no person set it.

THE BEFORE/AFTER FIGURE HAD NO GROUP FACETS. FOUNDATIONS §9 says effects run in
opposite directions by group, so a pooled panel can hide a sign change.
`--per-detector --facet group` writes one page per detector with one group per
column, y shared across a stream's facets and nowhere else.

Two silent defects found on the way, both in code that was already shipping.

The shared PNG renderer clipped at a literal `1120.0` — the old viewport as a
constant — so widening the viewport widened the window and clipped back anyway.
A four-facet page came out with three facets and a sliced fourth, no error, and a
file that looks finished unless somebody opens it. That is the same shape as the
height bug two comments above it in the same function.

`make_detector_table.py` reads `rates()` by tuple position across a module
boundary. Adding the group field turned it into a TypeError, which is the good
outcome — a shifted numeric field would have been wrong numbers and no complaint.

Full suite green (2270 passed); sapper clear.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@syncytium2
syncytium2 enabled auto-merge (squash) September 9, 2026 19:18
@syncytium2
syncytium2 merged commit f33bb0e into main Sep 9, 2026
3 checks passed
@syncytium2
syncytium2 deleted the full-cohort-senktide-ttx branch September 9, 2026 19:33
syncytium2 added a commit that referenced this pull request Sep 9, 2026
…pend what #507-509 bought (#512)

The full senktide (29) and TTX (38) cohorts, artifact-excluded, assessed at K = 10% of
each recording's ROI population floored at 3, simulated from their own baseline,
calibrated and trained on 72 held-out simulated recordings, detected at that calibration
with those trained models, and drawn as 24 pages -- twelve detectors x two treatments,
four group facets x two streams each.

Everything derived from the recordings is in the darkroom, per FOUNDATIONS §5. This
handoff is the pointer and carries no such number; the run record beside the artifacts
carries all of them.

WHAT IS DIFFERENT FROM THE PILOT is most of what mattered. Both of pipeline.md's blockers
closed yesterday and this run is the first to spend them: the six ran at a calibration
derived from this cohort's own simulated data rather than at shipped operating points, and
all six learned models ran from checkpoints in a separate process from training. Two more
things the pilot could not do: background heterogeneity was FITTED from the folder rather
than inherited -- eighty-four baselines where the pilot had six -- and the input was the
producer's artifact-excluded export rather than a folder nobody had scanned.

WHAT IT COST TO GET THERE is in #511: the newest export had been on disk six days with no
file in this repo naming it, K as a percentage had no floor, derive_spec could only work at
one absolute K, and the figure had no group facets and clipped its own fourth column.

THE EDGE-OF-GRID ASYMMETRY is filed rather than left in a transcript. `tube` is the best
learned model on this cohort and its deployed threshold sits at the top of the searched
grid. The coded branch treats that as a refusal in terms -- "an optimum at the edge is not
an optimum, it is the search telling you it stopped too early" -- and the learned branch
warns and ships. The todo does not assume which side is right: for a probability threshold
an edge may mean the model separates cleanly, and there is one measurement that tells the
two apart. Nobody has run it.

THE FIREFLIES NOTE IS MARKED AS A DRAFT AND MUST NOT BE SENT. It is addressed to another
team, which makes it a document deliverable, and it has not been reviewed or released.

Co-authored-by: richard defazio <defazio@umich.edu>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants