The newest export was on disk for six days and no file here named it

current_export.toml

@@ -41,6 +41,71 @@ note = """
 The standard export. Two streams, region labels, and the full per-event set the
 contract asks for — `time_sec` (t50rise), `width_sec` + `width_def`, `peak_sec`, `amp`.
 Carries no `analysis_*` columns, deliberately.
+
+⚠ NOT the newest folder the producer has shipped, and `default` is deliberately left
+pointing here rather than moved. `steps_excluded` below is 2026-09-03 and its own README
+says "for any new analysis, use this folder"; moving `default` would silently re-point
+every existing analysis and every parity fixture at a different corpus. Changing it is a
+decision, not a housekeeping edit — see the note on `steps_excluded`.
+"""
+
+[steps_excluded]
+# The producer's 2026-09-03 export with field-step artifacts REMOVED.
+name = "2026-09-03_revised_2v_long_STEPS_EXCLUDED"
+recordings = 84
+note = """
+Whole-field brightness steps GENERATE detections — onsets lock to the step, 74% (fast)
+and 82% (slow) of ROIs fire at once, at 8.5x and 22.7x each slice's own rate, in both
+streams. That is the signature a coordination detector exists to find, and it was in the
+data. Nine steps confirmed by eye; every event within ±2 s of one removed (381 events,
+9 slices), each listed in `field_steps_excluded.tsv`.
+
+Pairs with `2026-09-03_revised_2v_long_PRE_ARTIFACT_KILLER`, which keeps the artifacts and
+is on disk so the runs made before the removal stay reproducible. `slices.csv` and
+`regions.csv` are byte-identical between the two, and the 75 unaffected recordings are
+byte-identical once the two new columns are stripped — so the pair is a clean control.
+
+Carries `analysis_start_sec` / `analysis_end_sec` (the `long_window_20` regime), which
+`default` does not, plus two new per-event columns: `on_field_step` and `field_step_id`.
+
+⚠ THE LOADER DROPS BOTH NEW COLUMNS SILENTLY. `io.py` recognises only width_sec /
+width_def / peak_sec / amp beyond roi and time_sec, and `store.Stream` has no per-event
+flag field, so `csv.DictReader` picks them up and they are discarded with no error. In
+THIS folder that costs nothing — every surviving row reads `0` — but it is why the
+FLAGGED_FOR_REVIEW companion cannot be read correctly here yet.
+
+⚠ Known contamination this folder does NOT address: non-rigid motion correction pinned
+12 ROIs to the frame floor across `20260629_312`, `20260629_309`, `20260630_316` and
+`20250926_235`. Not flagged in any column. Producer's own caveat.
+"""
+
+[senktide]
+# First-treatment subset of `steps_excluded`. A view, not a re-export.
+name = "2026-09-03_revised_2v_long_STEPS_EXCLUDED_SENKTIDE"
+recordings = 29
+note = """
+Every recording in `steps_excluded` whose first non-baseline region is `senktide`, carried
+WHOLE — every region, every event, full recording extent — with the sidecars subset to
+match. Event data byte-identical to the parent's; nothing re-derived, re-windowed or
+recounted. Groups: ORX 10, OVX 8, DI 6, MALE 5.
+
+The parent remains the record. Where the two disagree about anything but which recordings
+are present, the parent wins.
+"""
+
+[ttx]
+# First-treatment subset of `steps_excluded`. A view, not a re-export.
+name = "2026-09-03_revised_2v_long_STEPS_EXCLUDED_TTX"
+recordings = 38
+note = """
+Same construction as `senktide`, on a first treatment of `TTX`. Groups: DI 11, MALE 9,
+ORX 9, OVX 9.
+
+⚠ TTX IS NOT A SILENCING CONTROL HERE — FOUNDATIONS §9. Coordination persists under it and
+splits by stream (FAST median 0.46 of baseline, SLOW median 2.50 with 44% of slices at or
+above). A detector returning little in a TTX window is not thereby validated, and a nonzero
+coactivity excess on these slices is evidence about the preparation, not a false-alarm floor
+to tune away.
 """
 
 [pensub]

docs/INDEX.md

@@ -32,6 +32,8 @@ enforced by `tests/test_index_resolves.py`, which reads code spans too.
 | **the centrepiece — a person's verdicts on the machine's candidates** | human in the loop, machine assisted, identification, confirm, reject, annotate, verdict, judgement, who decided, agreement | [`src/bugarach/annotate.py`](../src/bugarach/annotate.py) — records the verdict **and the view it was made in**. `assess` proposes, this disposes; browser UI in `docs/site/raster_viewer.html` |
 | **MAHICE** | mahice, machine assisted, human identification, the centrepiece, the review, who decided | **Machine-Assisted Human Identification of Coordinated Events** — Tony's name for stage 2, and the step K is set in. **Defined in [`GLOSSARY.md`](GLOSSARY.md)** — with the retired spelling, and why its D was wrong. `annotate.MAHICE`; the record is **mahice.json** (`annotate.MahiceSession`, name in `annotate.SESSION_FILE`), which carries the K, who set it, when, and the ROI population it resolved against |
 | **K — who sets it, and in what units** | K, min_rois, floor, how many ROIs, coactivity floor, percentage, blocked, who chose K | **K is set by the user during MAHICE, and it is a PERCENTAGE of each recording's ROI population** (Tony, 2026-09-03). `bugarach assess --k-percent` shows the consequence; `assess.k_from_fraction` resolves it per recording *using the generator's own rule*; `derive_spec --session <the review record>` carries it into the spec. ⚠ An absolute K is not comparable across recordings — 3 of 10 ROIs is a third of the field and 3 of 51 is six percent, and both are in this corpus |
+| **a FLOOR under the percentage** | k floor, minimum ROIs, max(3, 10%), clamp, K=1, two rules | `--k-floor` on `bugarach assess`, `tools/assess_archive.py` and `tools/derive_spec.py`. `k_from_fraction` clamps only at 1 and leaves the clamp to the caller — this is it. ⚠ **Read `min_rois_floor_bound` beside `min_rois`**: where the floor binds, K came from the floor and not the percentage, so a pooled number came from **two rules**. On the senktide/TTX cohort at 10% it binds on 26 of 84 |
+| deriving a spec at a percentage nobody signed off on | unreviewed K, no MAHICE, skip the human, percentage_unreviewed | `derive_spec --k-percent --k-floor --unreviewed`. Aggregates ACROSS recordings at each one's own resolved count instead of selecting one column of the scan — selecting a column drops every recording whose K landed elsewhere. `k_source` reads **`percentage_unreviewed`**, never `mahice`; a consumer must be able to tell those apart |
 | K, cross-checked against the labels | derive_k, does my K agree, separation, Youden, second opinion | `annotate.cross_check_k` — **reports, never overrides.** `annotate.derive_k` is the arithmetic's opinion and is **not** where K comes from. ⚠ Candidates must be proposed **below** the K being set, or the labels are censored at the boundary they are meant to check — [the trap](todo/2026-08-28-derive-k-from-confirmed-events.md), and `assess --for-annotation` |
 | the question the loop opens with and never answers | contrast, before and after, baseline vs drug, paired, does it change, publish | ⚠ **nothing computes it.** No function in `src/` puts two regions side by side — [`the question nothing computes`](todo/2026-08-23-the-treatment-contrast-is-the-question-nothing-computes.md) |
 
@@ -44,6 +46,9 @@ enforced by `tests/test_index_resolves.py`, which reads code spans too.
 | what a folder is allowed to contain | contract, columns, producer, optional fields, silent ROI | [`export_folder_spec.md`](export_folder_spec.md) |
 | why analysis must not read the `.mat` store | store is closed, exclusions, withdrawn recordings, dead ROI | `CLAUDE.md` "The export folder is the input"; sapper **SAP007** |
 | the tour of where things live on disk | data root, darkroom, mount | [`where_the_data_are.md`](where_the_data_are.md) |
+| **field-step artifacts, and the folder with them removed** | artifact, brightness step, whole-field, step, excluded, artifact killer, on_field_step, field_step_id, contamination, 381 events | [`current_export.toml`](../current_export.toml) role **`steps_excluded`** (2026-09-03), and the folder's own `README.md` + `field_steps_excluded.tsv`. ⚠ Steps GENERATE detections — 74–82% of ROIs firing at once in both streams — so this is not cosmetic. ⚠ The loader **drops** `on_field_step` / `field_step_id` silently; harmless here (all zero), not harmless for the FLAGGED companion |
+| the senktide-only or TTX-only cohort | senktide, ttx, first treatment, subset, split, per-treatment folder | `current_export.toml` roles **`senktide`** (29) and **`ttx`** (38) — the producer's own split of `steps_excluded` on each recording's first non-baseline region, carried whole. ⚠ TTX is **not** a silencing control (FOUNDATIONS §9) |
+| ⚠ **the newest folder is not `default`** | which folder, current, newest, why is default old, should I switch | `current_export.toml`. `default` stays at `2026-08-18_revised_2v_periods` on purpose: moving it re-points every existing analysis and every parity fixture. The 2026-09-03 export is declared beside it as `steps_excluded`. **Switching `default` is Tony's call, not a housekeeping edit** |
 
 ## Another lab's data — DANDI / Cossart
 
@@ -108,6 +113,8 @@ enforced by `tests/test_index_resolves.py`, which reads code spans too.
 | reviewing a document before it ships | murderboard, adversarial, roles, anti-slop | [`doc_review_process.md`](doc_review_process.md) |
 | **quoting someone's email, or anything a person said in private** | correspondence, personal communication, private mail, quote a letter, permission, is this public, de-identify, attribution leak | **Cite it; do not quote it.** `CLAUDE.md` §*Other people's words* — paraphrase plus `<name>, personal communication, <date>`. **This repo is public**, it has leaked a private letter twice, and asking afterwards is not asking. Mechanized both at the commit and in CI: [`tools/check_quotes.py`](../tools/check_quotes.py). Where the letters themselves should live is open: [`todo/2026-09-02-correspondence-has-nowhere-private-to-live.md`](todo/2026-09-02-correspondence-has-nowhere-private-to-live.md) |
 | **draw a figure — how this repo renders one** | figure, plot, chart, PNG, holoviews, bokeh, panel, playwright, screenshot | copy the shape of any `tools/make_*_figure.py`: `measure()` → `build()` (holoviews panels) → `header_html()` → save HTML, screenshot it with Playwright chromium. Destination **defaults to `bugarach.paths.darkroom()`**, `--also` takes the repo copy (SAP006 blocks a required `--out`). Conventions — no titles, identity in the y-label, nothing drawn on a raster — are in `CLAUDE.md` |
+| ⚠ **a figure wider than 1120 px loses its right-hand column** | clipped, cut off, missing panel, right edge, facet missing, viewport, screenshot width | `_write(..., viewport_width=N)` in [`tools/make_generator_figures.py`](../tools/make_generator_figures.py). The clip was a **literal 1120.0** until 2026-09-09, so widening the viewport widened the window and clipped back anyway — no error, and a PNG that looks finished. Same shape as the height bug two comments above it in that file |
+| before/after per detector, faceted by experimental group | fireflies plot, before after, paired, per group, facet, one page per detector, treatment contrast | [`tools/make_before_after_figure.py`](../tools/make_before_after_figure.py) `--per-detector --facet group`. Y shared across a stream's group facets and nowhere else. ⚠ It reports recordings the detections file never mentions — at zero they cannot be told from a detector that ran and found nothing |
 | an algorithm implemented from a spec alone | clean room, spec, adversarial validation | [`clean_room/`](clean_room/) |
 | a decision already made | ADR, ruling, settled | [`adr/`](adr/) |
 | open work | todo, backlog, next | `docs/todo/` — 166 files, `status:` in frontmatter. **A record, not a queue** |

docs/SESSIONS.md

@@ -10,6 +10,22 @@ cannot travel (live process ids, that box's free disk, local scratch paths).
 
 ---
 
+### Mac/full-cohort-senktide-ttx — DARKROOM claim: `bugarach/2026-09-09-full-cohort-senktide-ttx/`
+- **Status:** ACTIVE 2026-09-09. The full senktide and TTX cohorts through the loop —
+  assess, spec, simulate, calibrate, train, held-out comparison, detect, figures. Successor
+  to the APV+CNQX+GZ pilot, and the first run able to detect at its own calibration and to
+  run its own trained models on real data, both of which #507–#509 closed yesterday.
+- **Holds:** that one darkroom directory and nothing else. No deploy, no MATLAB, no site.
+  `<darkroom>/constellation/` untouched — that is the MATLAB producer team's.
+- **Input:** the producer's `2026-09-03_..._STEPS_EXCLUDED_{SENKTIDE,TTX}` folders, which
+  no file in this repo declared until this branch. ⚠ **`current_export.toml`'s `default`
+  still points at `2026-08-18_revised_2v_periods`** and is deliberately left there — moving
+  it would re-point every existing analysis and every parity fixture at a different corpus.
+  New roles were added beside it instead; changing `default` is a decision for Tony.
+- **MAHICE skipped on instruction**, so K is a percentage nobody signed off on: `k_source`
+  reads `percentage_unreviewed`, never `mahice`. RESET.md §1 applies to everything
+  downstream and the artifacts say so.
+
 ### Mac/deploy-the-figure-and-the-viewer — SITE claim RELEASED: `bugarach.tonydefazio.com`
 - **Status:** **RELEASED 2026-09-08. THE SITE IS HELD BY NOBODY.** Published `3cb89fd`,
   wrangler **Version ID `ab3cc81d-1ed5-415b-ada8-14deb77574ce`**, 7 of 8 assets changed.

src/bugarach/assess.py

@@ -226,6 +226,22 @@ class Assessment:
     a folder of mixed ROI counts the second one differs per recording while the
     first does not."""
 
+    min_rois_floor: int | None = None
+    """An absolute floor applied under :attr:`min_rois_frac`, or ``None``.
+
+    :func:`k_from_fraction` clamps only at 1, and its own docstring says the clamp
+    belongs to whoever is asking — this is that clamp, recorded rather than applied
+    silently. A percentage alone reaches K=1 on a small field, and one co-active ROI
+    is not coordination.
+
+    **Whether it BOUND is** :attr:`min_rois_floor_bound`, and the two are meant to be
+    read together. A floor that never binds is a setting; a floor that binds on a
+    third of the corpus is a second K rule governing the small recordings, and a
+    number quoted over that population came from two rules rather than one."""
+
+    min_rois_floor_bound: bool = False
+    """True when the floor raised K above what the fraction resolved to here."""
+
 
 def _coact_count(trains, win_dur, bin_width, n_bins, offsets=None):
     """Per-bin distinct-ROI coactivity. An ROI contributes 1 to a bin if it has
@@ -341,6 +357,7 @@ def assess_coactivity(
     region: str = "baseline",
     min_rois=None,
     min_rois_frac=None,
+    min_rois_floor: int | None = None,
     bin_width_sec: float | None = None,
     wm_factor: float = 1.5,
     merge_bins: int = 2,
@@ -406,20 +423,46 @@ def assess_coactivity(
             raise ValueError(
                 f"min_rois_frac must be fractions in (0, 1], got {fracs}. "
                 f"Percentages go in as 0.10 rather than 10.")
+        # The floor is applied HERE, after the fraction has met this recording's
+        # own ROI count, because that is the only place the two can be compared.
+        # It is recorded on every result — `min_rois_floor` and
+        # `min_rois_floor_bound` — so a reader can tell a K a percentage produced
+        # from a K the floor produced. On a corpus of mixed field sizes those are
+        # different populations, and averaging across them without saying so
+        # reports one rule where two were running.
+        floor = None if min_rois_floor is None else int(min_rois_floor)
+        if floor is not None and floor < 1:
+            raise ValueError(
+                f"min_rois_floor must be at least 1, got {floor}. A floor below "
+                f"one co-active ROI is not a floor.")
         resolved = [(k_from_fraction(f, st.n_rois), f) for f in fracs]
+        bound_at = {}
+        if floor is not None:
+            bound_at = {max(floor, kk): kk < floor for kk, _ in resolved}
+            resolved = [(max(floor, kk), f) for kk, f in resolved]
         # Two fractions can land on one count in a small field — 5% and 10% of 12
         # ROIs are both 1. Keeping both rows would report the same measurement
         # twice under different labels, so the coarser fraction wins its count
-        # and the duplicate is dropped rather than silently averaged.
+        # and the duplicate is dropped rather than silently averaged. A floor
+        # collapses fractions the same way and for the same reason: below it they
+        # all name one K.
         seen: dict[int, float] = {}
         for kk, f in resolved:
             seen.setdefault(kk, f)
         ks = tuple(sorted(seen))
         frac_of: dict[int, float | None] = {kk: seen[kk] for kk in ks}
+        floor_of: dict[int, bool] = {kk: bound_at.get(kk, False) for kk in ks}
     else:
+        if min_rois_floor is not None:
+            raise ValueError(
+                "min_rois_floor applies to a K given as a fraction; with an "
+                "absolute min_rois the floor is either already in the number or "
+                "is a second opinion about it. Pass one.")
+        floor = None
         ks = tuple(int(K) for K in (min_rois if min_rois is not None
                                     else DEFAULT_MIN_ROIS))
         frac_of = {kk: None for kk in ks}
+        floor_of = {kk: False for kk in ks}
     min_rois = ks
 
     if window is None:
@@ -436,7 +479,7 @@ def assess_coactivity(
         if picked is None:
             return [Assessment(min_rois=int(K), meets_floor=False,
                                win_dur=float("nan"), n_roi=st.n_rois,
-                               n_events_win=0, min_rois_frac=frac_of[K])
+                               n_events_win=0, min_rois_frac=frac_of[K], min_rois_floor=floor, min_rois_floor_bound=floor_of[K])
                     for K in min_rois]
         win_start, win_end, win_dur = picked.win_start, picked.win_end, picked.win_dur
         meets = picked.meets_floor
@@ -448,7 +491,7 @@ def assess_coactivity(
     if not meets:
         return [Assessment(min_rois=int(K), meets_floor=False, win_dur=win_dur,
                            n_roi=st.n_rois, n_events_win=0,
-                           min_rois_frac=frac_of[K]) for K in min_rois]
+                           min_rois_frac=frac_of[K], min_rois_floor=floor, min_rois_floor_bound=floor_of[K]) for K in min_rois]
 
     bin_width = 1.0 if bin_width_sec is None else float(bin_width_sec)
     wm = wm_factor * bin_width
@@ -531,7 +574,7 @@ def assess_coactivity(
         jit_obs, jit_null = _med(sd_obs), _med(sds_null[K])
         defined = bool(sd_obs) and bool(sds_null[K])
         out.append(Assessment(
-            min_rois=K, min_rois_frac=frac_of[K],
+            min_rois=K, min_rois_frac=frac_of[K], min_rois_floor=floor, min_rois_floor_bound=floor_of[K],
             meets_floor=True, win_dur=win_dur, n_roi=n_roi,
             n_events_win=int(sum(n_in_win)),
             roi_rate=roi_rate, roi_rate_med=_med(roi_rate),

src/bugarach/assess_folder.py

@@ -85,7 +85,8 @@ def skipped(self) -> list[RecordingAssessment]:
 def assess_folder(folder, *, stream: str | None = None,
                   n_surrogates: int = 1000, bin_width_sec: float | None = None,
                   limit: int | None = None, progress=None,
-                  min_rois=None, min_rois_frac=None) -> FolderAssessment:
+                  min_rois=None, min_rois_frac=None,
+                  min_rois_floor: int | None = None) -> FolderAssessment:
     """Assess every recording in an export folder that may be assessed.
 
     Reads the folder with the same loader the rest of bugarach uses, so a folder
@@ -172,6 +173,8 @@ def assess_folder(folder, *, stream: str | None = None,
                 **({} if min_rois is None else {"min_rois": tuple(min_rois)}),
                 **({} if min_rois_frac is None
                    else {"min_rois_frac": tuple(min_rois_frac)}),
+                **({} if min_rois_floor is None
+                   else {"min_rois_floor": int(min_rois_floor)}),
                 **({} if bin_width_sec is None else {"bin_width_sec": bin_width_sec}))
         except Exception as e:                        # noqa: BLE001
             rec.skipped = f"{type(e).__name__}: {e}"