Skip to content

feat(data): first-class Data mode for project inventory and reclassification - #1166

Merged
wesm merged 4 commits into
mainfrom
cool-lillipilli
Jul 31, 2026
Merged

feat(data): first-class Data mode for project inventory and reclassification#1166
wesm merged 4 commits into
mainfrom
cool-lillipilli

Conversation

@wesm

@wesm wesm commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

Adds a top-level Data mode for reviewing project identity across an archive and managing folder-to-project mappings. It replaces the mapping editor in Settings with an inventory, a project workspace, and centralized rules that can be previewed and applied atomically.

Behavior and correctness

  • Keys inventory rows by opaque project identity, so sanitized-label collisions remain distinct and every deep link and reclassification targets the exact project.
  • Presents observed folders in a compact workspace suited to projects with many source directories. Unknown identities appear as Unclassified, and filesystem roots are not suggested as automatic mappings.
  • Binds reclassification previews to the normalized draft, collision state, mapping revision, and exact affected-session snapshot. Changed state returns a conflict instead of applying a stale preview.
  • Reconciles identity evidence when sessions move between projects while preserving trustworthy historical Git evidence for missing sources and orphaned sessions.
  • Emits project/session changes after mutations and refreshes Data, Sessions, and Activity metadata across clients. Background rule refreshes update the list without discarding unsaved edits.
  • Keeps mapping mutations local to writable SQLite archives. PostgreSQL and DuckDB expose the same inventory and governance data as read-only views.

Publication and storage

Filtered PostgreSQL and DuckDB publication keeps mappings, snapshots, provenance, and tombstones within each configured project scope. Incremental watch work is bounded by changed sessions, project moves publish in both directions, placeholder snapshots are excluded, and mirrored sessions retain their source machine, file path, and exact database generation.

The SQLite data version advances from 76 to 77 to reparse project-identity source snapshots while preserving live and orphaned archive data through non-destructive migration. PostgreSQL gains source-scoped publication metadata and session provenance fields. DuckDB schema v8 adds source archive and mapping provenance; existing disposable mirrors rebuild rather than migrate in place.

@roborev-ci

roborev-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown

roborev: Combined Review (ba1415b)

Medium-severity issues remain in reclassification and incremental synchronization.

Medium

  • internal/db/project_identity.go:888 — Reclassification deletes all identity observations for affected projects and rebuilds them only from snapshots. While background backfill is incomplete, older sessions may have aggregate observations but no snapshot, permanently discarding valid identity evidence. Preserve or re-key compatible observations for snapshotless sessions, or synchronously backfill affected sessions before rebuilding. Add coverage for reclassification during incomplete snapshot migration.

  • internal/sync/engine.go:1827, internal/db/orphaned.go:456 — Full resync preserves mirror watermarks but not local_modified_at, while mapping replay avoids restamping it. Reclassification after a push but before resync can therefore cause the next incremental PostgreSQL or DuckDB push to skip the session, leaving the mirror under its former project indefinitely. Copy local_modified_at for live and orphaned sessions during resync, or restamp pending mapped moves after rebuilding. Test reclassification followed by resync and an incremental mirror push.

  • internal/postgres/push.go:266 — Each filtered PostgreSQL push loads every local session in scope, while reconcilePGProjectScopeMoves scans every owned remote session. Incremental synchronization therefore scales with total archive size instead of the changed batch. Track project-scope moves or prior membership by changed session ID and reconcile them incrementally in bounded batches. Add a small-versus-large archive cardinality regression test.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 21m46s

@roborev-ci

roborev-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown

roborev: Combined Review (4e04526)

Review found two medium-severity correctness issues affecting project reclassification and UI freshness.

Medium

  • internal/db/orphaned.go:648 — During the version-68 resync, metadata copying overwrites the freshly parsed snapshot’s corrected project with the old snapshot label. This leaves stale source labels in incremental state and PostgreSQL/DuckDB identity publications. Preserve the destination snapshot’s project when copying historical evidence, and add an upgrade test covering an old label and a freshly reparsed label.

  • internal/sync/engine.go:2294 — Successful reclassification rewrites sessions without emitting a sessions data-change event. The Activity report refreshes explicitly, but the sidebar and other connected clients can show stale project groups until the five-minute safety refresh. Emit a sessions event after a successful mutation with updated sessions, outside the exclusive lock, and add emitter coverage.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 19m58s

@roborev-ci

roborev-ci Bot commented Jul 18, 2026

Copy link
Copy Markdown

roborev: Combined Review (5153c96)

Reclassification risks permanent loss of aggregate-only identity evidence.

Medium

  • internal/db/project_identity.go:888, internal/db/worktree_reclassification.go:149 — Reclassification deletes all identity aggregates for each affected project, then rebuilds them solely from session snapshots. Because aggregate-only evidence is supported when the parser project is empty, even a no-op mapping can discard that evidence, including observations unrelated to the mapped path.

    Fix: Reconcile only identity keys for sessions that actually moved, or preserve aggregate observations without reconstructable snapshots. Add coverage for mapped sessions with an empty parser project and unrelated aggregate-only evidence.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 12m37s

@wesm
wesm force-pushed the cool-lillipilli branch from 5153c96 to e5d5c78 Compare July 18, 2026 13:56
@roborev-ci

roborev-ci Bot commented Jul 18, 2026

Copy link
Copy Markdown

roborev: Combined Review (e5d5c78)

Medium-severity issue found: aggregate rebuilding may discard valid project identity evidence.

Medium

  • internal/db/project_identity.go:888 — Aggregate rebuilding deletes all affected observations and restores evidence only from session snapshots. Sessions with missing snapshots, including preserved orphans, or with placeholder snapshots can lose or downgrade compatible aggregate evidence used during candidate discovery.
    • Suggested fix: Preserve compatible existing observations before deletion, then merge them with trustworthy snapshot evidence during rebuilding. Add coverage for reclassifying sessions with aggregate-only and placeholder-snapshot identities.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 20m54s

@wesm

wesm commented Jul 19, 2026

Copy link
Copy Markdown
Member Author

I'm still working on this

@wesm
wesm force-pushed the cool-lillipilli branch from e5d5c78 to 7b8e83c Compare July 19, 2026 23:35
@roborev-ci

roborev-ci Bot commented Jul 19, 2026

Copy link
Copy Markdown

roborev: Combined Review (7b8e83c)

The changes have two medium-severity backend compatibility and data-integrity issues; no exploitable security vulnerabilities were identified.

Medium

  • PostgreSQL compatibility checks omit newly required schemainternal/postgres/schema.go:2081
    CheckSchemaCompat does not probe sessions.source_archive_id, sessions.file_path, or source_worktree_project_mappings. As a result, pg serve can accept an older schema at startup, then return SQL errors from the new project inventory, rules, and candidate endpoints. Add every column and table required by these read paths to CheckSchemaCompat, not only to the push-specific compatibility check.

  • DuckDB push loses the session’s originating machineinternal/duckdb/push.go:910
    Session rows use the DuckDB push machine while worktree mappings retain their original machine. For sessions imported from another machine, governance matching on (source_archive_id, machine) fails, producing incorrect governed counts, rules, and candidates. Preserve each session’s real non-sentinel machine, using the push machine only as a fallback, and test mappings and sessions with a genuine remote-machine value.


Reviewers: 2 done | Synthesis: codex, 12s | Total: 9m3s

@wesm wesm changed the title feat(activity): add project reclassification feat(data): first-class Data mode for project inventory and reclassification Jul 20, 2026
@wesm
wesm force-pushed the cool-lillipilli branch from 7b8e83c to 55502b6 Compare July 20, 2026 12:19
@roborev-ci

roborev-ci Bot commented Jul 20, 2026

Copy link
Copy Markdown

roborev: Combined Review (10b3d07)

Code review found two medium-severity issues involving project identity collisions and unbounded incremental push work.

Medium

  • internal/db/project_inventory.go:167 — Inventory rows are merged solely by sanitized display label while retaining only the first project key. Distinct projects sharing a display label can disappear, and aggregate rows or deep links may reclassify only the first project. Group by project key, disambiguate colliding labels, and preserve backend parity across SQLite, PostgreSQL, and DuckDB.

  • internal/postgres/push.go:284 — Every filtered incremental push materializes all local sessions in scope and scans every PostgreSQL session owned by the archive, even when nothing changed. This makes latency and memory scale with total archive size. Use a bounded change journal or previously published membership state to process sessions changed since the last cursor, reserving full reconciliation for repair or full pushes.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 28m46s

@wesm
wesm force-pushed the cool-lillipilli branch from 10b3d07 to ad65ba0 Compare July 23, 2026 20:32
@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (ad65ba0)

Changes requested: two medium-severity data integrity issues need resolution.

Medium

  • internal/db/orphaned.go:1173 — The copy guard accepts archives from version 68 onward, but version 70 archives may contain stale mapping-target project snapshots. During a 70→71 full resync, these can overwrite freshly parsed source-project snapshots and preserve the provenance corruption the migration is intended to fix. Raise the compatibility threshold to version 71 and add an upgrade regression test using a version 70 archive.

  • internal/db/project_inventory.go:173 — Projects sharing the same sanitized display label are merged into one inventory row, while only the first project key is retained. Candidate lookup requires an exact label/key match, so sessions associated with other merged keys are counted but cannot be viewed or reclassified. Keep separate rows per project key, or retain all merged keys and apply consistent grouping during candidate selection.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 10m38s

@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (2492f93)

PostgreSQL provenance backfill is incomplete for filtered pushes, causing persistent archive-scoping errors after upgrades.

Medium

  • internal/postgres/push.go:1349 — Filtered PostgreSQL pushes skip the provenance backfill. After upgrading an existing filtered mirror, unchanged sessions indefinitely retain default values (source_archive_id='', file_path=NULL), breaking archive-scoped snapshot joins and Data-mode provenance unless an unfiltered/full push occurs.
    • Fix: Force a one-time full push per filter-scoped sync state without completing the target-wide marker, and add an upgrade test covering existing filtered watermarks and fingerprints.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 36m36s

@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (1f8f34e)

The PR is generally sound, but one medium-severity snapshot compatibility issue should be fixed before merging.

Medium

  • internal/db/orphaned.go:1173projectIdentitySourceSnapshotDataVersion aliases the mutable current dataVersion. After the next version bump, valid snapshots created by version 74 will be treated as legacy and discarded for orphaned sessions whose source files no longer exist.
    • Fix: Set the snapshot compatibility threshold to the fixed introduction version (74) and add a regression test covering migration from version 74 to a later version.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 23m23s

@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (1f8f34e)

Medium-severity issue found: snapshot trust is incorrectly coupled to the mutable schema version.

Medium

  • internal/db/orphaned.go:1173 — The snapshot trust threshold uses the current mutable dataVersion. A future schema-version bump would cause valid version-74 parser-source snapshots to be treated as legacy and discarded during full resync, potentially losing identity evidence for orphaned or missing-source sessions. Set projectIdentitySourceSnapshotDataVersion to the fixed introduction version, 74, independently of future dataVersion changes.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 7m51s

@wesm
wesm force-pushed the cool-lillipilli branch from 1f8f34e to 97806c6 Compare July 28, 2026 02:21
@roborev-ci

roborev-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown

roborev: Combined Review (97806c6)

The change has one medium-severity correctness issue; no material security regressions were identified.

Medium

  • Project identity is lost when sanitized labels collideinternal/db/project_inventory.go:167, internal/postgres/project_inventory.go:123, internal/duckdb/project_inventory.go:143

    Inventory rows are merged by sanitized display label while retaining only the first raw project’s project_key. Distinct path- or URL-like projects can sanitize to the same label but have different keys, causing merged counts to disagree with the emitted key. Reclassification then selects only sessions matching the first key, and Activity links for the other keys appear unknown.

    Suggested fix: Aggregate by (display label, project_key) or raw project identity, and annotate using that same identity. Add collision and deep-link tests for all three backends.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 18m47s

@roborev-ci

roborev-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown

roborev: Combined Review (d40a928)

Changes need revision: three medium-severity correctness and scalability issues remain.

Medium

  • internal/db/orphaned.go:1175 — Snapshots from data versions 68–74 are incorrectly considered parser-source-safe, even though version 75 corrected snapshots containing mapped target labels. During resync, stale snapshots can overwrite freshly parsed data and preserve the wrong source project. Raise the cutoff to version 75 or preserve the freshly parsed destination snapshot, with upgrade coverage for versions 68–74.

  • internal/db/project_identity.go:912 — Aggregate rebuilding creates an unbounded SQL placeholder for every affected project. Large reclassifications can exceed SQLite’s bind-variable limit and roll back the entire operation. Process projects in bounded chunks or use a temporary table, and add a regression test above the supported variable limit.

  • internal/db/project_inventory.go:173 (also internal/postgres/project_inventory.go:129 and internal/duckdb/project_inventory.go:149) — Projects sharing the same sanitized label are merged into one inventory row, but only the first project key is retained while counts from every colliding project are summed. This makes some projects unselectable in Data mode and causes displayed totals to disagree with the retained key. Preserve a selectable row per project key with safe disambiguation, or retain all colliding keys and align aggregate counts with selection behavior.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 14m54s

@wesm
wesm force-pushed the cool-lillipilli branch from d40a928 to ad13a88 Compare July 29, 2026 23:09
@roborev-ci

roborev-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown

roborev: Combined Review (ad13a88)

Code review found three medium-severity issues involving project identity, sync consistency, and filtered metadata exposure.

Medium

  • Inventory merges distinct projectsinternal/db/project_inventory.go:173
    Rows are grouped by sanitized display label instead of opaque project key. Distinct projects with identical or empty sanitized labels are merged, producing inaccurate counts and losing project keys needed for Activity deep links and reclassification. Group by project key across SQLite, PostgreSQL, and DuckDB, then disambiguate duplicate display labels.

  • Incremental sync recreates stale identity aggregatesinternal/sync/engine.go:11612
    After applying a worktree mapping, sync records the identity observation using the pre-mapping inc.project. This can recreate aggregates under the old project immediately after reconciliation removes them. Refetch or return the session’s final mapped project before recording the observation; retain inc.sourceProject only for the immutable source snapshot.

  • Filtered PostgreSQL pushes leak excluded-project metadatainternal/postgres/worktree_mappings_push.go:48, internal/postgres/project_rules.go:42
    Full and incremental filtered pushes publish unfiltered worktree mappings, exposing machine names, path prefixes, and project names for excluded projects through /api/v1/data/project-rules. Scope mapping publication to s.projects and s.excludeProjects, remove rows that leave scope, and derive dynamic rules from in-scope sessions—or skip mapping publication when it cannot be represented safely.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 16m30s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (58c2ba7)

Verdict: Four medium-severity issues need attention before merge.

Medium

  • Project inventory merges distinct projects with empty sanitized labels
    internal/db/project_inventory.go:167
    Rows are merged by sanitized display label instead of project key. Distinct absolute-path projects that sanitize to the same empty label have their counts combined while only the first key survives, breaking deep links and causing reclassification to cover only part of the displayed aggregate. Keep rows distinct by project key across SQLite, PostgreSQL, and DuckDB; use the sanitized label only for display.

  • Filtered PostgreSQL pushes can leave provenance permanently incomplete
    internal/postgres/push.go:1404
    Filtered pushes skip provenance backfill, so fingerprint-matched rows—especially orphaned sessions preserved through the v76 resync—may retain an empty source_archive_id indefinitely. Data-mode governance then excludes them, resulting in incomplete counts and rule attribution. Track backfill completion per effective filter scope and force one full push for each scope without marking the unfiltered target-wide backfill complete.

  • Project-filtered pushes leak out-of-scope mapping metadata
    internal/postgres/worktree_mappings_push.go:48
    internal/duckdb/worktree_mappings_push.go:40
    Filtered pushes still publish every mapping, exposing excluded project names, machine names, and local path prefixes through destination readers and the read-only Data API. Apply project scope to mapping publication. Omit rules that cannot safely be attributed to in-scope projects without advancing their publication cursor, or publish only rules governing in-scope sessions with out-of-scope labels redacted.

  • Data-mode E2E test targets obsolete UI elements
    frontend/e2e/data-mode.spec.ts:64
    The writable flow expects the removed .candidate-summary element and the old “Apply reclassification” button at line 89, so Chromium CI will time out. The read-only assertion at line 161 also uses the obsolete accessible name and no longer verifies that mutation controls are hidden. Update the test to inspect the observed-folder UI and target “Save and apply mapping” in both positive and negative assertions.


Reviewers: 2 done | Synthesis: codex, 15s | Total: 18m29s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (2571146)

Potential stale snapshot regression during full resync requires a fix.

Medium

  • internal/db/orphaned.go:1282 — The snapshot-copy threshold remains at version 68, although version 76 is intended to repair snapshots containing mapped target labels. During a full resync from versions 68–75, CopySessionMetadataFrom can overwrite newly parsed snapshots with stale data, potentially restoring the wrong project identity. Raise the threshold to 76 (or otherwise preserve newly parsed snapshots when importing older archives) and add a regression test for upgrading a version-75 archive with a stale mapped snapshot.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 10m2s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (9d32b50)

Three medium-severity issues require changes before approval.

Medium

  • internal/postgres/push.go:322 — Filtered pushes scale with total archive size. Under pg watch, every filtered push materializes all sessions currently in scope, causing per-event work and memory to grow with the archive rather than the changed batch. Track project transitions or tombstones and reconcile only changed session IDs. Reserve full scans for one-time backfills and add a small-versus-large archive regression test.

  • internal/db/worktree_reclassification.go:123 — Preview tokens are insufficiently bound to accepted state. The token hashes only the stored mapping set, not the proposed draft, collision state, or affected-session snapshot. A token obtained for one draft can authorize another, and newly added sessions can be rewritten without appearing in the preview. Bind the token to the normalized draft, collision state, and a stable session-impact revision; validate it inside the apply transaction and return a conflict if anything changed.

  • internal/duckdb/project_inventory.go:342 — Multi-machine governance results can be incorrect. Governance joins sessions and mappings by archive and machine, but internal/duckdb/push.go:911 stamps mirrored sessions with the syncing machine while mappings retain their original machine. Preserve each session’s machine during publication, using the syncing machine only as a fallback for empty or local identities, and add a push-backed multi-machine test.


Reviewers: 2 done | Synthesis: codex, 12s | Total: 15m24s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (6175d73)

Filtered publication has three medium-severity correctness and confidentiality issues.

Medium

  • Filtered publication scopes snapshots by stale project identityinternal/postgres/push.go:745, internal/duckdb/push.go:290, internal/db/project_identity.go:177
    Immutable snapshots are scoped using their original project instead of the owning session’s current project. After reclassification, a scoped push publishes the moved session but omits its identity snapshot, causing backend candidate results to diverge. Scope snapshots by joining them to the current session project, and publish snapshot upserts/deletes whenever a session crosses a filtered scope.

  • Placeholder snapshots become authoritative after publicationinternal/postgres/push.go:827, internal/duckdb/push.go:387
    Push sanitization derives a non-empty identity key from placeholder snapshots whose root_path is merely the session working directory. SQLite ignores these uninspected placeholders, while PostgreSQL and DuckDB treat them as authoritative evidence, changing candidate grouping and precedence. Exclude placeholders from publication or preserve their empty key source, and test mirror behavior with ordinary imported/upserted sessions.

  • Filtered PostgreSQL pushes disclose excluded-project metadatainternal/postgres/push.go:754
    Full filtered publication loads every identity tombstone without applying project filters and sends tombstone metadata—including project, machine, absolute root path, repository URL, and session ID—to PostgreSQL, even for excluded projects that were never published. Track identity keys previously published to each target/filter scope and transmit only matching tombstones, or derive stale keys from rows already present at the target.


Reviewers: 2 done | Synthesis: codex, 15s | Total: 23m9s

@wesm
wesm force-pushed the cool-lillipilli branch from 93ffdef to aac482e Compare July 30, 2026 15:27
@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (aac482e)

Code changes need fixes for upgrade correctness and PostgreSQL backend parity.

Medium

  • internal/db/orphaned.go:1839 — The trust watermark remains at version 68, so upgrading a version 68–75 archive can copy target-labelled legacy identity snapshots over freshly parsed source snapshots. This defeats the version 76 reparse and preserves unreliable orphan snapshots. Raise the watermark to version 76 and test upgrading from version 75 for both live and orphaned sessions.

  • internal/postgres/schema.go:97 — PostgreSQL stores sessions.file_path, but pgSessionCols and scanPGSession omit it. Sessions served through PostgreSQL therefore return no file path, breaking backend parity and source-path UI functionality. Add file_path to pgSessionCols, scan it into db.Session.FilePath, and cover list/detail reads with PostgreSQL tests.


Reviewers: 2 done | Synthesis: codex, 14s | Total: 19m3s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (84ba0ca)

Code review found three medium-severity correctness issues; no critical or high-severity findings.

Medium

  • Filtered publications can erase metadata from other filter scopes
    Location: internal/postgres/push.go:808, internal/postgres/worktree_mappings_push.go:127
    A filtered publication deletes all identity and mapping metadata for the source archive. Since filters have independent watermarks while sessions from other filters remain in PostgreSQL, alternating --projects scopes can erase each other’s metadata. An unchanged scope may then skip republishing because its cursor is already current.
    Suggested fix: Reconcile metadata per filter scope without deleting rows owned by other scopes, and add an integration test alternating two filtered pushes.

  • Full resync can reset project classification for trashed sessions
    Location: internal/db/project_identity.go:1006, internal/db/project_identity.go:1050
    RestoreSessionProjectsFromIdentitySnapshots includes trashed sessions, resetting them to their parser-source project. Subsequent mapping application excludes deleted rows, so restoring a session can expose the wrong project classification.
    Suggested fix: Restrict both the affected-row query and update to deleted_at IS NULL, with a regression test covering a mapped session that is trashed, resynced, and restored.

  • Worktree candidate selection excludes aliases sharing an identity key
    Location: internal/db/worktree_candidates.go:78
    Candidate selection requires both the opaque project key and the inventory row’s chosen display label. When multiple raw labels resolve to the same identity key, aliases with different display labels are excluded, producing incomplete candidate counts and folder suggestions. PostgreSQL and DuckDB share this behavior.
    Suggested fix: Select all raw labels belonging to the requested project key, retaining the display label only for collision validation, and add alias-identity tests across all backends.


Reviewers: 2 done | Synthesis: codex, 13s | Total: 23m51s

@wesm
wesm force-pushed the cool-lillipilli branch from 26828dd to 16a7e10 Compare July 30, 2026 18:04
@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (16a7e10)

The PR has one medium-severity data consistency issue; no security vulnerabilities were identified.

Medium

  • Stale project identity aggregates during session reclassificationinternal/db/project_identity.go:811, internal/db/session_batch.go:320

    Session upserts overwrite sessions.project and publish identity evidence under the new project without reconciling the previous project’s aggregate. Sync-driven project changes, such as disabling or deleting a mapping, can leave stale observations that keep the old label incorrectly resolved or ambiguous in Data inventories.

    Suggested fix: Capture the previous project before upserting. When it changes, reconcile both the old and new project aggregates in the standalone and batch paths. Add regression coverage for resync after disabling or deleting a mapping.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 21m19s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (b7097db)

The change has two medium-severity correctness and scalability issues that should be addressed before merging.

Medium

  • Historical identity evidence may be lost during rebuildinternal/db/project_identity.go:942

    Rebuilding an affected project deletes all existing identity observations and reconstructs them solely from snapshots. Pre-v76 source-missing sessions may retain historical aggregate evidence without a trustworthy snapshot or checkout to reprobe, so reclassification can permanently discard repository-remote evidence and downgrade identity resolution.

    Suggested fix: Preserve aggregate observations not represented by trustworthy snapshots, or reconcile only evidence keys changed by reclassified sessions. Add coverage for a pre-v76 source-missing session with historical remote evidence.

  • Large reclassifications can exceed SQLite’s bind-variable limitinternal/db/project_identity.go:934

    The rebuild query creates one SQL parameter per affected project. A broad mapping involving approximately 32,766 distinct project labels can exceed SQLite’s variable limit and fail the entire reclassification transaction.

    Suggested fix: Stage affected projects in a temporary table or process them in batches below the bind-variable limit. Add a cardinality regression test.


Reviewers: 2 done | Synthesis: codex, 14s | Total: 20m32s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (80f3d73)

Project reclassification has a cache invalidation gap that can leave stale project filters across views and clients.

Medium

  • internal/sync/engine.go:3139; frontend/src/lib/components/data/DataPage.svelte:94 — Reclassification and Apply mappings can change sessions.project, but they emit no session-change event and only reload the Data inventory. As a result, cached project filters in Sessions and Activity can retain removed or renamed labels, and other open clients are not notified. Emit a project/session change after successful mutations, then invalidate and refetch the Sessions and Activity project metadata caches.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 36m44s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (6de1163)

The changes are generally sound, but single-session deletion can corrupt the canonical root-group count.

Medium

  • Incorrect total after deleting a sessionfrontend/src/lib/stores/sessions.svelte.ts:1288
    total now counts canonical root groups, but single-session deletion decrements it for every removed row. Deleting a child incorrectly reduces the count, while deleting a root may remove multiple rows. The stale count can persist if the SSE refresh races or is missed. Reload sidebar data after single deletions, as batch deletion does, or update the count using root-group semantics. Add tests covering root and child deletion.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 27m45s

@roborev-ci

roborev-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (030d461)

Changes need fixes: two medium-severity correctness and freshness issues remain.

Medium

  • internal/postgres/worktree_candidates.go:131 — Snapshot selection orders database-generation UUIDs as though they were chronological. Filtered publications may retain snapshots from multiple generations, so tied or preserved observed_at values can select stale project identity and produce incorrect worktree candidates. Track the source database generation on mirrored sessions and join the exact snapshot, or atomically supersede older-generation snapshots. Add a multi-generation filtered-publication test.

  • frontend/src/lib/stores/data.svelte.ts:44 — The Data store reloads only on mount, navigation, or local mutations. It does not subscribe to session or sync events, so inventory totals, projects, machines, and governed-rule counts can remain stale after background syncs or imports. Subscribe to relevant debounced data-change or sync-completion events while attached, refresh inventory in the background, and reload the rules view when mounted.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 20m39s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (cbe7a02)

Changes are generally sound, but one medium-severity UI state-loss issue should be addressed.

Medium

  • frontend/src/lib/components/data/DataPage.svelte:87 — Every session or sync event increments rulesRefreshVersion, forcing a keyed remount of the rules editor and silently discarding unsaved rule edits. Refresh the rule list in place while preserving draft state, or defer background refreshes while the editor is dirty.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 22m55s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (338c813)

Medium-severity issue found in multi-project reclassification safety detection.

Medium

  • internal/db/worktree_reclassification.go:396evaluation.projects only includes sessions whose project changes. Sessions already assigned to the target are excluded, so a prefix matching the target project and another project can report only one affected project, suppressing the UI’s multi-project safety warning.
    • Fix: Count every matched session’s current project before the !shouldUpdate early return, while keeping updates and change samples conditional. Add a regression test covering one already-target session and one changing-project session.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 17m26s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (ddb2938)

The PR has one medium-severity data integrity issue; no security vulnerabilities were identified.

Medium

  • Unfiltered mappings can be deleted by a later filtered publicationinternal/postgres/worktree_mappings_push.go:133
    An unfiltered publication inserts mappings without scope ownership. A subsequent filtered full publication deletes every unowned mapping for the archive, including valid mappings outside the filter.

    Suggested fix: Track ownership for unfiltered publications, or limit filtered cleanup to rows previously owned exclusively by that filtered scope. Add a regression test covering an unfiltered publication followed by a filtered publication with both in-scope and out-of-scope mappings.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 23m41s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (f0a8970)

Verdict: One medium-severity data preservation issue found; no security vulnerabilities identified.

Medium

  • internal/postgres/project_identity_upsert.go:119 — Unfiltered identity publications create no ownership records. A subsequent filtered publication can therefore treat valid observations and snapshots for live, out-of-scope projects as unowned metadata and delete them.
    • Suggested fix: Assign unfiltered publications a stable ownership scope, consistent with worktree mappings, and add an unfiltered-then-filtered test verifying preservation of observations and snapshots.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 20m37s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (05bdadc)

Overall verdict: Two medium-severity correctness issues should be addressed before merge; no security vulnerabilities were identified.

Medium

  • Filtered pushes can erase shared mapping metadatainternal/postgres/worktree_mappings_push.go:193

    A filtered publication redacts out-of-scope original_project values, then overwrites the shared mapping row even when another publication scope still owns it. As a result, an unfiltered mapping can lose its original-project metadata after a filtered push.

    Recommended fix: Preserve the existing nonempty value when another scope owns the row, or store payloads per scope. Add a regression test covering an unfiltered publication followed by a filtered publication.

  • Archive identity changes do not invalidate session fingerprintsinternal/postgres/push.go:454

    source_archive_id is written to PostgreSQL sessions but omitted from sessionPushFingerprint. If an archive identity is regenerated or repaired, unchanged sessions may be skipped and remain associated with the old archive, disconnecting them from newly published mappings and identities.

    Recommended fix: Include the archive ID in the session fingerprint and add a test confirming that changing it forces session republication.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 16m36s

@wesm
wesm force-pushed the cool-lillipilli branch 2 times, most recently from e763307 to 5d4d2fa Compare July 31, 2026 12:12
@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (5d4d2fa)

The review found one medium-severity migration boundary issue; no high or critical findings were reported.

Medium

  • internal/db/orphaned.go:1888 — Incorrect trusted snapshot threshold

    The threshold is 76, but parser-source snapshots begin at data version 77. When upgrading a version-76 archive, legacy mapped-target snapshots can overwrite freshly reparsed source snapshots, preserving incorrect orphan identities.

    Suggested fix: Change the threshold to 77 and add a version-76 boundary test confirming that fresh source snapshots take precedence and legacy orphan snapshots are discarded.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 17m28s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (4f117c1)

One medium-severity issue should be addressed before merging.

Medium

  • internal/sync/engine.go:2385 — Full resync treats failures discovering or applying active worktree mappings as warnings and still swaps in the rebuilt database. Because sessions are initially restored to parser-source projects, mapping failures can leave sessions misclassified, including remote or orphaned sessions. Abort the resync and retain the existing database unless every active mapping machine is discovered and applied successfully.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 25m4s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (fbeee7b)

Changes need revision: one medium-severity stale snapshot deletion issue affects filtered PostgreSQL and DuckDB publications.

Medium

  • internal/db/project_identity.go:224 — Live snapshots use the owning session’s current project, while tombstones use the snapshot’s immutable project. If a session moves from source_project to target_project, a filtered target_project push publishes the source-labelled snapshot, but after hard deletion its tombstone is filtered out. Filtered PostgreSQL and DuckDB destinations can retain stale snapshot metadata until a full publication.
    • Suggested fix: Load snapshot tombstones without project filtering and let destination scope ownership determine applicable deletions, or journal the owner project with each deletion. Add filtered reclassified-session deletion tests for both backends.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 21m5s

@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (1cbb0bd)

Changes need fixes for two medium-severity data integrity issues.

Medium

  • internal/sync/engine.go:10080preserveUnavailableSourceProjects marks sessions as resolved before loading identity snapshots and treats lookup failures as success. A later write can replace both the session project and durable snapshot label with the parser fallback, preventing recovery of the original resolved project. Return the lookup error and abort or defer affected writes; set sourceProjectResolved only after the snapshot lookup succeeds.

  • internal/postgres/project_identity_upsert.go:119 — Existing PostgreSQL installations receive empty identity-ownership tables, but the first v3 filtered publication deletes rows only when matching ownership records exist. Legacy observations and snapshots deleted or moved outside the filter can remain ownerless and continue influencing identity resolution and Data candidates indefinitely. Add a one-time migration or adoption step that assigns legacy identity rows to their prior publication scope before filtered reconciliation.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 35m17s

…ication

Centralize archive-wide project governance in a dedicated Data mode so folder mappings are explicit, reviewable, and applied atomically without relying on Activity filters or ambiguous worktree terminology.

Follow-up fixes:
- reconcile reclassified identity aggregates
- bound reclassification identity updates
- refresh project filters after reclassification
- reload grouped session counts after deletion
- keep project identity views current
- preserve rule drafts during refresh
- count all matched reclassification projects
- preserve mappings across publication scopes
- preserve identity across publication scopes
- preserve PostgreSQL publication provenance
- reject untrustworthy version 76 identity snapshots
- abort resync when mapping restoration fails
- deliver snapshot tombstones across publication scopes
- fail closed when unavailable-source identity recovery fails
@wesm
wesm force-pushed the cool-lillipilli branch from bf76433 to fad6e06 Compare July 31, 2026 17:01
@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (fad6e06)

Changes look solid overall, but one medium-severity sync fallback issue should be fixed before merging.

Medium

  • internal/sync/engine.go:10082preserveUnavailableSourceProjects treats an empty parsed project or any os.Stat error other than ENOENT as successfully resolved. Permission errors, inaccessible mounts, or empty parser fallbacks can therefore bypass the durable snapshot and overwrite a previously resolved project classification.

    Suggested fix: Mark the source resolved only when os.Stat succeeds; otherwise consult the snapshot, including when the parsed project is empty. Add coverage for non-ENOENT errors and empty fallbacks.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 31m10s

An inaccessible checkout or an empty parser result is not authoritative evidence that a session changed projects. Treat those cases like a missing source and recover the last durable resolved identity, preventing transient filesystem failures from replacing a known classification.
@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (0a8a912)

Changes need fixes: three medium-severity data consistency and provenance issues remain.

Medium

  • internal/duckdb/probe.go:292 — DuckDB rebuild detection tracks the source database ID but not the propagated archive ID. Repairing an archive ID without changing the database ID leaves unchanged sessions and worktree mappings stamped with the old ID, breaking provenance-based joins. Store the source archive ID in mirror metadata, rebuild when it changes, and add a regression test for archive-ID repair.

  • internal/postgres/push.go:214 — Archive-ID mismatch handling clears push state but retains observations, snapshots, scopes, and worktree mappings owned by the old archive ID. A subsequent publication removes only rows associated with the new ID, leaving stale or duplicate rules and obsolete identity observations. Preserve the previous archive ID during repair, retire or reconcile its metadata, and test that no old archive-scoped rows remain.

  • internal/parser/vibe.go:103 — When an optional Vibe metadata field is malformed, fallback parsing recovers only the session ID and emits the default "vibe" project with an empty working directory. This can overwrite an existing repository classification, while the new preservation logic cannot protect records with empty working directories. Recover identity-bearing fields such as working directory and branch during fallback, or retain the stored project and working directory when full parsing fails.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 29m43s

Archive identity repairs must invalidate every mirrored provenance surface, not only session fingerprints and local watermarks. Rebuild disposable DuckDB mirrors when the propagated archive identity changes, and retire PostgreSQL metadata owned by the superseded identity before republishing it.

Malformed optional Vibe metadata is likewise not authoritative evidence that repository identity disappeared. Recover its independent working-directory and branch fields so transient parse failures cannot replace a known project classification with a generic fallback.
@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (c4f0e2a)

The changes introduce three medium-severity risks involving remote identity recovery and PostgreSQL metadata repair/publication.

Medium

  • internal/sync/engine.go:10099 — Identity snapshots are queried using parser IDs, but remote sessions are stored with idPrefix applied later. Remote imports therefore cannot recover an existing resolved identity when source lookup fails and may overwrite it with fallback classification. Apply idPrefix when querying stored snapshots while retaining a mapping to the corresponding parsed sessions.

  • internal/postgres/push.go:218 — Archive-ID repair deletes all metadata for the old archive, even during filtered publication. Sessions owned by other publication scopes can remain under that archive ID while losing identity and worktree metadata indefinitely. Retire only metadata exclusively owned by the current scope, or perform a target-wide provenance rewrite before deleting the old archive metadata.

  • internal/postgres/push.go:666 — The repaired source archive ID is persisted before identity and worktree metadata publication succeeds. If publication fails, the next push no longer triggers archive repair, and revision cursors may treat unpublished metadata as current and skip retries. Persist the repaired archive ID only after all metadata publication succeeds, or reset/key publication cursors by archive ID so failures force a retry.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 21m51s

Archive identity repair must remain retryable when governance publication fails, and a filtered publisher must not retire metadata still owned by another scope. Delaying repair finalization until identity and mapping publication succeeds preserves both guarantees while allowing the final repaired scope to clean up the superseded archive.

Remote identity recovery also needs to address snapshots by the prefixed ID used in storage; otherwise an unavailable remote checkout can fall back to the wrong project.
@roborev-ci

roborev-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown

roborev: Combined Review (a3343c2)

Code review passed with no Medium, High, or Critical findings.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 32m14s

@wesm
wesm merged commit cad0508 into main Jul 31, 2026
23 checks passed
@wesm
wesm deleted the cool-lillipilli branch July 31, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant