Skip to content

parser: capture sessionKind and promptSource from Claude Code transcripts - #1253

Merged
mjacobs merged 20 commits into
kenn-io:mainfrom
TweedBeetle:capture-sessionkind-promptsource
Aug 1, 2026
Merged

parser: capture sessionKind and promptSource from Claude Code transcripts#1253
mjacobs merged 20 commits into
kenn-io:mainfrom
TweedBeetle:capture-sessionkind-promptsource

Conversation

@TweedBeetle

@TweedBeetle TweedBeetle commented Jul 24, 2026 •
edited by mjacobs
Loading

Copy link
Copy Markdown
Contributor

What

Capture two Claude Code launch/prompt-provenance fields that the indexer currently reads past and discards:

  • sessionKind — session-level (e.g. "bg"). Present on background/headless sessions, absent on interactive ones. Lands in a new session_kind column on sessions.
  • promptSource — per user turn (e.g. "typed", "queued", "system", "sdk"). Marks whether a turn was a human-typed prompt or a queued/system/SDK-injected one. Lands in a new prompt_source column on messages.

Why

Both signals exist on essentially every current Claude Code transcript but are thrown away at parse time, so the archive can't answer two questions it otherwise has the data for: was a session interactive or headless, and was a given prompt typed by a human or injected. This mirrors the recently-added entrypoint capture — same shape of information, same reason to keep it.

How

The change follows the existing entrypoint / agent_label and source_type / is_sidechain handling rather than inventing a new pattern:

  • sessionKind is treated as a session-identity field alongside entrypoint: first-non-empty-wins across the file, wired into the incremental identity-escalation check, and blanked by the Cowork and Qoder parsers that reuse the Claude parse body (exactly as they blank entrypoint).
  • promptSource is treated as a per-message source column alongside source_type/is_sidechain, including the retained-field whitelist in compactClaudeEntry and the message token fingerprints the PG/DuckDB push fast-paths compare against.

Both columns are NOT NULL DEFAULT '', so transcripts that predate the fields and other agents that never emit them read empty and are otherwise untouched. A dataVersion bump (79) triggers the standard reparse so existing Claude rows backfill both fields. Capture is mirrored across the SQLite, Postgres, and DuckDB stores (mirror SchemaVersion 9) and the artifact wire format, where the manifest advances to v3 and message segments to v2: older artifacts still decode with the fields empty, while pre-provenance peers park new artifacts behind the existing future-version gate instead of silently dropping the fields. The fields also round-trip through artifact import, re-enqueue a published session's export when only session_kind changes, and count toward the export preflight's message-byte bound.

Scope / limitations

  • Pure field capture — no counting, aggregation, health-signal, or automation-classification logic is touched.
  • session_kind/prompt_source backfill via the dataVersion-79 reparse; rows whose source files no longer exist stay empty.
  • No new API/CLI surface or UI is added; the columns are available for consumers that read the archive directly.

Where to look

  • internal/parser/claude.go — the capture site (sessionKind next to entrypoint; promptSource next to isSidechain) and the compactClaudeEntry retained-field whitelist.
  • internal/parser/claude_provider.go, provider.go, internal/sync/engine.go — incremental identity threading for sessionKind.
  • internal/db/{schema.sql,db.go,sessions.go,messages.go} — SQLite columns, migrations, and read/write plumbing.
  • internal/postgres/* and internal/duckdb/* — the mirrored columns and fingerprints.

The source of truth for both fields is the Claude Code transcript format, where sessionKind and promptSource appear as top-level keys on session JSONL entries.

@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (e35f71d)

The change is incomplete: three Medium-severity issues could leave Claude metadata stale, break fingerprint comparisons, or lose prompt_source across persistence paths.

Medium

  • internal/db/db.go:324 — The parser’s persisted output changed, but dataVersion remains 71. Existing unchanged Claude sessions will be considered current and retain empty session_kind and prompt_source values indefinitely. Bump the data version to 72, document the reparse reason, and update the regression test.

  • internal/db/fingerprints_batch.go:125 — tokenFingerprintRow.appendTo hashes promptSource, but the batched query and scan never populate it. PostgreSQL batch comparison may skip backfilling a changed prompt_source or unnecessarily rewrite messages. Select and scan prompt_source, then test that batched and per-session fingerprints match for non-empty values.

  • internal/db/orphaned.go:975, internal/artifact/wire.go:175, internal/server/upload.go:208 — Several persistence paths omit PromptSource: full-resync orphan copying loses it, artifact segments do not export it, and uploaded Claude transcripts discard it during database conversion. Add prompt_source to the orphan-copy columns, artifact DTO and mapping, and upload conversion, with round-trip coverage for each path.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 5m43s

@mjacobs

mjacobs commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

looking

@mjacobs
mjacobs force-pushed the capture-sessionkind-promptsource branch from e35f71d to 0db83fc Compare July 24, 2026 20:19
@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (0db83fc)

The changes are generally sound, but uploaded Claude sessions lose prompt provenance.

Medium

  • internal/server/upload.go:208 — The Claude transcript upload converter omits m.PromptSource, so uploaded sessions silently lose per-message prompt provenance. Because uploaded files live outside normal Claude discovery roots, the value will not be backfilled automatically. Map PromptSource: m.PromptSource into each db.Message and add an upload-conversion test covering both SessionKind and PromptSource.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 8m28s

@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (0db83fc)

One medium-severity issue found: the Claude upload path drops per-message prompt provenance.

Medium

  • internal/server/upload.go:208 — sessionBatchWriteFromParsed does not copy ParsedMessage.PromptSource into the corresponding db.Message. Claude transcripts imported through the upload API therefore permanently lose prompt provenance even though the parser captured it. Populate PromptSource during conversion and add an upload-conversion round-trip test.

Reviewers: 2 done | Synthesis: codex, 5s | Total: 4m3s

@roborev-ci

roborev-ci Bot commented Jul 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (7219443)

No issues found.

Reviewers: 2 done | Synthesis: codex | Total: 5m1s

@roborev-ci

roborev-ci Bot commented Jul 25, 2026

Copy link
Copy Markdown

roborev: Combined Review (7219443)

Frontend provenance fields are not fully propagated to the response/domain types.

Medium

  • Locations: frontend/src/lib/api/types/core.ts:22, frontend/src/lib/api/types/core.ts:90, frontend/src/lib/api/types/core.ts:143, frontend/src/lib/api/generated/models/ServiceSessionDetail.ts:6
  • Finding: The frontend response/domain types omit session_kind and prompt_source. As a result, sidebarIndexRowToSession discards session_kind, and session detail/message consumers cannot access the new fields type-safely. Updating only DbSession, DbSidebarSessionIndexRow, and DbMessage does not cover endpoints consumed through ServiceSessionDetail and the manual core interfaces.
  • Recommended fix: Add the fields to Session, SidebarSessionIndexRow, and Message; preserve them in the sidebar mapping; regenerate ServiceSessionDetail; and add frontend coverage verifying the fields survive API-to-store conversion.

Reviewers: 2 done | Synthesis: codex, 11s | Total: 7m0s

@mariusvniekerk
mariusvniekerk force-pushed the capture-sessionkind-promptsource branch from 7219443 to 386d033 Compare July 27, 2026 02:08
@roborev-ci

roborev-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown

roborev: Combined Review (386d033)

Code review passed: no Medium, High, or Critical findings.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 11m20s

@mjacobs
mjacobs force-pushed the capture-sessionkind-promptsource branch from 386d033 to 2f23a52 Compare August 1, 2026 01:57
@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (2f23a52)

One medium-severity issue: artifact imports silently discard PromptSource.

Medium

  • internal/artifact/wire_decode.go:336 — segmentMessage.dbMessage() does not copy PromptSource, so exports contain the field but imports lose it. Map PromptSource: m.PromptSource and add an encode/decode round-trip test.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 6m0s

@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (e35810f)

The changes are broadly sound, but two medium-severity artifact consistency and compatibility issues should be fixed before merging.

Medium

  • internal/db/db.go:2302 — Session-kind-only updates do not trigger artifact export

    session_kind is export-visible but missing from artifact_sessions_update_queue change detection. Updating only this field on an already-published session leaves its artifact manifest stale.

    Fix: Add OLD.session_kind IS NOT NEW.session_kind to the trigger and cover a session-kind-only update with an export-queue test.

  • internal/artifact/wire.go:42 — Schema changes were made without advancing format versions

    The manifest and segment schemas gained session_kind and prompt_source, but their format versions were not advanced. Older peers may accept these artifacts under versions they claim to understand, silently discard the unknown provenance, and mark the import complete.

    Fix: Emit new manifest and segment versions while retaining decode support for the preceding versions, defaulting the new fields to empty for older artifacts.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 7m41s

@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (905325a)

Provenance support is broadly sound, but artifact export memory accounting misses one new field.

Medium

  • internal/db/artifact_export_load.go:36 — artifactMessageRawBytesSQL omits prompt_source, causing export preflight to undercount message bytes and potentially exceed the explicit MessageBytes memory bound. Include length(CAST(prompt_source AS BLOB)) in the calculation and add a raw-byte boundary test using PromptSource.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 7m46s

@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (ac9b3ed)

Frontend provenance metadata is incomplete, so the PR needs a mapping and type fix before merge.

Medium

  • frontend/src/lib/stores/sessions.svelte.ts:1539 — The sidebar mapper drops session_kind, and the manual Session, SidebarSessionIndexRow, and Message types omit session_kind/prompt_source. ServiceSessionDetail also lacks session_kind. As a result, index-only sessions lose provenance and frontend code cannot safely consume the new fields. Add the fields to the frontend types, preserve session_kind in sidebarIndexRowToSession, regenerate or fix ServiceSessionDetail, and add mapping tests.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 8m23s

TweedBeetle and others added 16 commits July 31, 2026 23:51
@TweedBeetle @mjacobs
parser: capture sessionKind and promptSource from Claude Code transcr…
7f65f2f 
…ipts

Claude Code JSONL entries carry two launch/prompt-provenance fields that
the indexer currently discards:

- `sessionKind` (session-level, e.g. "bg"): distinguishes headless /
  background sessions from interactive ones.
- `promptSource` (per user turn, e.g. "typed", "queued", "system",
  "sdk"): distinguishes a human-typed prompt from a queued, system, or
  SDK-injected one.

Neither is currently recoverable from the archive, so downstream
consumers cannot tell a headless run from an interactive one, nor a typed
prompt from an injected one.

Both are captured by mirroring the existing `entrypoint`/`agent_label`
handling exactly:

- `sessionKind` becomes a session-identity field alongside `entrypoint`
  (first-non-empty-wins, incremental identity escalation, sessions
  column). It is blanked for the Cowork and Qoder parsers that reuse the
  Claude parse body, matching how they blank `entrypoint`.
- `promptSource` becomes a per-message column alongside `source_type`
  and `is_sidechain`, including the retained-field whitelist in
  compactClaudeEntry and the message token fingerprints that the PG/DuckDB
  push fast paths compare against.

Both columns default to empty, so historical transcripts that predate the
fields and other agents that never emit them are unaffected. Capture is
mirrored across the SQLite, PostgreSQL, and DuckDB stores and the artifact
manifest. No counting or aggregation logic changes; this is field capture
only.

The source of truth for both fields is the Claude Code transcript format
itself, where `sessionKind` and `promptSource` appear as top-level keys on
session JSONL entries.
@mjacobs @claude
fix(db): select prompt_source in batched message token fingerprints
8d32d5e 
The batched MessageTokenFingerprints query omitted prompt_source while
tokenFingerprintRow.appendTo already hashed it, so batched fingerprints
disagreed with the per-session MessageTokenFingerprint for any non-empty
value, causing perpetual push mismatches. The fixture now carries a
non-empty prompt_source so the batched/per-session identity test pins
the field.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
fix(db): keep message inserts under the 999-variable SQLite limit
69b91a8 
Rows now bind 26 parameters, so 39 rows per statement overflowed the
historic 999-variable limit (1014 binds). Reduce to 38 rows (988).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
fix(db): preserve prompt_source across orphan copy and revision recon…
f5352db 
…cile

The full-resync orphan copy's message column allowlist omitted
prompt_source, silently dropping it for sessions whose source files no
longer exist. The transcript-revision reconcile comparison now treats
prompt_source as a peer of source_subtype; as with that column's own
introduction, sources predating the column skip the reconcile pass. Adds
a round-trip test covering session_kind and prompt_source survival.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
fix(db): bump dataVersion to 72 for session_kind/prompt_source backfill
51b8a70 
Parser output changed but the data version stayed at 71, so already
archived Claude sessions would never re-parse and their session_kind and
prompt_source columns would stay empty indefinitely. Bumping forces the
standard full resync, matching how agent_label/entrypoint landed in v66.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
fix(sync): count prompt_source in recompute heap accounting
ca5a4cb 
recomputeHeapBytes sums per-message string field sizes to bound heap
release; the new prompt_source field was missing from the sum.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
test(sync): cover session_kind and prompt_source parse-diff drift
f384eb8 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
docs: record sessionKind/promptSource in Claude Code format evidence
c5a233d 
Per the provider format provenance rule, note the 2026-07-24 local
reverification that observed the two top-level keys the parser now
captures; evidence class stays no-public-source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
feat(artifact): carry prompt_source in message segments
9a158ec 
segmentMessage mirrored source_subtype but omitted the new
prompt_source field, so artifact exports dropped it. The canonical
golden is unchanged because the field is omitempty and absent there.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
test(db): pin current data version at 72
9e18b80 
Follows the dataVersion bump for session_kind/prompt_source backfill;
the pin test's name and reason now reference the new version.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
feat(db): expose session_kind in the sidebar session index
7b39268 
SidebarSessionIndexRow carried agent_label and entrypoint but not the
new session_kind, so the sidebar API could surface one identity field
and not the other. Adds the column to the SQLite, PostgreSQL, and
DuckDB sidebar queries and scans, keeping all backends in lockstep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
chore(frontend): regenerate API models for session_kind/prompt_source
3385d5a 
npm run generate:api picked up the new fields on DbMessage, DbSession,
and DbSidebarSessionIndexRow. Unrelated generator drift from earlier
merges (duckdb attach_timeout, vector push flags, source-layout stats)
is left out to keep this change scoped; it needs its own regen chore.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
fix(duckdb): bump mirror SchemaVersion to 6 for new columns
0872899 
The mirror gained sessions.session_kind and messages.prompt_source; per
the mirror invariant, a schema change bumps SchemaVersion and forces a
full rebuild. The column shape gate already rebuilds old mirrors, so
this makes the version metadata agree with the shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs @claude
fix(parser): carry promptSource through queued-command attachments
26820e5 
extractQueuedCommand dropped the top-level promptSource on
type=attachment queued_command entries, so queued prompts stored an
empty messages.prompt_source while standard user turns captured it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mjacobs
test(storage): cover Claude provenance round trips
788adcd
@mjacobs
fix(server): preserve prompt source on upload
e4fdaf8
mjacobs added 4 commits July 31, 2026 23:52
@mjacobs
fix(artifact): restore prompt source on segment import
e114a45 
Exports already carry prompt_source on segment messages, but
segmentMessage.dbMessage() dropped it during import, so round-tripped
archives lost the field. Copy it through and extend the round-trip test
to cover the decode side.
@mjacobs
fix(db): enqueue artifact export on session_kind-only updates
b8c062e 
session_kind is export-visible via the artifact manifest, but the
artifact_sessions_update_queue trigger's change-detection list did not
include it, so a session-kind-only update on an already-published
session left its manifest stale. Add the column to the trigger and pin
it with an export-queue test.
@mjacobs
fix(artifact): advance wire versions for provenance fields
8b5f16b 
The manifest and segment schemas gained session_kind and prompt_source
without a version bump, so pre-provenance peers would import new
artifacts under versions they claim to understand, silently drop the
unknown fields, and mark the import complete. Advance the manifest
format to v3 and the message segment format to v2 so older peers park
new artifacts behind the future-version gate until they upgrade, and
add explicit min-decode versions so v2 manifests and v1 segments still
decode with the provenance fields defaulting to empty.
@mjacobs
fix(db): count prompt_source in export message byte preflight
bd05ef3 
artifactMessageRawBytesSQL bounds the raw string bytes materialized for
one artifact export, but it omitted the new prompt_source column, so
preflight undercounted message bytes against the explicit MessageBytes
limit. Include the column and pin the boundary with a PromptSource
raw-byte test.
@mjacobs
mjacobs force-pushed the capture-sessionkind-promptsource branch from ac9b3ed to bd05ef3 Compare August 1, 2026 03:55
@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (bd05ef3)

No issues found.

Reviewers: 2 done | Synthesis: codex | Total: 6m10s

@mjacobs
mjacobs merged commit 1cd581f into kenn-io:main Aug 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@TweedBeetle @mjacobs