Voice commenting: push-to-talk dictation pinned to what you were reading - #175
Conversation
Hold Shift (or tap the mic) and say 'this section is way too formal' — it posts as a comment pinned to the passage on screen. The pieces: - Capture: ear-at-keydown push-to-talk (recording starts at the press, not after the hold is confirmed), MediaRecorder Opus/WebM or Safari MP4/AAC, with a level meter driving the button's ring and a fail-open silence check. Browser SpeechRecognition is the fallback when no AI provider is configured. - Transcription: gpt-4o-transcribe, prompted with the text that was readably on screen during the take so jargon and figures survive. Guards for the ways that goes wrong: prompt echo (silence in, page text out) and fabrication (a 'transcript' longer than the recording had seconds to hold — the model answering an instruction-shaped remark instead of writing it down; caught by duration, retried without the prompt). - Interpretation: one gpt-4o call cleans the words and quotes the exact span they were about, vetted verbatim against the excerpt and resolvability against the markdown source. A remark about several passages becomes several pinned comments. Length trust band keeps the cleanup from becoming a summary or an invention; every failure falls back to a locally tidied transcript. - Placement: the excerpt accumulates across mid-take scrolls, spans resolve to the copy nearest the viewport, unresolvable pins fall back to the section heading, and the posted thread auto-opens so you see where it landed. The controller watches for an agent presence pill and speaks an acknowledgment when one engages; without the (upcoming) agent collaboration surfaces that's a no-op by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d06280f8a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| this._onInterrupt = () => { | ||
| if (this.pushToTalk) this._cancel() | ||
| } |
There was a problem hiding this comment.
When the window loses focus during the initial 350 ms hold delay, pushToTalk is still false, so this handler does nothing: the timer later starts recording while the window is unfocused, and the corresponding Shift keyup may never reach the page. This can leave the microphone recording indefinitely and submit everything when the user eventually releases Shift in the page; the interrupt handler must also clear the pending timer and close this.ear.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🤖 Confirmed and fixed in 596bb81 — interrupt() now clears the hold timer and closes the ear unconditionally, so a blur during the hold delay can no longer arm a take whose keyup never arrives.
| disconnect() { | ||
| this._closeEar() | ||
| this._releaseMic() | ||
| this.recognition?.abort() |
There was a problem hiding this comment.
When Turbo navigates away while _startRecording() is awaiting getUserMedia, that method has already detached this.ear, and no stream exists yet for these cleanup calls to release. Because listening is also left true, the promise can later resume and start a MediaRecorder on the disconnected controller with its button and key handlers gone, potentially leaving the microphone active indefinitely; disconnect must mark the take discarded/stopped or otherwise invalidate the pending async capture.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🤖 Confirmed and fixed in 596bb81 — disconnect() now marks the take dead (listening = false, discarded = true) before teardown, so a _startRecording resumed after navigation bails through its own cleanup and releases the stream.
| document.addEventListener("keydown", this._onKeyDown) | ||
| document.addEventListener("keyup", this._onKeyUp) | ||
| window.addEventListener("blur", this._onInterrupt) |
There was a problem hiding this comment.
The document and window are stable targets that Stimulus supports through keydown@document, keyup@document, and blur@window, so the dynamic-element exception does not apply here. Move these handlers into data-action bindings rather than maintaining a parallel manual listener lifecycle.
AGENTS.md reference: AGENTS.md:L90-L90
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🤖 Done in 596bb81 — push-to-talk is now bound via data-action="keydown@document->coplan--voice#keyDown keyup@document->coplan--voice#keyUp blur@window->coplan--voice#interrupt"; the manual listener lifecycle is gone.
| def recent_comments | ||
| Comment.joins(:comment_thread) | ||
| .where(comment_thread: { plan_id: @plan.id }) | ||
| .order(created_at: :desc).limit(3) | ||
| .includes(:comment_thread) |
There was a problem hiding this comment.
When one of the three newest comments has been soft-deleted, this unscoped query still reads its body_markdown and sends it to the AI provider as conversational context. Deleted comments are intentionally excluded elsewhere via Comment.kept, so a later voice dictation can unexpectedly disclose text the author removed; apply the kept scope before ordering and limiting the context.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🤖 Fixed in 596bb81 with Comment.kept, plus a request spec asserting a deleted comment's text never reaches the provider.
The coplan--comment-nav div was gated on @threads.any?, decided at page render. But comments arrive live — voice dictation, selection comments, other viewers' broadcasts — so on a fresh plan the first comment appeared, its popover auto-opened, and d/j/k/r/a/s did nothing at all: the controller holding the key listeners was never on the page. Render it unconditionally; it no-ops fine with zero highlights. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three ways the microphone or a retracted comment could outlive its moment, all from review: - Losing window focus during the 350ms hold delay left the pending timer armed: it started a take whose keyup could never arrive, recording indefinitely. interrupt() now clears the timer and closes the ear regardless of whether push-to-talk was confirmed. - Turbo navigation while _startRecording awaited getUserMedia left listening=true, so the promise resumed onto a disconnected controller and started a recorder nobody could see or stop. disconnect() now marks the take dead first; the pending resumption bails through its own cleanup and releases the stream. - The push-to-talk listeners are now declarative Stimulus actions (keydown@document / keyup@document / blur@window) per AGENTS.md — document and window are stable targets, so the manual listener lifecycle was unwarranted. - The dictation context now uses Comment.kept: a deleted comment's text was removed on purpose and must not ride along to the AI provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
What
Hold Shift (or tap the floating mic) and talk — what you said posts as a comment pinned to the passage that was on screen, cleaned up and auto-opened so you see where it landed.
How it works
Capture. Recording starts at the Shift press, not after the 350ms hold-confirmation — people start talking immediately, and the first word matters ("oh, I meant both of them"). A tap, a Shift-shortcut, or extending a selection discards the take unheard. A level meter drives a ring on the button (the fill says "mic is open", the ring says "it can hear you") and backstops against posting silence — failing open when the AudioContext never gets a user gesture. Browsers without a configured AI provider fall back to the Web Speech API.
Transcription.
gpt-4o-transcribe(override:COPLAN_TRANSCRIBE_MODEL), prompted with the text that was readably visible during the take — that hint is the difference between "CoPlan" and "co-plan". Two guards for how that model fails:Interpretation. One
gpt-4ocall cleans the words (fillers, false starts, mis-transcribed jargon) and quotes the exact span the remark was about. Nothing it returns is trusted:CommentThread.strip_markdown) — a span crossing table cells degrades to its longest resolvable line/run, or is dropped;A remark about several passages ("rename both of these") becomes up to four pinned comments; a repeated span means successive occurrences. Recent comments ride along as context so follow-ups resolve.
Placement. The excerpt accumulates across mid-take scrolls (blocks count only when readably visible, not one-pixel slivers). The occurrence picked is the copy nearest the viewport. An unresolvable pin falls back to the current section heading; a server 422 (anchor-must-resolve validation) retries the same way. The posted thread scrolls into view and opens.
Notes for review
_watchAgentPill/ spoken acknowledgments reference an agent-presence pill that doesn't exist yet — they no-op by design and light up when the agent-collaboration PR lands.Tests
76 examples green: request specs for the dictation endpoint (echo, fabrication, formats, auth), unit specs for interpretation (span vetting, length band, multi-comment, fallbacks), and 14 system specs driving a fake MediaRecorder through capture, mid-take scrolls, multi-pin, auto-open, and failure reporting.
🤖 Generated with Claude Code