Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions .empire/VOICE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Buzz Voice Panel

Issue #13 adds a read-only `/voice` surface backed by Codex Realtime through the existing ChatGPT subscription login. It never uses `OPENAI_API_KEY`, a paid API fallback, an upgrade, or a mutating tool.

## Preflight evidence

- Issue #2 is closed.
- Issue #14 is the follow-up Voice-State-Tools scope, not a prerequisite.
- OAuth issue #18 is closed.
- `codex login status` reports `Logged in using ChatGPT`.
- `OPENAI_API_KEY` was absent from the environment and explicitly removed from the live probe child.
- `codex app-server --stdio --enable realtime_conversation` initialized with model `gpt-5.6-sol` and delivered `thread/realtime/sdp` for voice `spruce`; no 403 or quota error occurred.
- The immediate `thread/realtime/start` result was `{}`. The client therefore waits for the SDP notification instead of treating acknowledgement as success.

## Security boundary

- One process-backed Rust actor serializes app-server JSON-RPC.
- The frontend owns microphone capture, WebRTC, remote audio, transcript rendering, and cleanup.
- Session instructions contain only a maximum 16-KiB local snapshot and explicit read-only refusal rules.
- No Realtime tools are attached.
- Snapshot inputs are the existing local cockpit contract plus a direct authenticated relay query.
- Missing sources are named gaps; they are never represented as zero or healthy.
- SDP, snapshot content, signing keys, auth tokens, environment values, and hidden paths are not logged.
- Stop is idempotent and terminates the app-server child when the actor can no longer be reused safely.

## Visual system

The ten references below were generated on 2026-08-13 through the built-in keyless image-generation path. That path requires no `OPENAI_API_KEY` and caused no user spend. Their prompts and SHA-256 hashes are recorded in [`voice-visuals/manifest.json`](voice-visuals/manifest.json).

The implementation takes four recurring decisions from this set:

- near-black ink/graphite surface with subtle noise rather than a generic AI gradient;
- warm amber reserved for active state, focus, and the primary action;
- one large voice control paired with a quieter transcript/context region;
- explicit entitlement, quota, and source-gap states that never masquerade as success.

### 01 — Idle

![Idle voice panel](voice-visuals/01-idle.png)

### 02 — Microphone permission

![Microphone permission](voice-visuals/02-microphone-permission.png)

### 03 — Connecting

![Connecting to Codex Realtime](voice-visuals/03-connecting.png)

### 04 — Listening

![Active listening session](voice-visuals/04-listening.png)

### 05 — User transcript

![User transcript focus](voice-visuals/05-user-transcript.png)

### 06 — Assistant speaking

![Assistant speaking](voice-visuals/06-assistant-speaking.png)

### 07 — Entitlement denied

![Realtime entitlement denied](voice-visuals/07-entitlement-denied.png)

### 08 — Quota unavailable

![Realtime quota unavailable](voice-visuals/08-quota-unavailable.png)

### 09 — Snapshot gaps

![Partial read-only snapshot](voice-visuals/09-snapshot-gap.png)

### 10 — Narrow laptop

![Narrow laptop composition](voice-visuals/10-narrow-laptop.png)

## Local verification contract

The supported Windows parity gate is the sequence documented in `.empire/BUILD.md`: workspace format and clippy, Tauri format, desktop and web checks, and unit tests. Whole-desktop Windows Tauri clippy findings already named there are baseline exceptions; warnings introduced by the voice module are not accepted.

Live proof runs with `OPENAI_API_KEY` absent. Success requires a real `thread/realtime/sdp` notification, route rendering, deterministic WebRTC lifecycle coverage, and clean stop. Physical microphone interaction is recorded separately when desktop automation cannot grant the host device.

## Local proof — 2026-08-14

- A no-key `codex app-server --stdio --enable realtime_conversation` probe initialized through the existing ChatGPT login, started thread `019ffcb6…`, and received a real `thread/realtime/sdp` notification for `spruce` with no entitlement or quota error.
- The focused Rust suite passed 12/12 tests after a fresh Tauri test build, including read-only params, acknowledgement-vs-SDP sequencing, sanitized errors, snapshot gaps, and the 16-KiB bound.
- The focused frontend suite passed 7/7 tests, including WebRTC negotiation and failure cleanup.
- Chromium E2E passed 1/1: sidebar navigation, `/voice`, `voice_start`, snapshot rendering, clean `voice_stop`, and zero page errors.
- The E2E screenshot is [`voice-panel-live-proof.png`](voice-panel-live-proof.png), SHA-256 `6e9ec6202066d9cbc8b3aa42c94306658b648ec19dab4689063aefe510ee69e5`.
- The browser fixture replaces the host microphone and peer connection deterministically; the no-key app-server probe supplies the local entitlement/SDP parity that browser automation cannot exercise through Tauri IPC.
Binary file added .empire/voice-panel-live-proof.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/01-idle.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/03-connecting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/04-listening.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/05-user-transcript.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/06-assistant-speaking.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/07-entitlement-denied.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/08-quota-unavailable.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/09-snapshot-gap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .empire/voice-visuals/10-narrow-laptop.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
80 changes: 80 additions & 0 deletions .empire/voice-visuals/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
{
"schemaVersion": 1,
"generatedAt": "2026-08-13T23:17:00+02:00",
"generationPath": "built-in-keyless-imagegen",
"cost": "no-user-spend",
"format": "horizontal PNG UI reference",
"sharedPrompt": "Ink and graphite Buzz desktop application, warm amber/honey signal, neutral Swiss editorial hierarchy, subtle noise texture, asymmetric premium layout, accessible contrast, simple cards, no purple, no blue AI gradients, no glassmorphism overload, no pill clutter, no logos, no watermark.",
"references": [
{
"index": 1,
"state": "idle",
"file": "01-idle.png",
"prompt": "Premium read-only voice command panel before microphone activation, with one dominant circular microphone control, quiet snapshot status, concise read-only security cue, and restrained empty transcript area.",
"sha256": "524a07b9c6410b10bd695d7f2ced0807bf9aae31e3393a8c96cae528eb712512"
},
{
"index": 2,
"state": "microphone-permission",
"file": "02-microphone-permission.png",
"prompt": "Voice panel explaining that microphone access is needed, with a focused permission card, concise privacy note, and one dominant allow-microphone action.",
"sha256": "b8a124016a55b2a3642a89d6d6533f0f03c347c13a02f7c271b981e568fcdc7e"
},
{
"index": 3,
"state": "connecting",
"file": "03-connecting.png",
"prompt": "Voice panel during WebRTC and Codex Realtime negotiation, with restrained connection motion, snapshot preparation status, and disabled microphone control.",
"sha256": "51f63585a69a947e545f4947c32d0f158e2dc391cacf349b431354121981ca8a"
},
{
"index": 4,
"state": "listening",
"file": "04-listening.png",
"prompt": "Live read-only voice session with active amber waveform, elapsed time, snapshot freshness, transcript pane, and obvious stop control.",
"sha256": "d9a5a1ea3212aa7cba3c2b0ca438547bc67c53052a8a3f4bb14f3787c35dbf93"
},
{
"index": 5,
"state": "user-transcript",
"file": "05-user-transcript.png",
"prompt": "Live voice session where the user's latest phrase is the transcript focus while the microphone waveform remains active and earlier content is subdued.",
"sha256": "e8984344f07eb6b4b98285cb9959d649a9c1c8ed4dfec19ca6ec02d8950acba9"
},
{
"index": 6,
"state": "assistant-speaking",
"file": "06-assistant-speaking.png",
"prompt": "Read-only assistant actively replying, with amber response waveform, assistant transcript as focus, quieter user question, elapsed time, and stop control.",
"sha256": "4be9be52978855f87a0f77ac2f1342599164301fd0e3b523ffee93d89e074e73"
},
{
"index": 7,
"state": "entitlement-denied",
"file": "07-entitlement-denied.png",
"prompt": "Recoverable 403 entitlement error with inactive microphone, explicit no-paid-fallback message, preserved read-only snapshot status, and retry action.",
"sha256": "8c8a632ef4fd8f36d75099c0810b057eef66c00b557b1784089c3ecd23270ed7"
},
{
"index": 8,
"state": "quota-unavailable",
"file": "08-quota-unavailable.png",
"prompt": "Calm quota-unavailable state that refuses a paid fallback, keeps the latest snapshot visible, and offers a later retry action.",
"sha256": "e0b8bcc0176b511571f1786d384a889c69d3912dda975b8792694ff28cea4942"
},
{
"index": 9,
"state": "snapshot-gap",
"file": "09-snapshot-gap.png",
"prompt": "Partial-context state naming a stale cockpit snapshot and unavailable relay feed as explicit gaps, never as zero or healthy.",
"sha256": "debf86de498a50005cc6aba420d26b715366653bd68f6e5e51956bc6819fc0f9"
},
{
"index": 10,
"state": "narrow-laptop",
"file": "10-narrow-laptop.png",
"prompt": "Active listening panel compressed for a narrow laptop with compact navigation, stacked transcript and snapshot health, large stop control, and no horizontal overflow.",
"sha256": "3e116c0af8f770f8156374029d92dd3742cdf4f30e34231c753be06daa92f84a"
}
]
}
1 change: 1 addition & 0 deletions desktop/playwright.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ export default defineConfig({
"**/harness-catalog-screenshots.spec.ts",
"**/inline-custom-harness.spec.ts",
"**/huddle-transcription.spec.ts",
"**/voice-panel.spec.ts",
],
use: {
...devices["Desktop Chrome"],
Expand Down
6 changes: 6 additions & 0 deletions desktop/src-tauri/src/commands/empire_cockpit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,12 @@ fn envelope_for(path: &Path, collector: Option<CollectorOutcome>) -> EmpireSnaps
}
}

pub(crate) fn load_empire_snapshot_for_voice() -> Result<serde_json::Value, String> {
let path = snapshot_path()?;
let (snapshot, read_error, _) = load_snapshot(&path);
snapshot.ok_or_else(|| read_error.unwrap_or_else(|| "Snapshot nicht verfuegbar".to_string()))
}

/// Reads the current cockpit snapshot. Never fails for "no data" — that is a
/// populated `readError`, which is what the UI must render as a gap.
#[tauri::command]
Expand Down
10 changes: 5 additions & 5 deletions desktop/src-tauri/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ mod templates;
#[cfg(target_os = "macos")]
mod tray_menu;
mod util;
mod voice_assistant;
#[cfg(target_os = "linux")]
pub mod webkit_rendering;
use app_state::{build_app_state, resolve_persisted_identity, AppState};
Expand Down Expand Up @@ -72,6 +73,7 @@ use tauri::{Listener, WindowEvent};
use tauri_plugin_window_state::StateFlags;
#[cfg(target_os = "macos")]
use tray_menu::show_main_window;
use voice_assistant::{voice_start, voice_stop, VoiceAssistantState};

#[cfg(target_os = "macos")]
const INITIAL_RENDER_READY_EVENT: &str = "initial-render-ready";
Expand Down Expand Up @@ -142,11 +144,6 @@ async fn wait_for_stable_initial_window_geometry<R: tauri::Runtime>(window: &tau

#[cfg_attr(mobile, tauri::mobile_entry_point)]
pub fn run() {
// mesh-llm's async chains (model download, node start/join) overflow
// tokio's default 2 MiB worker stacks — a stack-guard SIGABRT, not a
// panic. Upstream mesh-llm and mesh-console both run on 8 MiB worker
// stacks for this reason; give Tauri's command runtime the same headroom
// before anything else touches tauri::async_runtime.
#[cfg(feature = "mesh-llm")]
match tokio::runtime::Builder::new_multi_thread()
.enable_all()
Expand Down Expand Up @@ -368,6 +365,7 @@ pub fn run() {
.manage(BuilderlabSession::default())
.manage(BuilderlabLogin::default())
.manage(commands::pairing::PairingHandle::new())
.manage(VoiceAssistantState::default())
.setup(move |app| {
let app_handle = app.handle().clone();
#[cfg(target_os = "macos")]
Expand Down Expand Up @@ -667,6 +665,8 @@ pub fn run() {
acknowledge_pending_community_deep_link,
read_empire_snapshot,
refresh_empire_snapshot,
voice_start,
voice_stop,
start_builderlab_login,
cancel_builderlab_login,
get_builderlab_auth,
Expand Down
Loading
Loading