elevenlabs · tadaspetra · May 21, 2026 · May 18, 2026 · May 21, 2026 · May 21, 2026
diff --git a/.cursor/skills/scaffold-elevenlabs-example/reference.md b/.cursor/skills/scaffold-elevenlabs-example/reference.md
@@ -22,6 +22,7 @@ Current repo skills:
 - `/music`
 - `/setup-api-key`
 - `/sound-effects`
+- `/speech-engine`
 - `/speech-to-text`
 - `/text-to-speech`
 
@@ -41,17 +42,18 @@ Ignore the deprecated root `examples/` folder for new work.
 
 ## Current example matrix
 
-| Path                                   | Shared template        | Prompt sections                                                                 | Setup extras                                                                                                        |
-| -------------------------------------- | ---------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
-| `text-to-speech/typescript/quickstart` | `templates/typescript` | `index.ts`                                                                      | Copies `.env`, preserves `node_modules`, installs with `pnpm`                                                       |
-| `text-to-speech/python/quickstart`     | `templates/python`     | `main.py`                                                                       | Copies `.env`, preserves `.venv`, installs with `pip`                                                               |
-| `speech-to-text/typescript/quickstart` | `templates/typescript` | `index.ts`                                                                      | Optional `assets/`, copies `.env`, preserves `node_modules`                                                         |
-| `speech-to-text/python/quickstart`     | `templates/python`     | `main.py`                                                                       | Optional `assets/`, copies `.env`, preserves `.venv`                                                                |
-| `music/typescript/quickstart`          | `templates/typescript` | `index.ts`                                                                      | Copies `.env`, preserves `node_modules`, installs with `pnpm`                                                       |
-| `music/nextjs/quickstart`              | `templates/nextjs`     | `app/api/generate-music/route.ts`, `app/page.tsx`                               | Adds `@elevenlabs/elevenlabs-js`, copies `.env.local`, preserves `node_modules` and `.next`                         |
-| `speech-to-text/nextjs/realtime`       | `templates/nextjs`     | `app/api/scribe-token/route.ts`, `app/page.tsx`                                 | Adds `@elevenlabs/react` and `@elevenlabs/elevenlabs-js`, copies `.env.local`, preserves `node_modules` and `.next` |
-| `agents/nextjs/quickstart`             | `templates/nextjs`     | `app/api/agent/route.ts`, `app/api/conversation-token/route.ts`, `app/page.tsx` | Same Next.js setup pattern, removes `@elevenlabs/client` if present                                                 |
-| `agents/nextjs/guardrails`             | `templates/nextjs`     | `app/api/agent/route.ts`, `app/api/conversation-token/route.ts`, `app/page.tsx` | Same as quickstart, but prompt targets guardrails and `onGuardrailTriggered`                                        |
+| Path                                   | Shared template        | Prompt sections                                                                                         | Setup extras                                                                                                        |
+| -------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
+| `text-to-speech/typescript/quickstart` | `templates/typescript` | `index.ts`                                                                                              | Copies `.env`, preserves `node_modules`, installs with `pnpm`                                                       |
+| `text-to-speech/python/quickstart`     | `templates/python`     | `main.py`                                                                                               | Copies `.env`, preserves `.venv`, installs with `pip`                                                               |
+| `speech-to-text/typescript/quickstart` | `templates/typescript` | `index.ts`                                                                                              | Optional `assets/`, copies `.env`, preserves `node_modules`                                                         |
+| `speech-to-text/python/quickstart`     | `templates/python`     | `main.py`                                                                                               | Optional `assets/`, copies `.env`, preserves `.venv`                                                                |
+| `music/typescript/quickstart`          | `templates/typescript` | `index.ts`                                                                                              | Copies `.env`, preserves `node_modules`, installs with `pnpm`                                                       |
+| `music/nextjs/quickstart`              | `templates/nextjs`     | `app/api/generate-music/route.ts`, `app/page.tsx`                                                       | Adds `@elevenlabs/elevenlabs-js`, copies `.env.local`, preserves `node_modules` and `.next`                         |
+| `speech-to-text/nextjs/realtime`       | `templates/nextjs`     | `app/api/scribe-token/route.ts`, `app/page.tsx`                                                         | Adds `@elevenlabs/react` and `@elevenlabs/elevenlabs-js`, copies `.env.local`, preserves `node_modules` and `.next` |
+| `agents/nextjs/quickstart`             | `templates/nextjs`     | `app/api/agent/route.ts`, `app/api/conversation-token/route.ts`, `app/page.tsx`                         | Same Next.js setup pattern, removes `@elevenlabs/client` if present                                                 |
+| `agents/nextjs/guardrails`             | `templates/nextjs`     | `app/api/agent/route.ts`, `app/api/conversation-token/route.ts`, `app/page.tsx`                         | Same as quickstart, but prompt targets guardrails and `onGuardrailTriggered`                                        |
+| `speech-engine/nextjs/quickstart`      | `templates/nextjs`     | `server.mts`, `scripts/create-engine.mts`, `app/api/token/route.ts`, `app/page.tsx`, `lib/assistant.ts` | Adds `openai`, `dotenv`, `tsx`; Speech Engine scripts; `livekit-client` override; copies `.env.example`             |
 
 ## Runtime setup rules
 
@@ -93,6 +95,7 @@ Ignore the deprecated root `examples/` folder for new work.
 - CLI transcription or file-based Scribe example: start from the speech-to-text quickstarts.
 - Realtime microphone UI: start from `speech-to-text/nextjs/realtime`.
 - Voice agent creation and conversation UI: start from `agents/nextjs/quickstart`.
+- Speech Engine WebSocket server plus Next.js WebRTC client: start from `speech-engine/nextjs/quickstart`.
 - First Expo full-stack example for a product: start from the closest same-product `nextjs` example until a dedicated Expo reference exists.
 - For specialized agent behavior, start from `agents/nextjs/quickstart` and consult `agents/nextjs/guardrails` only as an existing reference, not as a scaffold mode.
 

diff --git a/README.md b/README.md
@@ -24,6 +24,7 @@ Shared base templates live in `templates/` (Expo, Next.js, Python, TypeScript).
 - [Sound Effects Quickstart (Python)](sound-effects/python/quickstart/example/README.md) — Generate a sound effect MP3 from a text prompt with the ElevenLabs Python SDK.
 - [Sound Effects Playground (Next.js)](sound-effects/nextjs/quickstart/example/README.md) — Enter a prompt, generate a sound effect, and play it back in the browser.
 - [Real-Time Speech-to-Text (Next.js)](speech-to-text/nextjs/realtime/example/README.md) — Live microphone transcription with VAD in a Next.js app.
+- [Speech Engine Quickstart (Next.js)](speech-engine/nextjs/quickstart/example/README.md) — Add voice to your own LLM-backed chat agent with a Speech Engine WebSocket server and Next.js client.
 - [Real-Time Voice Agent (Next.js)](agents/nextjs/quickstart/example/README.md) — Live voice conversations with the ElevenLabs Agents Platform using the React Agents SDK.
 - [Voice Agent Guardrails Demo (Next.js)](agents/nextjs/guardrails/example/README.md) — Demonstrate custom guardrails and the `guardrail_triggered` client event in a live voice agent.
 - [Voice Isolator (Next.js)](voice-isolator/nextjs/quickstart/example/README.md) — Record your voice in the browser and remove background noise with the Voice Isolator API.

diff --git a/speech-engine/nextjs/quickstart/PROMPT.md b/speech-engine/nextjs/quickstart/PROMPT.md
@@ -0,0 +1,83 @@
+Before writing any code, invoke the `/speech-engine` skill to learn the correct ElevenLabs SDK patterns.
+
+## 1. `package.json`
+
+- Add `@elevenlabs/react`, `@elevenlabs/elevenlabs-js`, `openai`, `dotenv`, and `tsx`.
+- Add scripts: `speech-engine:create` (`tsx scripts/create-engine.mts`), `speech-engine:enable-first-message` (`tsx scripts/enable-first-message.mts`), `speech-engine:server` (`tsx server.mts`).
+- Pin `livekit-client` to `2.16.1` under `pnpm.overrides` for WebRTC stability.
+
+## 2. `.env.example`
+
+- Document `ELEVENLABS_API_KEY`, `ELEVENLABS_SPEECH_ENGINE_ID`, `OPENAI_API_KEY`, and `PUBLIC_WS_URL` (`wss://…/ws` from ngrok).
+
+## 3. `scripts/create-engine.mts`
+
+- Load env with `dotenv/config`.
+- Create a Speech Engine with `elevenlabs.speechEngine.create`, using `speechEngine.wsUrl` from `PUBLIC_WS_URL`.
+- Call a shared helper to enable `overrides.firstMessage` on the new resource.
+- Print `engineId` and next-step instructions.
+
+## 4. `scripts/enable-first-message.mts`
+
+- Update an existing Speech Engine (`ELEVENLABS_SPEECH_ENGINE_ID`) so the client may set `overrides.agent.firstMessage`.
+
+## 5. `lib/speech-engine-overrides.ts`
+
+- Export `enableFirstMessageOverride(client, speechEngineId)` using `speechEngine.update` with `overrides: { firstMessage: true }`.
+
+## 6. `lib/assistant.ts`
+
+- Shared OpenAI Responses API helpers: `ASSISTANT_INSTRUCTIONS`, `VOICE_FIRST_MESSAGE`, `isChatRole`, `normalizeChatMessages`, `transcriptToChatMessages`, `createAssistantReply(messages, signal)`, and `createAssistantStream(messages, signal)`.
+- Map Speech Engine `agent` role to OpenAI `assistant`; keep responses concise.
+- Normalize history before sending it to OpenAI: trim content, drop empty messages, cap message count and per-message length.
+
+## 7. `lib/voice-history.ts`
+
+- Add a tiny local store that lets the Next.js app hand typed chat history to the standalone Speech Engine server.
+- Store normalized `ChatMessage[]` in `.next/cache/voice-history.json` by random `historyId`.
+- Export `createVoiceHistory(messages)`, `linkVoiceHistory(conversationId, historyId)`, and `loadVoiceHistory(conversationId)`.
+- Keep this demo-only and file-backed; no database or auth layer.
+
+## 8. `server.mts`
+
+- Standalone HTTP server on port **3001** with `speechEngine.attach(SPEECH_ENGINE_ID, httpServer, "/ws", { debug: true, … })`.
+- `onTranscript`: load any typed chat history linked to `session.conversationId`, append `transcriptToChatMessages(transcript)`, pass `AbortSignal` into `createAssistantStream`, and `session.sendResponse(stream)`.
+- Cache loaded initial history per conversation and retry briefly because the browser may link history immediately after the voice conversation is created.
+- Log `onInit`, `onClose`, and `onError`. Require `ELEVENLABS_SPEECH_ENGINE_ID` and `OPENAI_API_KEY`.
+
+## 9. `app/api/chat/route.ts`
+
+- Secure POST route for regular typed chat.
+- Accept `{ messages }`, validate each message as `{ role: "user" | "assistant", content: string }`, cap request history, and call `createAssistantReply(messages, request.signal)`.
+- Return `{ message }`; return JSON errors for invalid payloads or OpenAI failures.
+- This route and voice mode must use the same `ChatMessage` shape so context can move between modes.
+
+## 10. `app/api/voice-history/route.ts`
+
+- POST route that accepts the current typed chat history before voice mode starts.
+- Validate and normalize messages with `isChatRole`, call `createVoiceHistory(messages)`, and return `{ historyId }`.
+- Return a 400 if no valid messages are provided.
+
+## 11. `app/api/voice-history/link/route.ts`
+
+- POST route that accepts `{ historyId, conversationId }` from the browser once the Speech Engine client creates a voice conversation.
+- Call `linkVoiceHistory(conversationId, historyId)` and return `{ ok: true }`, or 404 if the history id is unknown.
+
+## 12. `app/api/token/route.ts`
+
+- Secure GET route; never expose `ELEVENLABS_API_KEY` to the client.
+- Return `{ token }` from `conversationalAi.conversations.getWebrtcToken({ agentId: ELEVENLABS_SPEECH_ENGINE_ID })`.
+- Handle missing env and API errors with JSON error responses.
+
+## 13. `app/page.tsx`
+
+- Build a single chat UI that works like a normal assistant chat first, then can switch into voice mode without losing context.
+- Keep one `messages` state array with `{ id, role: "user" | "assistant", content, channel: "chat" | "voice", pending?, error? }`.
+- Text submit path: append the user message, POST the full normalized history to `/api/chat`, replace the pending assistant message with the reply, and keep the composer usable.
+- Voice start path: request microphone access, POST current non-pending/non-error chat history to `/api/voice-history`, fetch `/api/token`, then `startSession({ conversationToken: token, overrides: { agent: { firstMessage: VOICE_FIRST_MESSAGE } }, onConversationCreated })`.
+- In `onConversationCreated`, link the stored `historyId` to `voiceConversation.getId()` with `/api/voice-history/link`; if useful, also send a contextual update summarizing recent typed chat.
+- Voice event path: use `useConversation({ onMessage })` to append or update transcript messages in the same `messages` array, mapping SDK role `agent` to local role `assistant`.
+- While voice is connected, allow typed messages too: append them to the same thread and call `conversation.sendUserMessage(content)`; avoid duplicating echoes from `onMessage`.
+- The next typed chat after voice mode must include voice transcript messages in the history sent to `/api/chat`.
+- Show a scrollable message thread, a text composer, a mic/start voice control, a stop voice control, connection status, mute/unmute when connected, and separate chat/voice error messages.
+- Follow `DESIGN.md`; make the page feel like a regular chat app with voice as a mode, not a voice-only demo.
diff --git a/speech-engine/nextjs/quickstart/README.md b/speech-engine/nextjs/quickstart/README.md
@@ -0,0 +1,56 @@
+# Speech Engine Quickstart (Next.js)
+
+Add real-time voice to your own LLM-backed agent: a Speech Engine WebSocket server streams OpenAI responses to ElevenLabs, and a Next.js client starts WebRTC voice sessions.
+
+## Setup
+
+1. Copy the environment file and add your credentials:
+
+   ```bash
+   cp .env.example .env
+   ```
+
+   Then edit `.env` and set:
+   - `ELEVENLABS_API_KEY` — [create one in the dashboard](https://elevenlabs.io/app/settings/api-keys)
+   - `OPENAI_API_KEY` — for the assistant LLM in the Speech Engine server
+   - `PUBLIC_WS_URL` — public `wss://` URL for your Speech Engine WebSocket (see step 2)
+
+2. Expose port **3001** with [ngrok](https://ngrok.com/) (run this before creating the Speech Engine resource):
+
+   ```bash
+   ngrok http 3001
+   ```
+
+   Set `PUBLIC_WS_URL` to your forwarding URL with `/ws` appended, for example `wss://abc123.ngrok-free.app/ws`.
+
+3. Install dependencies:
+
+   ```bash
+   pnpm install
+   ```
+
+4. Create the Speech Engine resource and enable client first-message overrides:
+
+   ```bash
+   pnpm run speech-engine:create
+   pnpm run speech-engine:enable-first-message
+   ```
+
+   Copy the printed Speech Engine ID into `.env` as `ELEVENLABS_SPEECH_ENGINE_ID`.
+
+## Run
+
+Three processes must run together:
+
+1. **ngrok** — `ngrok http 3001`
+2. **Speech Engine server** — `pnpm run speech-engine:server` (port 3001)
+3. **Next.js app** — `pnpm run dev` (port 3000)
+
+Open [http://localhost:3000](http://localhost:3000) in your browser.
+
+## Usage
+
+- Click **Start conversation** and allow microphone access when prompted.
+- Speak naturally; the agent responds with streamed speech. With `debug: true` on the server, transcripts and responses log to the terminal.
+- Click **End conversation** to stop the session.
+- The agent greets first using `overrides.agent.firstMessage` (requires `speech-engine:enable-first-message` once per Speech Engine ID).
diff --git a/speech-engine/nextjs/quickstart/example/.env.example b/speech-engine/nextjs/quickstart/example/.env.example
@@ -0,0 +1,12 @@
+# Server-only ElevenLabs API key.
+ELEVENLABS_API_KEY=
+
+# Speech Engine ID returned by `pnpm run speech-engine:create`.
+ELEVENLABS_SPEECH_ENGINE_ID=
+
+# Server-only OpenAI API key for assistant responses.
+OPENAI_API_KEY=
+
+# Public WebSocket URL, usually from ngrok, including `/ws`.
+# Example: wss://your-subdomain.ngrok.app/ws
+PUBLIC_WS_URL=
diff --git a/speech-engine/nextjs/quickstart/example/README.md b/speech-engine/nextjs/quickstart/example/README.md
@@ -0,0 +1,56 @@
+# Speech Engine Quickstart (Next.js)
+
+Add real-time voice to your own LLM-backed agent: a Speech Engine WebSocket server streams OpenAI responses to ElevenLabs, and a Next.js client starts WebRTC voice sessions.
+
+## Setup
+
+1. Copy the environment file and add your credentials:
+
+   ```bash
+   cp .env.example .env
+   ```
+
+   Then edit `.env` and set:
+   - `ELEVENLABS_API_KEY` — [create one in the dashboard](https://elevenlabs.io/app/settings/api-keys)
+   - `OPENAI_API_KEY` — for the assistant LLM in the Speech Engine server
+   - `PUBLIC_WS_URL` — public `wss://` URL for your Speech Engine WebSocket (see step 2)
+
+2. Expose port **3001** with [ngrok](https://ngrok.com/) (run this before creating the Speech Engine resource):
+
+   ```bash
+   ngrok http 3001
+   ```
+
+   Set `PUBLIC_WS_URL` to your forwarding URL with `/ws` appended, for example `wss://abc123.ngrok-free.app/ws`.
+
+3. Install dependencies:
+
+   ```bash
+   pnpm install
+   ```
+
+4. Create the Speech Engine resource and enable client first-message overrides:
+
+   ```bash
+   pnpm run speech-engine:create
+   pnpm run speech-engine:enable-first-message
+   ```
+
+   Copy the printed Speech Engine ID into `.env` as `ELEVENLABS_SPEECH_ENGINE_ID`.
+
+## Run
+
+Three processes must run together:
+
+1. **ngrok** — `ngrok http 3001`
+2. **Speech Engine server** — `pnpm run speech-engine:server` (port 3001)
+3. **Next.js app** — `pnpm run dev` (port 3000)
+
+Open [http://localhost:3000](http://localhost:3000) in your browser.
+
+## Usage
+
+- Click **Start conversation** and allow microphone access when prompted.
+- Speak naturally; the agent responds with streamed speech. With `debug: true` on the server, transcripts and responses log to the terminal.
+- Click **End conversation** to stop the session.
+- The agent greets first using `overrides.agent.firstMessage` (requires `speech-engine:enable-first-message` once per Speech Engine ID).