This repository was archived by the owner on Jun 17, 2026. It is now read-only.
Scaffold behavioral evals in 'pc init' - #96
Closed
jamsea wants to merge 4 commits into
Closed
Conversation
For cascade bots on standard transports, 'pc init' can now include Pipecat's behavioral eval framework (pipecat-ai/pipecat#4655): - New 'enable_evals' feature: wizard question (default yes when eligible), '--evals/--no-evals' flag, and an 'evals' key in --config JSON. Realtime mode and the Daily PSTN / Twilio+Daily SIP flows reject it with a validation error. - The generated bot exposes an 'eval' transport entry, so 'uv run bot.py -t eval' works. - New 'server/evals/scenario.yaml': a text-mode smoke scenario. The judge follows the project LLM (openai when the bot uses OpenAI, otherwise local Ollama with a commented openai alternative). - The pipecat-ai dependency gains the 'cli' extra so 'uv run pipecat eval run evals/scenario.yaml' works. - README gains a 'Testing your bot (evals)' section. Requires the pipecat release after 1.3.0.
There was a problem hiding this comment.
Pull request overview
Adds opt-in scaffolding for Pipecat behavioral evals during pc init project generation, including generated scenario YAML, bot transport wiring, dependency extras, README guidance, and validation/test coverage.
Changes:
- Introduces an
enable_evalsfeature flag across interactive prompts, non-interactive config/CLI, validation, and project generation. - Adds templates for
server/evals/scenario.yamlplus an evals README block and bot-t evaltransport exposure. - Extends registry/import generation and updates tests to cover evals scenarios and validation errors.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_project_generation.py | Adds generation assertions for evals scaffolding (scenario YAML, bot transport, pyproject extras, README section). |
| tests/test_config_validator.py | Adds validator tests for rejecting evals in unsupported mode/transports and JSON field inclusion. |
| src/pipecat_cli/templates/server/evals/scenario.yaml.jinja2 | New scenario template for behavioral eval harness with OpenAI/Ollama judge selection. |
| src/pipecat_cli/templates/server/bot_cascade.py.jinja2 | Adds optional "eval" transport_params entry using WebsocketServerParams when evals enabled. |
| src/pipecat_cli/templates/README.md.jinja2 | Conditionally includes an evals README block when evals enabled. |
| src/pipecat_cli/templates/_readme_blocks/evals.jinja2 | New README block documenting how to run evals. |
| src/pipecat_cli/registry/service_metadata.py | Adds evals feature definition for import generation. |
| src/pipecat_cli/registry/service_loader.py | Includes evals feature imports when enabled. |
| src/pipecat_cli/registry/_imports.py | Adds generated import for WebsocketServerParams under the evals feature. |
| src/pipecat_cli/prompts/questions.py | Adds evals eligibility logic, config flag, and interactive prompt wiring. |
| src/pipecat_cli/generators/project.py | Generates evals files, injects feature flags, adds cli extra, and prints next-steps eval commands. |
| src/pipecat_cli/config_validator.py | Validates --evals constraints and includes enable_evals in built config and JSON. |
| src/pipecat_cli/commands/init.py | Adds --evals/--no-evals, config-file merging, and passes evals into validation. |
| scripts/imports/import_generator.py | Adds explicit module mapping for WebsocketServerParams for deterministic import regeneration. |
| CHANGELOG.md | Documents the new pc init eval scaffolding capability and release coupling note. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Addresses Copilot review feedback: with a plain bool default, an explicit --no-evals could not override "evals": true in a --config file (the or-merge always preferred the file). The flag now defaults to None, the file value only applies when the flag is omitted, and the resolved value is coerced to bool before validation. Adds CLI-level tests covering flag-vs-file precedence.
Contributor
|
The CLI is moving into Pipecat and this repo and package are being deprecated. We can add the evals to it, but let's follow this commit exactly: Mind reworking this to align? For example, let's remove |
Author
|
@markbackman thanks for the info I'll rework this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds opt-in scaffolding for Pipecat's new behavioral eval framework (pipecat-ai/pipecat#4655) when creating a project.
For cascade bots on standard transports (daily, smallwebrtc, websocket, twilio, telnyx, plivo, exotel):
--evals/--no-evalsor an"evals"key in--configJSON."eval"entry intransport_params, souv run bot.py -t evalworks.server/evals/scenario.yaml: a text-mode smoke scenario (greeting check + a question graded by an LLM judge). The judge follows the project LLM:openaiwhen the bot uses OpenAI (key is already in .env), otherwise local Ollama with a commented OpenAI alternative.cliextra, souv run pipecat eval run evals/scenario.yamlworks.Realtime mode and the Daily PSTN / Twilio+Daily SIP special flows reject
--evalswith a validation error (they don't go through the unifiedcreate_transportpath).The eval framework merged into pipecat main on June 11 but is not in a release yet (latest is v1.3.0). Until the next pipecat release:
-t evaldoesn't existKeep this as a draft until that release ships.
Testing
uv run pytest -m "not slow": 448 passed (new generation configs for both judge variants, plus 4 new validator tests)--no-evalsbaseline (no eval traces in bot.py/pyproject/README), realtime rejection,--dry-runJSON includesenable_evalsuv run scripts/update_registry.py(diff is the single newevalsentry inFEATURE_IMPORTS)🤖 Generated with Claude Code