Skip to content

feat: add Agent Studio test case resource support#165

Open
Ruari-Phipps wants to merge 9 commits into
mainfrom
ruari/feat/testing
Open

feat: add Agent Studio test case resource support#165
Ruari-Phipps wants to merge 9 commits into
mainfrom
ruari/feat/testing

Conversation

@Ruari-Phipps
Copy link
Copy Markdown
Collaborator

@Ruari-Phipps Ruari-Phipps commented May 29, 2026

Summary

Adds local ADK support for Agent Studio test cases, including YAML resources under test_suite/, protobuf sync commands, pull/push integration, documentation, and unit tests.

Motivation

Agent Studio now supports simulated conversation tests with prompt and function-call assertions. The ADK needs to read, write, validate, and sync these resources locally so projects can manage tests alongside other agent configuration.

Changes

  • Add TestCase, TestCaseAssertion, TestCaseTags, and function-call assertion types in src/poly/resources/test_suite.py
  • Register test cases in project resource mapping and sync client projection parsing
  • Update protobuf bindings (including new testing_pb2) and command types for create/update/delete/assertions/tags
  • Add tests.md documentation and link it from the main docs index
  • Add unit tests in resources_test.py and project discovery coverage in project_test.py
  • Add sample test cases to the test project (test_suite/greeting_flow_test.yaml, test_suite/webchat_smoke_test.yaml)
  • Set __test__ = False on resource classes to avoid pytest collection warnings
  • Validate test case language against configured project languages (default + additional)
  • Validate function call assertion names against known global functions

Test strategy

  • Added/updated unit tests
  • Manual CLI testing (poly <command>)
  • Tested against a live Agent Studio project
  • N/A (docs, config, or trivial change)

Checklist

  • ruff check . and ruff format --check . pass
  • pytest passes
  • No breaking changes to the poly CLI interface (or migration path documented)
  • Commit messages follow conventional commits

Screenshots / Logs

name: Test Weather flow
scenario: Ask for the weather for berlin.
channel: voice
language: es-ES
tags:
- weather
prompt_assertions:
- It gives the weather
- It gives the weather for Berlin
function_call_assertions:
- name: get_weather
  arguments:
  - parameter_name: city
    expected_value: Berlin
    value_type: string

@Ruari-Phipps Ruari-Phipps requested a review from a team May 29, 2026 12:17
@Ruari-Phipps Ruari-Phipps requested a review from a team as a code owner May 29, 2026 12:17
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Copy Markdown
Contributor

Coverage Report

Base (main) PR Change
69.6% 69.9% +0.3% ✅

Changed file coverage

File Coverage Change
poly/handlers/sync_client.py 24.5% -0.7% ⚠️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant