v0.4.0: Tool result assertions

agentverify 0.4.0 adds Tool Result Assertions: knowing the agent called the right tool is not the same as knowing the call worked. A tool can return a 5xx, time out, or hand back an empty body, and the LLM will often smooth that over in its final reply. These assertions check the result a tool handed back, not just which tool was called and with what arguments.

Highlights

• Tool result assertions: assert_tool_invocation_succeeded, assert_no_tool_errors, assert_tool_result_matches, and assert_retry_count verify that tool calls actually returned a usable result (vs. errored, timed out, or came back empty), what the result contained, and how many times a flaky tool was retried. See README "Tool Result Assertions".
• Per-result error signal on Step via the additive tool_results_meta field: populated by the built-in adapters from each framework's native tool-error convention, reconstructed on cassette replay, and settable from custom converters and step_probe. Three-valued (known error / known success / unknown) so live and replayed runs give the same verdict.

Bug Fixes

• Example test suites on Python 3.14: pytest examples/<name>/tests no longer fails with a duplicate --cassette-mode registration during collection.

---

Install:

pip install agentverify==0.4.0

Full changelog: CHANGELOG.md

v0.3.0: Step-level assertions and step-to-step data flow testing

agentverify 0.3.0 extends deterministic testing from flat tool-call sequences to the full step structure of multi-step agents: ReAct, Plan-and-Execute, probabilistic refinement loops, and workflow-style agents with cache / validation / conditional branches.

Highlights

• Step-level assertions: assert_step / assert_step_output / assert_step_uses_result_from verify tool calls, intermediate outputs, and step-to-step data flow on the new Step data model.
• Step-to-step data flow verification: assert_step_uses_result_from(step=N, depends_on=M) catches the "agent regenerated but ignored the previous step" bug. Works on cassette replay; tolerates numeric/string type coercion and multi-line string serialization.
• step_probe context manager: mark logical step boundaries in agent code (cache hits, validation, conditional branches) so tests can assert on non-LLM steps. Zero-cost no-op outside of test contexts. Safe to leave in production code.
• MATCHES(pattern) regex matcher: verify string tool-call arguments against a regex, with the same semantics as ANY.
• MockLLM + mock_response(...): replay predefined LLM responses in-memory. Test agent routing without a cassette or a real LLM call.
• assert_latency(result, max_ms=...): response-time SLAs. ExecutionResult.duration_ms is captured automatically by the cassette fixture and MockLLM.
• Three new example suites exercising step-level and data-flow testing on different agent shapes:
  • openai-agents-llm-as-a-judge: OpenAI Agents SDK LLM-as-a-Judge (probabilistic refinement loop with feedback-chain data flow)
  • langgraph-multi-agent-supervisor: LangGraph research + math handoff with numeric running-total data flow
  • custom-converter-python-agent: pure-Python Anthropic SDK ReAct with an ~80-line converter reference

Cassette adapter improvements

• OpenAI cassette adapter now also intercepts AsyncCompletions.create, so agent frameworks that drive the SDK through AsyncOpenAI internally (including the OpenAI Agents SDK) are recorded and replayed transparently.
• OpenAI cassette adapter strips openai.omit / openai.NOT_GIVEN sentinels from tools, per-message dicts, and extra parameters before they reach the cassette YAML.
• OpenAI cassette adapter handles the with_raw_response.create code path used by langchain-openai v1.x.
• Anthropic cassette adapter flattens SDK content-block objects to plain dicts at record time, so cassettes from ReAct-style agents load cleanly regardless of the installed Anthropic SDK version.

Dependency

• Minimum pytest>=7 is now declared in the runtime dependency. CI tests against pytest 7, 8, and 9 majors on Python 3.10-3.14.

Breaking changes

• ExecutionResult.to_dict() now emits steps: [...] instead of tool_calls: [...]. Read-side assertions and the ExecutionResult(tool_calls=[...]) constructor remain backward compatible; from_dict() still accepts the legacy tool_calls key on input.

---

Install:

pip install agentverify==0.3.0

Full changelog: CHANGELOG.md

v0.2.0: Built-in framework adapters, cassette matching, sanitization

Highlights

• Built-in framework adapters for Strands Agents, LangChain, LangGraph, and OpenAI Agents SDK. Extract ExecutionResult directly from agent framework outputs without writing a custom converter (from_strands, from_langchain, from_langgraph, from_openai_agents).
• Cassette request matching detects stale cassettes by verifying model name and tool names during replay. Enabled by default; raises CassetteRequestMismatchError with a clear diff on mismatch.
• Cassette sanitization automatically redacts API keys and sensitive data when recording. Built-in patterns cover OpenAI, Anthropic, AWS, and Bearer tokens; extendable with custom SanitizePattern objects.
• Strands Weather Forecaster example: end-to-end example testing the official Strands sample with a pre-recorded Bedrock cassette.

---

Install:

pip install agentverify==0.2.0

Full changelog: CHANGELOG.md

v0.1.0: Initial release

Initial release.

Features

• Tool call assertions: assert_tool_calls with EXACT, IN_ORDER, and ANY_ORDER modes; ANY wildcard and partial_args for flexible argument matching.
• Cost budget assertions: assert_cost enforces max_tokens and max_cost_usd limits.
• Safety guardrails: assert_no_tool_call detects forbidden tool invocations.
• Final output assertions: assert_final_output with contains, equals, and matches (regex).
• Batch assertions: assert_all collects all failures without stopping at the first.
• LLM Cassette Record & Replay: VCR-style recording of LLM API calls for deterministic CI testing. Human-readable YAML / JSON cassettes that you commit to git.
• 5 LLM provider adapters: OpenAI, Amazon Bedrock, Google Gemini, Anthropic, LiteLLM.
• pytest plugin: Auto-registers on install, provides the cassette fixture and @pytest.mark.agentverify marker.
• Structured error messages: Clear diffs with mismatch position highlighting.

---

Install:

pip install agentverify==0.1.0