refactor proof of concept

.env.example

@@ -529,6 +529,10 @@ EVA_EN_USER_M=your_elevenlabs_agent_id_for_default_user_m
 #d csv_list
 #v EVA_RECORD_IDS=
 
+#i Comma-separated record IDs to skip (applied after EVA_RECORD_IDS). Empty = skip none.
+#d csv_list
+#v EVA_EXCLUDE_RECORD_IDS=
+
 #i Logging verbosity.
 #d enum
 #e DEBUG,INFO,WARNING,ERROR,CRITICAL

README.md

@@ -97,6 +97,7 @@ EVA_DOMAIN=airline
 EVA_MAX_CONCURRENT_CONVERSATIONS=5
 EVA_DEBUG=false                       # Run only 1 record for testing when enabled
 EVA_RECORD_IDS=1.2.1,1.2.2            # Run specific records only (remove to run all records)
+EVA_EXCLUDE_RECORD_IDS=6.1.1          # Skip specific records (applied after EVA_RECORD_IDS)
 
 # User Simulator Configuration
 EVA_USER_SIMULATOR__PROVIDER=elevenlabs      # elevenlabs | openai_realtime

Readme.google.md

@@ -0,0 +1,15 @@
+## Evaluate all scenarios and get overall scores
+
+When running against AIS:
+
+```
+GOOGLE_GENAI_USE_VERTEXAI=0 python main.py --debug
+```
+
+When running against Vertex:
+
+```
+GOOGLE_GENAI_USE_VERTEXAI=1 python main.py --debug
+```
+
+Always use `--debug` to test as it executes a single case run only.

apps/analysis.py

@@ -78,12 +78,19 @@ def _build_metric_group_map() -> dict[str, str]:
     "Other": "#AAAAAA",
 }
 
-_NON_NORMALIZED_METRICS = {"response_speed", "tool_call_validity__num_tool_calls"}
+_NON_NORMALIZED_METRICS = {
+    "response_speed",
+    "tool_call_validity__num_tool_calls",
+    "time_to_completion",
+    "turns_to_completion",
+}
 
 # Axis title + hover suffix for non-normalized metrics. Sub-metrics fall back to their parent's entry.
 _NON_NORMALIZED_UNITS: dict[str, tuple[str, str]] = {
     "response_speed": ("Seconds", "s"),
     "tool_call_validity__num_tool_calls": ("Count", ""),
+    "time_to_completion": ("Seconds", "s"),
+    "turns_to_completion": ("Turns", ""),
 }
 
 

docker_build.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+TAG="${1:-latest}"
+PUSH="${2:-}"
+shift 2 2>/dev/null || true
+
+GIT_COMMIT_SHA=$(git rev-parse HEAD)
+GIT_BRANCH=$(git branch --show-current)
+GIT_DIRTY=$([[ -n $(git status --porcelain) ]] && echo true || echo false)
+GIT_DIFF_HASH=$(git diff | shasum -a 256 | cut -c1-12)
+
+IMAGE="registry.console.elementai.com/snow.core_llm/eva:$TAG"
+
+# Use BuildKit — the legacy builder ("Step X/Y" output) does not cache
+# multi-stage builds well and re-runs apt-get / re-copies layers needlessly.
+export DOCKER_BUILDKIT=1
+
+docker build \
+  --platform linux/amd64 \
+  --build-arg GIT_COMMIT_SHA="$GIT_COMMIT_SHA" \
+  --build-arg GIT_BRANCH="$GIT_BRANCH" \
+  --build-arg GIT_DIRTY="$GIT_DIRTY" \
+  --build-arg GIT_DIFF_HASH="$GIT_DIFF_HASH" \
+  -t "$IMAGE" \
+  "$@" .
+
+if [[ "$PUSH" == "--push" ]]; then
+  docker push "$IMAGE"
+fi

docs/metrics/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The EVA metrics system provides comprehensive evaluation of voice assistant conversations. The system includes 15 metrics organized into four categories, each answering a different question about the conversation.
+The EVA metrics system provides comprehensive evaluation of voice assistant conversations. The system includes 17 metrics organized into four categories, each answering a different question about the conversation.
 
 ## Why These Categories?
 
@@ -48,7 +48,7 @@ Measures the quality of the user's conversational experience:
 | [`conciseness`](conciseness.md) | Judge | Language Model | Whether responses are appropriately concise for voice (1-3) |
 | [`conversation_progression`](conversation_progression.md) | Judge | Language Model | Whether assistant moves conversation forward without repetition (1-3) |
 
-### Diagnostic (7 metrics)
+### Diagnostic (9 metrics)
 
 Metrics that help isolate root causes of failures. These provide signals for understanding what went wrong, but are not directly used in final evaluation scores.
 
@@ -57,6 +57,8 @@ Metrics that help isolate root causes of failures. These provide signals for und
 | [`tts_fidelity`](tts_fidelity.md) | Audio Judge (Gemini) | Speech Synthesis | Whether assistant speech audio matches intended text (0-1). **Opt-in** — excluded from the default run; enable via `--metrics tts_fidelity`. |
 | [`authentication_success`](authentication_success.md) | Deterministic | Speech Recognition, Language Model | Whether get_reservation was called successfully (0-1) |
 | [`response_speed`](response_speed.md) | Deterministic | VAD, Pipeline | Latency between user utterance end and assistant response start (seconds) |
+| [`time_to_completion`](time_to_completion.md) | Deterministic | Pipeline | Wall-clock time to complete the task, successful runs only (seconds, lower is better) |
+| [`turns_to_completion`](turns_to_completion.md) | Deterministic | Language Model | Number of turns to complete the task, successful runs only (count, lower is better) |
 | [`speakability`](speakability.md) | Judge | Language Model | Whether text is voice-friendly and appropriate for TTS (0-1) |
 | [`stt_wer`](stt_wer.md) | Deterministic | Speech Recognition | Speech-to-Text Word Error Rate using jiwer (0.0+) |
 | [`tool_call_validity`](tool_call_validity.md) | Deterministic | Language Model | Fraction of tool calls with correctly formatted parameters (0.0-1.0) |

docs/metrics/time_to_completion.md

@@ -0,0 +1,79 @@
+# Time to Completion
+
+> **Diagnostic Metric**: How long did a successful conversation take? Reported only for conversations that completed the task, so it answers "when the agent succeeds, how long does it take?" — not directly used in final pass/fail scores.
+
+## Overview
+
+Deterministic metric that reports the total wall-clock duration (in seconds) of a conversation, but **only for conversations where the task was actually completed**. Mixing in the durations of failed conversations would make the value meaningless as an efficiency signal, so unsuccessful conversations are skipped.
+
+Task completion is determined using the exact same criteria as [`task_completion`](task_completion.md): the session must be authenticated correctly and the final scenario database state must match the expected state (SHA-256 hash comparison).
+
+### Capabilities Measured
+
+- **Pipeline**: End-to-end wall-clock efficiency of the full system in reaching the correct outcome. Not attributable to a single model capability.
+
+## How It Works
+
+### Evaluation Method
+
+- **Type**: Deterministic (reads `duration_seconds`)
+- **Granularity**: Conversation-level
+
+### Input Data
+
+Uses the following MetricContext fields:
+- `expected_scenario_db`, `final_scenario_db`, `final_scenario_db_hash`: used to determine whether the task was completed (same as `task_completion`).
+- `duration_seconds`: total conversation duration.
+
+### Scoring
+
+- **Scale**: Seconds (lower is better)
+- **Normalization**: None. Raw duration in seconds is not meaningfully normalizable to a 0-1 scale.
+- **Skipped when**: the task was not completed, or no valid duration was recorded. Skipped records are excluded from the run-level efficiency aggregate.
+
+## Example Output
+
+```json
+{
+  "name": "time_to_completion",
+  "score": 42.5,
+  "normalized_score": null,
+  "details": {
+    "task_completed": true,
+    "duration_seconds": 42.5,
+    "reason": "Task completed — reporting total conversation duration"
+  }
+}
+```
+
+When the task was not completed:
+
+```json
+{
+  "name": "time_to_completion",
+  "score": null,
+  "normalized_score": null,
+  "skipped": true,
+  "details": {
+    "task_completed": false,
+    "reason": "Final database state differs from expected state"
+  }
+}
+```
+
+## Summary Aggregation
+
+The run-level `metrics_summary.json` includes an `overall_scores.efficiency.time_to_completion` block with the mean/min/max duration across successful conversations, alongside the per-metric aggregate under `per_metric.time_to_completion`.
+
+## Related Metrics
+
+- [turns_to_completion.md](turns_to_completion.md) - Number of turns (rather than seconds) to complete the task
+- [task_completion.md](task_completion.md) - The binary completion check that gates this metric
+- [response_speed.md](response_speed.md) - Per-turn response latency (not total conversation time)
+
+## Implementation Details
+
+- **File**: `src/eva/metrics/diagnostic/time_to_completion.py`
+- **Class**: `TimeToCompletionMetric`
+- **Base Class**: `CodeMetric`
+- **Configuration**: None (deterministic computation)

docs/metrics/turns_to_completion.md

@@ -0,0 +1,78 @@
+# Turns to Completion
+
+> **Diagnostic Metric**: How many turns did a successful conversation take? Reported only for conversations that completed the task, so it answers "when the agent succeeds, how many turns does it take?" — not directly used in final pass/fail scores.
+
+## Overview
+
+Deterministic metric that reports the total number of conversation turns, but **only for conversations where the task was actually completed**. Mixing in the turn counts of failed conversations would make the value meaningless as an efficiency signal, so unsuccessful conversations are skipped.
+
+Task completion is determined using the exact same criteria as [`task_completion`](task_completion.md): the session must be authenticated correctly and the final scenario database state must match the expected state (SHA-256 hash comparison).
+
+### Capabilities Measured
+
+- **Language Model**: How efficiently the agent drives the conversation to the correct outcome (fewer back-and-forth turns for the same result is better).
+
+## How It Works
+
+### Evaluation Method
+
+- **Type**: Deterministic (reads `num_turns`)
+- **Granularity**: Conversation-level
+
+### Input Data
+
+Uses the following MetricContext fields:
+- `expected_scenario_db`, `final_scenario_db`, `final_scenario_db_hash`: used to determine whether the task was completed (same as `task_completion`).
+- `num_turns`: total number of conversation turns.
+
+### Scoring
+
+- **Scale**: Turn count (lower is better)
+- **Normalization**: None. Raw turn count is not meaningfully normalizable to a 0-1 scale.
+- **Skipped when**: the task was not completed, or no valid turn count was recorded. Skipped records are excluded from the run-level efficiency aggregate.
+
+## Example Output
+
+```json
+{
+  "name": "turns_to_completion",
+  "score": 8.0,
+  "normalized_score": null,
+  "details": {
+    "task_completed": true,
+    "num_turns": 8,
+    "reason": "Task completed — reporting total conversation turns"
+  }
+}
+```
+
+When the task was not completed:
+
+```json
+{
+  "name": "turns_to_completion",
+  "score": null,
+  "normalized_score": null,
+  "skipped": true,
+  "details": {
+    "task_completed": false,
+    "reason": "Authentication failed — session mismatch on keys: ['user_id']"
+  }
+}
+```
+
+## Summary Aggregation
+
+The run-level `metrics_summary.json` includes an `overall_scores.efficiency.turns_to_completion` block with the mean/min/max turn count across successful conversations, alongside the per-metric aggregate under `per_metric.turns_to_completion`.
+
+## Related Metrics
+
+- [time_to_completion.md](time_to_completion.md) - Wall-clock time (rather than turns) to complete the task
+- [task_completion.md](task_completion.md) - The binary completion check that gates this metric
+
+## Implementation Details
+
+- **File**: `src/eva/metrics/diagnostic/turns_to_completion.py`
+- **Class**: `TurnsToCompletionMetric`
+- **Base Class**: `CodeMetric`
+- **Configuration**: None (deterministic computation)