Build a library of scorers

[mattpocock]

@cantemizyurek from open-evals has expressed an interest in helping out and creating a library of scorers inside Evalite. Let's use this issue to discuss scope.

Related: #26

[cantemizyurek]

I plan to start by implementing the following Scorers as the initial target:

• Faithfulness: Evaluates how well the model’s response is grounded in the provided context.
• Answer Similarity: Measures the semantic similarity between the ground-truth and generated responses.
• Context Recall: Estimates how much of the reference answer can be attributed to the retrieved contexts, assessing recall quality.

These scorers will help evaluate how effectively the RAG pipeline performs. Once these (and possibly additional RAG scorers) are in place, the next step could involve integrating SDG-based test data generation, or extending the evaluation to agent-level scorers such as Topic Adherence and Tool Call Accuracy.

There will be a type called EvaluationSample with two variants:

• SingleTurnSample
• MultiTurnSample

@mattpocock how does this initial scope sound to you?

[mattpocock]

@cantemizyurek Initial set of scorers sound great.

What's the idea behind the single-turn and multi-turn samples?

[cantemizyurek]

@mattpocock The difference between Single Turn and Multi Turn samples mainly comes down to how the evaluation data is structured and what you want to measure.

Single Turn samples are straightforward Q&A pairs, which makes them great for RAG evaluation. Each one contains an isolated input, a reference answer, and some retrieved context. This works well for metrics that evaluate an LLM's response to a single question.

Multi Turn samples represent an ongoing conversation between a user and an LLM. This setup is important for agent evaluations, where the context builds up over multiple interactions or tool calls. Multi-turn data lets you test more realistic scenarios that reflect how agents actually behave in conversations (in a more concise manner it works on ModelMessages).

Not every metric works with both formats. Some are built specifically for single-turn data and can’t handle the context dependencies in multi-turn interactions, and vice versa.

RAG metrics like Faithfulness and Context Recall are usually single-turn.

Agentic metrics like Tool Call Accuracy, Agent Goal Accuracy need multi-turn data, since agent evaluation often involves multi-step reasoning or conversations.

[mattpocock]

@cantemizyurek Makes sense!

Were you thinking that most scorers would use 'messages' (either single or multi-turn) as an input? This makes sense, just want to check.

If so, couldn't we simply pass an array of ModelMessage[] to the evalite input? Keen not to introduce extra jargon (multi-turn vs single-turn) wherever possible.

[cantemizyurek]

@mattpocock I’d love to simplify this approach, but with that direction, how would we distinguish between the two sample types (were you thinking of differentiating by array length)?

Also, where would we store additional data like the retrieved contexts (the contexts used to generate the user query) and the “golden” answer if everything moves into the input as a ModelMessage[] maybe in a metadata?

Here’s roughly what single-turn and multi-turn samples usually look like:

interface SingleTurnSample {
query: string;
retrievedContexts?: string[];
reference?: string;
}

type MultiTurnSample = {
messages: ModelMessage[];
reference?: ModelMessage[];
retrievedContexts?: string[];
}

Would love to find a simpler, more unified way to represent both if possible.

[mattpocock]

@cantemizyurek I tell you what, I probably won't know the tradeoffs until I've played around with it a bit. I reckon put up a POC together of what you have so far and we can work from there. Maybe as a PR to #240?

[cantemizyurek]

@mattpocock Yep sure that sounds great I will make a pr to #240 with the first few metrics and we can test and try to improve it from there.

[cantemizyurek]

@mattpocock I’ve created the PR for #251. I also added Zod since it was needed inside generateObject.
I wasn’t sure which version to use, so I updated it to v4. If you prefer, I can downgrade it back to v3.

[mattpocock]

@cantemizyurek for internal stuff, I'd prefer to drop the Zod dependency and just use JSON schema. The AI SDK is moving towards using Standard Schema, so having Zod as a dependency of Evalite feels a bit awkward.

https://ai-sdk.dev/docs/reference/ai-sdk-core/json-schema