Evaluation Metrics

Metrics are a fundamental component of the Opik evaluation function. They provide quantitative assessments of your AI models' outputs, enabling objective comparisons and performance tracking over time.

What Are Metrics?

In Opik, a metric is a function that calculates a score based on specific inputs, such as model outputs and reference answers. All metrics in Opik extend the BaseMetric abstract class, which provides the core functionality for validation and tracking.

TypeScript

abstract class BaseMetric<GenericZodObjectType> {
  public readonly name: string;
  public readonly trackMetric: boolean;
  public abstract readonly validationSchema: GenericZodObjectType;

  abstract score(
    input: Infer<GenericZodObjectType>
  ):
    | EvaluationScoreResult
    | EvaluationScoreResult[]
    | Promise<EvaluationScoreResult>
    | Promise<EvaluationScoreResult[]>;
}

How Metrics Calculate Scores

Each metric must implement the score method, which:

Accepts an input object containing combined data from the task output, dataset item, and scoringKeyMapping
Processes the inputs to produce a score
Returns an EvaluationScoreResult or array of results, which includes:
- name: The metric name
- value: The numerical score (typically 0.0-1.0)
- reason: A human-readable explanation for the score

Types of Metrics

Opik supports different types of metrics:

Heuristic metrics: Simple rule-based evaluations (e.g., exact match, contains, regex match)
LLM Judge metrics: AI-powered evaluations that use language models to assess output quality

Built-in Metrics

Opik provides several built-in metrics for common evaluation scenarios:

ExactMatch

Checks if the model output exactly matches the expected output:

TypeScript

const exactMatch = new ExactMatch();
// Usage requires both 'output' and 'expected' parameters

Contains

Checks if the model output contains specific text:

TypeScript

const contains = new Contains();
// Usage requires both 'output' and 'expected' parameters

RegexMatch

Checks if the model output matches a regular expression pattern:

TypeScript

const regexMatch = new RegexMatch();
// Usage requires 'output' and 'pattern' parameters

IsJson

Checks if the output is valid JSON:

TypeScript

const isJson = new IsJson();
// Usage requires 'output' parameter

Metric Configuration

Custom Naming and Tracking

Each metric can be configured with a custom name and tracking option:

TypeScript

// Create metric with custom name
const exactMatch = new ExactMatch("my_exact_match");

// Create metric with tracking disabled
const regexMatch = new RegexMatch("custom_regex", false);

Combining Multiple Metrics

You can use multiple metrics in a single evaluation:

TypeScript

const metrics = [new ExactMatch(), new Contains(), new RegexMatch()];

// In your evaluation configuration
await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: metrics,
});

Input Requirements

Validation Schema

Each metric defines a Zod validation schema that specifies required inputs:

TypeScript

// ExactMatch validation schema example
const validationSchema = z.object({
  output: z.string(), // The model output
  expected: z.string(), // The expected output
});

The validation system ensures all required parameters are present before executing the metric.

Mapping Inputs

You can map dataset fields and task outputs to metric inputs using scoringKeyMapping:

TypeScript

await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [new ExactMatch()],
  scoringKeyMapping: {
    // Map dataset/task fields to metric parameter names
    output: "model.response",
    expected: "dataset.answer",
  },
});

Score Interpretation

Score Ranges

Most metrics in Opik return scores between 0.0 and 1.0:

1.0: Perfect match or ideal performance
0.0: No match or complete failure
Intermediate values: Partial matches or varying degrees of success

Creating Custom Metrics

Implementing Your Own Metric

To create a custom metric:

Extend the BaseMetric class
Define a validation schema using Zod
Implement the score method

Here's an example of a custom metric that checks if output length is within a specified range:

TypeScript

import z from "zod";
import { BaseMetric } from "@opik/sdk";
import { EvaluationScoreResult } from "@opik/sdk";

// Define validation schema
const validationSchema = z.object({
  output: z.string(),
  minLength: z.number(),
  maxLength: z.number(),
});

// Infer TypeScript type from schema
type Input = z.infer<typeof validationSchema>;

export class LengthRangeMetric extends BaseMetric {
  public validationSchema = validationSchema;

  constructor(name = "length_range", trackMetric = true) {
    super(name, trackMetric);
  }

  async score(input: Input): Promise<EvaluationScoreResult> {
    const { output, minLength, maxLength } = input;
    const length = output.length;

    // Calculate score (1.0 if within range, 0.0 otherwise)
    const isWithinRange = length >= minLength && length <= maxLength;
    const score = isWithinRange ? 1.0 : 0.0;

    // Return result with explanation
    return {
      name: this.name,
      value: score,
      reason: isWithinRange
        ? `Output length (${length}) is within range ${minLength}-${maxLength}`
        : `Output length (${length}) is outside range ${minLength}-${maxLength}`,
    };
  }
}

Validation Best Practices

When creating custom metrics:

Define clear validation schemas:

TypeScript

const validationSchema = z.object({
  output: z.string().min(1, "Output is required"),
  threshold: z.number().min(0).max(1),
});

Return meaningful reasons:

TypeScript

return {
  name: this.name,
  value: score,
  reason: `Score ${score.toFixed(2)} because [detailed explanation]`,
};

Normalize scores to a consistent range (typically 0.0-1.0) for easier comparison with other metrics

LLM Judge Metrics

LLM Judge metrics use language models to evaluate the quality of LLM outputs. These metrics provide more nuanced evaluation than simple heuristic checks.

AnswerRelevance

Evaluates how relevant the output is to the input question:

TypeScript

import { AnswerRelevance } from "opik";

// Using default model (gpt-5-nano)
const metric = new AnswerRelevance();

// With custom model ID
const metricWithModel = new AnswerRelevance({
  model: "claude-3-5-sonnet-latest",
});

// With LanguageModel instance
import { openai } from "@ai-sdk/openai";
const customModel = openai("gpt-5-nano");
const metricWithCustomModel = new AnswerRelevance({ model: customModel });

// Usage
const score = await metric.score({
  input: "What is the capital of France?",
  output: "The capital of France is Paris.",
  context: ["France is a country in Western Europe."], // Optional
});

console.log(score.value); // 0.0 to 1.0
console.log(score.reason); // Explanation of the score

Parameters

input (required): The question or prompt
output (required): The model's response to evaluate
context (optional): Additional context for evaluation

Score Range

1.0: Perfect relevance - output directly addresses the input
0.5: Partial relevance - output is somewhat related but incomplete
0.0: No relevance - output doesn't address the input

Hallucination

Detects whether the output contains hallucinated or unfaithful information:

TypeScript

import { Hallucination } from "opik";

const metric = new Hallucination();

// Without context - checks against general knowledge
const score1 = await metric.score({
  input: "What is the capital of France?",
  output:
    "The capital of France is Paris. It is famous for its iconic Eiffel Tower.",
});

// With context - checks faithfulness to provided context
const score2 = await metric.score({
  input: "What is the capital of France?",
  output:
    "The capital of France is Paris. It is famous for its iconic Eiffel Tower.",
  context: [
    "France is a country in Western Europe. Its capital is Paris, which is known for landmarks like the Eiffel Tower.",
  ],
});

console.log(score2.value); // 1.0 = hallucination detected, 0.0 = no hallucination
console.log(score2.reason); // Array of reasons for the score

Parameters

input (required): The original question or prompt
output (required): The model's response to evaluate
context (optional): Reference information to check against

Score Values

0.0: No hallucination - output is faithful to context/facts
1.0: Hallucination detected - output contains false or unsupported information

Moderation

Checks if the output contains harmful, inappropriate, or unsafe content:

TypeScript

import { Moderation } from "opik";

const metric = new Moderation();

const score = await metric.score({
  input: "Tell me about safety guidelines",
  output: "Here are some safety guidelines...",
});

console.log(score.value); // 1.0 = harmful content detected, 0.0 = safe
console.log(score.reason); // Explanation of moderation decision

Parameters

input (required): The original prompt
output (required): The model's response to evaluate

Score Values

0.0: Safe - no harmful content detected
1.0: Harmful - inappropriate or unsafe content detected

Usefulness

Evaluates how useful the output is in addressing the input:

TypeScript

import { Usefulness } from "opik";

const metric = new Usefulness();

const score = await metric.score({
  input: "How do I reset my password?",
  output:
    "To reset your password, click 'Forgot Password' on the login page, enter your email, and follow the instructions sent to your inbox.",
});

console.log(score.value); // 0.0 to 1.0
console.log(score.reason); // Explanation of usefulness score

Parameters

input (required): The question or request
output (required): The model's response to evaluate

Score Range

1.0: Very useful - comprehensive and actionable
0.5: Somewhat useful - partially helpful
0.0: Not useful - doesn't help address the input

GEval

GEval is a task-agnostic LLM-as-a-judge metric that allows you to define custom evaluation criteria. The metric first generates a chain-of-thought (CoT) evaluation plan, then scores the output on a 0-10 scale (normalized to 0.0-1.0).

TypeScript

import { GEval } from "opik";

const metric = new GEval({
  taskIntroduction: "You evaluate the politeness of customer service responses.",
  evaluationCriteria: "Score from 0 (rude) to 10 (very polite). Consider tone, word choice, and empathy.",
  model: "gpt-4o",
});

const score = await metric.score({
  output: "Thanks so much for your patience! I'm happy to help resolve this for you.",
});

console.log(score.value); // 0.0 to 1.0
console.log(score.reason); // Explanation of the score

Parameters

taskIntroduction (required): Description of what should be evaluated
evaluationCriteria (required): Detailed criteria defining what "good" looks like
model (optional): Model to use for evaluation (defaults to "gpt-4o")
name (optional): Custom metric name (defaults to "g_eval_metric")
temperature (optional): Sampling temperature for generation
seed (optional): Seed for reproducible outputs
maxTokens (optional): Maximum response length
modelSettings (optional): Advanced model configuration

Score Range

1.0: Perfect score (10/10 from the judge)
0.5: Average score (5/10 from the judge)
0.0: Lowest score (0/10 from the judge)

How It Works

GEval uses a two-stage process:

Chain of Thought Generation: Creates step-by-step evaluation instructions based on your task and criteria (cached for reuse)
Scoring: Evaluates the output using the CoT, returning a score with reasoning

When using OpenAI models, GEval leverages logprobs to compute a weighted average of score probabilities for more robust scoring.

Built-in GEval Judges

Opik provides pre-configured GEval judges for common evaluation scenarios. Each extends GEval with domain-specific prompts:

QARelevanceJudge

Evaluates whether an answer directly addresses the question:

TypeScript

import { QARelevanceJudge } from "opik";

const judge = new QARelevanceJudge({ model: "gpt-4o" });

const score = await judge.score({
  output: `QUESTION: What causes rainbows?
ANSWER: Rainbows are caused by refraction and reflection of light in water droplets.`,
});

console.log(score.value); // High score for relevant answer

SummarizationConsistencyJudge

Checks if a summary is faithful to the source material:

TypeScript

import { SummarizationConsistencyJudge } from "opik";

const judge = new SummarizationConsistencyJudge();

const score = await judge.score({
  output: `SOURCE: The company announced Q4 revenue of $2.5M.
SUMMARY: The company had strong Q4 performance with $2.5M revenue.`,
});

SummarizationCoherenceJudge

Evaluates the structure and clarity of summaries:

TypeScript

import { SummarizationCoherenceJudge } from "opik";

const judge = new SummarizationCoherenceJudge();

const score = await judge.score({
  output: "SUMMARY: First, the project started. Then it ended. Finally, it began.",
});

console.log(score.value); // Low score for incoherent summary

DialogueHelpfulnessJudge

Assesses how helpful an assistant reply is in dialogue context:

TypeScript

import { DialogueHelpfulnessJudge } from "opik";

const judge = new DialogueHelpfulnessJudge();

const transcript = `USER: How do I reset my password?
ASSISTANT: Visit settings and click reset.
USER: I cannot see that option.
ASSISTANT: Please contact support.`;

const score = await judge.score({ output: transcript });

Bias Detection Judges

Detect various forms of bias in responses:

TypeScript

import {
  DemographicBiasJudge,
  GenderBiasJudge,
  PoliticalBiasJudge,
  ReligiousBiasJudge,
  RegionalBiasJudge,
} from "opik";

// Demographic bias
const demographicJudge = new DemographicBiasJudge();
const score1 = await demographicJudge.score({
  output: "People from X group are always late.",
});

// Gender bias
const genderJudge = new GenderBiasJudge();
const score2 = await genderJudge.score({
  output: "Women are naturally worse at math.",
});

// Political bias
const politicalJudge = new PoliticalBiasJudge();
const score3 = await politicalJudge.score({
  output: "Vote for candidate X because Y is corrupt.",
});

Agent Evaluation Judges

Evaluate agent task completion and tool usage:

TypeScript

import { AgentTaskCompletionJudge, AgentToolCorrectnessJudge } from "opik";

// Task completion
const taskJudge = new AgentTaskCompletionJudge();
const score1 = await taskJudge.score({
  output: "Agent gathered quotes, compared options, and booked travel.",
});

// Tool correctness
const toolJudge = new AgentToolCorrectnessJudge();
const score2 = await toolJudge.score({
  output: "Tool weather_api called with city='Paris' but response ignored.",
});

PromptUncertaintyJudge

Estimates how ambiguous a prompt is:

TypeScript

import { PromptUncertaintyJudge } from "opik";

const judge = new PromptUncertaintyJudge();

const score = await judge.score({
  output: "Summarise the attached 400 page contract in one sentence and guarantee there are no mistakes.",
});

console.log(score.value); // High score indicates high uncertainty

ComplianceRiskJudge

Flags non-compliant or risky claims in regulated sectors:

TypeScript

import { ComplianceRiskJudge } from "opik";

const judge = new ComplianceRiskJudge({ model: "gpt-4o" });

const score = await judge.score({
  output: "This pill cures diabetes in a week.",
});

console.log(score.value); // High score indicates high risk

Available Judges

All built-in GEval judges:

QARelevanceJudge - Answer relevance to questions
SummarizationConsistencyJudge - Summary faithfulness
SummarizationCoherenceJudge - Summary structure and clarity
DialogueHelpfulnessJudge - Assistant helpfulness in dialogue
DemographicBiasJudge - Demographic stereotyping
GenderBiasJudge - Gender stereotyping
PoliticalBiasJudge - Political bias
ReligiousBiasJudge - Religious bias
RegionalBiasJudge - Geographic/cultural bias
AgentTaskCompletionJudge - Agent task fulfillment
AgentToolCorrectnessJudge - Agent tool usage correctness
PromptUncertaintyJudge - Prompt ambiguity
ComplianceRiskJudge - Regulatory compliance risk

Configuring LLM Judge Metrics

Model Configuration

All LLM Judge metrics accept a model parameter in their constructor:

TypeScript

import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { google } from "@ai-sdk/google";
import { Hallucination } from "opik";

// Using model ID string
const metric1 = new Hallucination({ model: "gpt-5-nano" });
const metric2 = new Hallucination({ model: "claude-3-5-sonnet-latest" });
const metric3 = new Hallucination({ model: "gemini-2.0-flash" });

// Using LanguageModel instance
const customModel = openai("gpt-5-nano");
const metric4 = new Hallucination({ model: customModel });

Async Scoring

All LLM Judge metrics support asynchronous scoring:

TypeScript

import { AnswerRelevance } from "opik";

const metric = new AnswerRelevance();

// Async/await
const score = await metric.score({
  input: "What is TypeScript?",
  output: "TypeScript is a typed superset of JavaScript.",
});

// Promise chain
metric
  .score({
    input: "What is TypeScript?",
    output: "TypeScript is a typed superset of JavaScript.",
  })
  .then((score) => console.log(score.value));

Combining Multiple LLM Judge Metrics

Use multiple metrics together for comprehensive evaluation:

TypeScript

import { AnswerRelevance, Hallucination, Moderation, Usefulness } from "opik";
import { evaluate } from "opik";

await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [
    new AnswerRelevance(),
    new Hallucination(),
    new Moderation(),
    new Usefulness(),
  ],
});

Custom Model for Each Metric

Different metrics can use different models:

TypeScript

import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { AnswerRelevance, Hallucination } from "opik";

// Use GPT-4o for answer relevance
const relevanceMetric = new AnswerRelevance({
  model: openai("gpt-5-nano"),
});

// Use Claude for hallucination detection
const hallucinationMetric = new Hallucination({
  model: anthropic("claude-3-5-sonnet-latest"),
});

await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [relevanceMetric, hallucinationMetric],
});

LLM Judge Metric Best Practices

1. Use Model ID Strings for Simplicity

For most use cases, use model ID strings directly:

TypeScript

import { Hallucination } from "opik";

const metric = new Hallucination({ model: "gpt-5-nano" });

The Opik SDK handles model configuration internally for optimal evaluation performance.

2. Provide Context When Available

Context improves evaluation accuracy:

TypeScript

// Better: With context
await metric.score({
  input: "What is the capital?",
  output: "The capital is Paris.",
  context: ["France is a country in Europe. Its capital is Paris."],
});

// OK: Without context (relies on general knowledge)
await metric.score({
  input: "What is the capital of France?",
  output: "The capital is Paris.",
});

3. Choose Appropriate Models

Match model capabilities to metric requirements:

TypeScript

// Complex reasoning: Use GPT-5 or Claude Sonnet
const complexMetric = new AnswerRelevance({ model: "gpt-5" });

// Simple checks: Use faster, cheaper models
const simpleMetric = new Moderation({ model: "gpt-5-nano" });

4. Handle Errors Gracefully

LLM calls can fail - handle errors appropriately:

TypeScript

try {
  const score = await metric.score({
    input: "What is TypeScript?",
    output: "TypeScript is a typed superset of JavaScript.",
  });
  console.log(score);
} catch (error) {
  console.error("Metric evaluation failed:", error);
  // Implement fallback or retry logic
}

5. Batch Evaluations

Use the evaluate function for efficient batch processing:

TypeScript

// More efficient for multiple items
await evaluate({
  dataset: myDataset,
  task: myTask,
  scoringMetrics: [new Hallucination()],
  scoringWorkers: 5, // Parallel scoring
});

// Less efficient for batch processing
for (const item of datasetItems) {
  await metric.score(item); // Sequential scoring
}

Score Interpretation

Understanding LLM Judge Scores

LLM Judge metrics return structured scores with:

TypeScript

interface EvaluationScoreResult {
  name: string; // Metric name
  value: number; // Numerical score (0.0-1.0 typically)
  reason: string | string[]; // Explanation for the score
}

Example Score Results

TypeScript

// AnswerRelevance
{
  name: "answer_relevance",
  value: 0.95,
  reason: "The answer directly addresses the question with accurate information"
}

// Hallucination
{
  name: "hallucination",
  value: 0.0,
  reason: ["All information is supported by the context", "No contradictions found"]
}

// Moderation
{
  name: "moderation",
  value: 0.0,
  reason: "Content is safe and appropriate"
}

Generation Parameters

Configuring Temperature, Seed, and MaxTokens

All LLM Judge metrics support generation parameters in their constructor:

TypeScript

import { Hallucination, AnswerRelevance } from "opik";

// Configure generation parameters
const metric = new Hallucination({
  model: "gpt-5-nano",
  temperature: 0.3, // Lower = more deterministic
  seed: 42, // For reproducible outputs
  maxTokens: 1000, // Maximum response length
});

// Different settings for different metrics
const relevanceMetric = new AnswerRelevance({
  model: "claude-3-5-sonnet-latest",
  temperature: 0.7, // Higher = more creative
  seed: 12345,
});

// Use the metrics
const score = await metric.score({
  input: "What is the capital of France?",
  output: "The capital of France is Paris.",
  context: ["France is a country in Western Europe."],
});

Advanced Model Settings

For provider-specific advanced parameters, use modelSettings:

TypeScript

import { Hallucination } from "opik";

const metric = new Hallucination({
  model: "gpt-5-nano",
  temperature: 0.5,
  modelSettings: {
    topP: 0.9, // Nucleus sampling
    topK: 50, // Top-K sampling
    presencePenalty: 0.1, // Reduce repetition
    frequencyPenalty: 0.2, // Reduce phrase repetition
    stopSequences: ["END"], // Custom stop sequences
  },
});

For provider-specific options not exposed through modelSettings, use LanguageModel instances:

TypeScript

import { openai } from "@ai-sdk/openai";
import { anthropic } from "@ai-sdk/anthropic";
import { google } from "@ai-sdk/google";

// OpenAI with structured outputs
const openaiModel = openai("gpt-5-nano", {
  structuredOutputs: true,
});

// Anthropic with cache control
const anthropicModel = anthropic("claude-3-5-sonnet-latest", {
  cacheControl: true,
});

// Google Gemini with specific configuration
const googleModel = google("gemini-2.0-flash");

const metric1 = new Hallucination({ model: openaiModel });
const metric2 = new Hallucination({ model: anthropicModel });
const metric3 = new Hallucination({ model: googleModel });

See Vercel AI SDK Provider Documentation for provider-specific options:

What Are Metrics?

How Metrics Calculate Scores

Types of Metrics

Built-in Metrics

ExactMatch

Contains

RegexMatch

IsJson

Metric Configuration

Custom Naming and Tracking

Combining Multiple Metrics

Input Requirements

Validation Schema

Mapping Inputs

Score Interpretation

Score Ranges

Creating Custom Metrics

Implementing Your Own Metric

Validation Best Practices

LLM Judge Metrics

AnswerRelevance

Parameters

Score Range

Hallucination

Parameters

Score Values

Moderation

Parameters

Score Values

Usefulness

Parameters

Score Range

GEval

Parameters

Score Range

How It Works

Built-in GEval Judges

QARelevanceJudge

SummarizationConsistencyJudge

SummarizationCoherenceJudge

DialogueHelpfulnessJudge

Bias Detection Judges

Agent Evaluation Judges

PromptUncertaintyJudge

ComplianceRiskJudge

Available Judges

Configuring LLM Judge Metrics

Model Configuration

Async Scoring

Combining Multiple LLM Judge Metrics

Custom Model for Each Metric

LLM Judge Metric Best Practices

1. Use Model ID Strings for Simplicity

2. Provide Context When Available

3. Choose Appropriate Models

4. Handle Errors Gracefully

5. Batch Evaluations

Score Interpretation

Understanding LLM Judge Scores

Example Score Results

Generation Parameters

Configuring Temperature, Seed, and MaxTokens

Advanced Model Settings

See Also