Define metrics

Metrics drive optimizer decisions. This guide highlights the fastest way to pick proven presets from Opik’s evaluation catalog, then shows how to extend them when your use case demands it. If you need the full theory, see Evaluation concepts and the metrics overview.

Metric anatomy

A metric is a callable with the signature (dataset_item, llm_output) -> ScoreResult | float. Use ScoreResult to attach names and reasons.

Python

from opik.evaluation.metrics.score_result import ScoreResult

def short_answer(item, output):
    is_short = len(output) <= 200
    return ScoreResult(
        name="short_answer",
        value=1.0 if is_short else 0.0,
        reason="Answer under 200 chars" if is_short else "Answer too long"
    )

Compose metrics

Use MultiMetricObjective to balance multiple goals (accuracy, style, safety).

Python

from opik_optimizer import MultiMetricObjective
from opik.evaluation.metrics import LevenshteinRatio, AnswerRelevance

objective = MultiMetricObjective(
    weights=[0.6, 0.4],
    metrics=[
        lambda item, output: LevenshteinRatio().score(reference=item["answer"], output=output),
        lambda item, output: AnswerRelevance().score(
            context=[item["answer"]], output=output, input=item["question"]
        ),
    ],
    name="accuracy_and_relevance",
)

Include cost and duration metrics

You can optimize for efficiency alongside quality by including span-based metrics like cost and duration in your composite objective. These metrics require access to the task_span parameter:

Python

from opik_optimizer import MultiMetricObjective
from opik.evaluation.metrics import AnswerRelevance
from opik_optimizer.metrics import SpanCost, SpanDuration

# Regular metric without task_span
def answer_relevance(dataset_item, llm_output):
    metric = AnswerRelevance()
    return metric.score(
        context=[dataset_item["answer"]], 
        output=llm_output, 
        input=dataset_item["question"]
    )

# Built-in span metrics can be normalized with target= for clean multi-metric weighting.
# invert=True (default) means lower raw value -> higher score.
cost = SpanCost(target=0.01, invert=True, name="cost_score")
duration = SpanDuration(target=6.0, invert=True, name="duration_score")

# Combine quality, cost, and speed metrics on a common [0, 1] scale
objective = MultiMetricObjective(
    metrics=[answer_relevance, cost, duration],
    weights=[0.33, 0.33, 0.33],  # equally optimize for accuracy, cost and duration/latency
    name="quality_cost_speed",
)

For a working end-to-end example in the repository, see: multi_metric_cost_duration_example.py

Recommended presets

Scenario	Metric	Notes
Factual QA	`LevenshteinRatio` or `ExactMatch`	Works with text-only datasets; deterministic and low cost.
Retrieval / grounding	`AnswerRelevance`	Pass reference context via `context=[item["answer"]]` or retrieved docs.
Safety	`Moderation` or custom LLM-as-a-judge	Combine with `MultiMetricObjective` to gate unsafe answers.
Multi-turn trajectories	Agent trajectory evaluator	Scores complete conversations, not just final outputs.

Reuse these heuristics before writing custom metrics—most are already imported in opik.evaluation.metrics.

Optimizer built-in metrics

Opik Optimizer also ships built-in metric helpers for common optimization setups:

Metric	Import	When to use
`LevenshteinAccuracyMetric`	`from opik_optimizer.metrics import LevenshteinAccuracyMetric`	Quick string-similarity accuracy using dataset keys like `answer` or `highlights`.
`SpanCost`	`from opik_optimizer.metrics import SpanCost`	Cost metric with `target=` normalization and `invert=` direction control.
`SpanDuration`	`from opik_optimizer.metrics import SpanDuration`	Duration metric with `target=` normalization and `invert=` direction control.

Example with built-ins:

Python

from opik_optimizer import MultiMetricObjective
from opik_optimizer.metrics import LevenshteinAccuracyMetric, SpanCost, SpanDuration

accuracy = LevenshteinAccuracyMetric(reference_key="answer")
cost = SpanCost(target=0.01, invert=True, name="cost_score")
duration = SpanDuration(target=6.0, invert=True, name="duration_score")

objective = MultiMetricObjective(
    metrics=[accuracy, cost, duration],
    weights=[0.5, 0.25, 0.25],  # all metrics already normalized to [0, 1]
    name="accuracy_cost_duration",
)

Checklist for great metrics

Return explanations – populate reason so reflective optimizers can group failure modes.
Avoid randomness – deterministic metrics keep optimizers from chasing noise.
Bound runtime – use cached references or lightweight models where possible; heavy metrics slow down trials.
Log metadata – include details in the ScoreResult if you want to visualize per-sample attributes later.

When you outgrow presets, move to Custom metrics for LLM-as-a-judge flows or domain-specific scoring.

Testing metrics

Dry-run against a handful of dataset rows before launching an optimization.
Use optimizer.task_evaluator.evaluate_prompt to evaluate a single prompt with your metric.
Inspect the per-sample reasons in the Opik dashboard to ensure they match expectations.

Deep dive: Multi-metric optimization guide
API reference: ScoreResult
Advanced topic: Custom metrics

Metric anatomy

Compose metrics

Include cost and duration metrics

Recommended presets

Optimizer built-in metrics

Checklist for great metrics

Testing metrics

Related resources