Evaluate threads

When you are running multi-turn conversations using frameworks that support LLM agents, the Opik integration will automatically group related traces into conversation threads using parameters suitable for each framework.

This guide will walk you through the process of evaluating and optimizing conversation threads in Opik using the evaluate_threads function in the Python SDK.

The Python SDK provides a simple and efficient way to evaluate and optimize conversation threads using the evaluate_threads function. This function allows you to specify a filter string to select specific threads for evaluation, a list of metrics to apply to each thread, and it returns a ThreadsEvaluationResult object containing the evaluation results and feedback scores.

Most importantly, this function automatically uploads the feedback scores to your traces in Opik! So, once evaluation is completed, you can also see the results in the UI.

To run the threads evaluation, you can use the following code:

from opik.evaluation import evaluate_threads
from opik.evaluation.metrics import ConversationalCoherenceMetric, UserFrustrationMetric

# Initialize the evaluation metrics
conversation_coherence_metric = ConversationalCoherenceMetric()
user_frustration_metric = UserFrustrationMetric()

# Run the threads evaluation
results = evaluate_threads(
    project_name="ai_team",
    filter_string='id = "0197ad2a"',
    eval_project_name="ai_team_evaluation",
    metrics=[
        conversation_coherence_metric,
        user_frustration_metric,
    ],
    trace_input_transform=lambda x: x["input"],
    trace_output_transform=lambda x: x["output"],
)

Threads consist of multiple traces, and each trace has an input and output. In practice, these typically contain user messages and agent responses. However, trace inputs and outputs are rarely just simple strings—they are usually complex data structures whose exact format depends on your agent framework.

To handle this complexity, you need to provide trace_input_transform and trace_output_transform functions. These are critical parameters that tell Opik how to extract the actual message content from your framework-specific trace structure.

Different agent frameworks structure their trace data differently:

• LangChain might store messages in {"messages": [{"content": "..."}]}
• CrewAI might use {"task": {"description": "..."}}
• Custom implementations can have any structure you've defined

Without transform functions, Opik wouldn't know where to find the actual user questions and agent responses within your trace data.

Using these functions, the Opik evaluation engine will convert your threads chosen for evaluation into the standardized format expected by all Opik thread evaluation metrics:

[
    {
        "role": "user",
        "content": "input string from trace 1"
    },
    {
        "role": "assistant",
        "content": "output string from trace 1"
    },
    {
        "role": "user",
        "content": "input string from trace 2"
    },
    {
        "role": "assistant",
        "content": "output string from trace 2"
    }
]

Example:

If your trace input has the following structure:

{
    "content": {
        "user_question": "Tell me about your service?"
    },
    "metadata": {...}
}

Then your trace_input_transform should be:

lambda x: x["content"]["user_question"]

Relevance and faithfulness metrics need to know what the agent's answer was grounded on. If your agent retrieves documents, pass a third transform, trace_context_transform, to extract them:

from opik.evaluation import evaluate_threads
from opik.evaluation.metrics import ConversationalCoherenceMetric

results = evaluate_threads(
    project_name="rag_chatbot",
    filter_string='id = "0197ad2a"',
    eval_project_name="rag_chatbot_evaluation",
    metrics=[ConversationalCoherenceMetric()],
    trace_input_transform=lambda x: x["input"],
    trace_output_transform=lambda x: x["output"],
    trace_context_transform=lambda trace: trace.metadata["retrieved_docs"],
)

Unlike the other two transforms, which receive trace.input and trace.output, trace_context_transform receives the whole trace object, because the retrieved documents can be logged anywhere - in trace.metadata, trace.output or trace.input. It should return a list of strings, or None for turns with no context.

The extracted context is attached to the agent message produced by the same trace:

[
    {
        "role": "user",
        "content": "input string from trace 1"
    },
    {
        "role": "assistant",
        "content": "output string from trace 1",
        "context": ["document retrieved for trace 1"]
    }
]

The evaluate_threads function takes a filter string as an argument. This string is used to select the threads that should be evaluated. For example, if you want to evaluate only threads that have a specific ID, you can use the following filter string:

filter_string='id = "0197ad2a"'

You can combine multiple filter strings using the AND operator. For example, if you want to evaluate only threads that have a specific ID and were created after a certain date, you can use the following filter string:

filter_string='id = "0197ad2a" AND start_time > "2024-01-01T00:00:00Z"'

Supported filter fields and operators

The evaluate_threads function supports the following filter fields in the filter_string using Opik Query Language (OQL). All fields and operators are the same as those supported by search_traces and search_spans:

Rules:

• String values must be wrapped in double quotes
• DateTime fields require ISO 8601 format (e.g., "2024-01-01T00:00:00Z")
• Use dot notation for nested objects: metadata.model, feedback_scores.accuracy
• Multiple conditions can be combined with AND (OR is not supported)

The feedback_scores field is a dictionary where the keys are the metric names and the values are the metric values. You can use it to filter threads based on their feedback scores. For example, if you want to evaluate only threads that have a specific user frustration score, you can use the following filter string:

filter_string='feedback_scores.user_frustration_score >= 0.5'

Where user_frustration_score is the name of the user frustration metric and 0.5 is the threshold value to filter by.

Once the evaluation is complete, you can access the evaluation results in the Opik UI. Not only you will be able to see the score values, but the LLM-judge reasoning behind these values too!

Team-based thread evaluation enables multiple evaluators to score conversation threads independently, providing more reliable assessment of multi-turn dialogue quality.

Key benefits for thread evaluation:

• Conversation complexity scoring - Multiple reviewers can assess different aspects like coherence, user satisfaction, and goal completion across conversation turns
• Reduced evaluation bias - Individual subjectivity in judging conversational quality is mitigated through team consensus
• Thread-specific metrics - Teams can collaboratively evaluate conversation-specific aspects like frustration levels, topic drift, and resolution success

This collaborative approach is especially valuable for conversational threads where dialogue quality, context maintenance, and user experience assessment often require multiple expert perspectives.

For more details on what metrics can be used to score conversational threads, refer to the conversational metrics page.

You can also define custom metrics to evaluate conversational threads, including LLM-as-a-Judge (LLM-J) reasoning metrics, as described in the following section: Custom Conversation Metrics guide.