Optimize agents

The Opik Agent Optimizer can optimize both simple prompts and complex agent workflows. For most use cases, you can optimize prompts directly using ChatPrompt. When you need multi-prompt workflows, agent orchestration, or custom execution logic, you'll use OptimizableAgent to create a custom agent class.

When to use OptimizableAgent vs ChatPrompt

Use ChatPrompt directly (default approach):

Single-prompt optimization - optimizing one prompt template
Most common use case
No custom execution logic needed

Use OptimizableAgent when you need:

Multi-prompt workflows - orchestrating multiple prompts in sequence
Agent framework integration - connecting to ADK, LangGraph, CrewAI, etc.
Custom execution logic - special tool handling, async workflows, etc.

Optimizers work seamlessly with both approaches. The optimizer calls your agent's invoke_agent() method repeatedly during optimization, passing different prompt candidates to evaluate.

Single-prompt optimization

For most optimization tasks, you can use ChatPrompt directly without creating a custom agent. The optimizer uses a default LiteLLM-based agent under the hood.

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot

dataset = hotpot(count=300)

def levenshtein_ratio(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"], 
        output=llm_output
    )

prompt = ChatPrompt(
    system="You are a helpful assistant.",
    user="{question}",
    model="openai/gpt-4o-mini"
)

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=levenshtein_ratio,
    max_trials=5,
    n_samples=50
)

result.display()

Custom agent for framework integration

When integrating with specific agent frameworks (Google ADK, LangGraph, CrewAI, etc.), you'll create a custom OptimizableAgent subclass. This allows the optimizer to work with your framework's execution model.

Here's an example for Google ADK:

Python

from typing import Any, TYPE_CHECKING
from opik_optimizer import OptimizableAgent

if TYPE_CHECKING:
    from opik_optimizer.api_objects import chat_prompt

class ADKAgent(OptimizableAgent):
    project_name = "adk-agent"

    def invoke_agent(
        self,
        prompts: dict[str, chat_prompt.ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # Single-prompt agents extract the prompt from the dict
        if len(prompts) > 1:
            raise ValueError("ADKAgent only supports single-prompt optimization.")
        
        prompt = list(prompts.values())[0]
        messages = prompt.get_messages(dataset_item)
        
        # Your framework-specific execution logic here
        # ... create ADK agent, run it, return response ...
        
        return response

The key points:

Extract the single prompt from prompts dict: prompt = list(prompts.values())[0]
Get formatted messages: messages = prompt.get_messages(dataset_item)
Execute using your framework and return the response string

Multi-prompt optimization

For multi-step agent workflows, you must use OptimizableAgent because ChatPrompt only handles a single prompt. Multi-prompt optimization allows you to optimize multiple prompts that work together in a pipeline.

When to use multi-prompt optimization

Sequential reasoning workflows (analyze → respond)
Multi-hop retrieval pipelines
Agent orchestration with multiple steps
Any workflow where one prompt's output feeds into another

Implementing a multi-prompt agent

Here's a simple example of a two-step workflow that analyzes input and then generates a response:

Python

from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent
from openai import OpenAI

class AnalyzeRespondAgent(OptimizableAgent):
    """Two-step agent: analyze input, then respond based on analysis."""
    
    def __init__(self, model: str = "gpt-4o-mini"):
        super().__init__()
        self.model = model
        self.client = OpenAI()

    def invoke_agent(
        self,
        prompts: dict[str, ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # Step 1: Analyze the input
        analyze_prompt = prompts["analyze"]
        analyze_messages = analyze_prompt.get_messages(dataset_item)
        
        analyze_response = self.client.chat.completions.create(
            model=self.model,
            messages=analyze_messages,
            seed=seed,
        )
        analysis = analyze_response.choices[0].message.content

        # Step 2: Generate response based on analysis
        respond_prompt = prompts["respond"]
        # Pass analysis result to the respond prompt
        respond_context = {**dataset_item, "analysis": analysis}
        respond_messages = respond_prompt.get_messages(respond_context)
        
        respond_response = self.client.chat.completions.create(
            model=self.model,
            messages=respond_messages,
            seed=seed,
        )
        
        return respond_response.choices[0].message.content

Using the multi-prompt agent

When optimizing, pass a dictionary of prompts instead of a single prompt:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio

# Define both prompts in the workflow
prompts = {
    "analyze": ChatPrompt(
        system="You are an analysis assistant. Extract key information from the input.",
        user="{text}",
        model="gpt-4o-mini"
    ),
    "respond": ChatPrompt(
        system="You are a response assistant. Generate a helpful response based on the analysis.",
        user="Analysis: {analysis}\n\nOriginal question: {text}",
        model="gpt-4o-mini"
    ),
}

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
    prompt=prompts,  # Pass dict of prompts
    agent_class=AnalyzeRespondAgent,  # Use your custom agent
    dataset=dataset,
    metric=levenshtein_ratio,
    max_trials=5,
    n_samples=50
)

result.display()

The optimizer will optimize both prompts in the dictionary, trying different combinations to improve performance.

Key implementation details

invoke_agent() method signature

All OptimizableAgent subclasses must implement invoke_agent():

Python

def invoke_agent(
    self,
    prompts: dict[str, ChatPrompt],
    dataset_item: dict[str, Any],
    allow_tool_use: bool = False,
    seed: int | None = None,
) -> str:
    # Your implementation here
    return response_string

Parameters:

prompts: Dictionary mapping prompt names to ChatPrompt objects
dataset_item: Dataset row used to format prompt messages
allow_tool_use: Whether tools may be executed (for tool-calling prompts)
seed: Optional random seed for reproducibility

Returns: A single string output that will be scored by your metric function

Extracting messages from prompts

Use ChatPrompt.get_messages() to format the prompt with dataset values:

Python

messages = prompt.get_messages(dataset_item)
# Returns list of message dicts: [{"role": "system", "content": "..."}, ...]

For multi-prompt workflows, pass additional context when calling get_messages():

Python

# Pass intermediate results to subsequent prompts
context = {**dataset_item, "intermediate_result": some_value}
messages = prompt.get_messages(context)

Best practices

Error handling: Return meaningful error messages if execution fails
Model parameters: Respect prompt.model and prompt.model_kwargs for consistency
Reproducibility: Use the seed parameter when making LLM calls
Opik tracing: The base class handles tracing automatically, but you can add custom metadata via self.trace_metadata

Complete examples

Single-prompt with ChatPrompt (default)

Python

from opik_optimizer import ChatPrompt, EvolutionaryOptimizer
from opik_optimizer.datasets import hotpot
from opik.evaluation.metrics import LevenshteinRatio

dataset = hotpot(count=300)

def metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"], 
        output=llm_output
    )

prompt = ChatPrompt(
    system="You are a helpful assistant.",
    user="{question}",
    model="openai/gpt-4o-mini"
)

optimizer = EvolutionaryOptimizer(
    model="openai/gpt-4o-mini",
    population_size=5,
    num_generations=3
)

result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=metric,
    n_samples=50
)

result.display()

Multi-prompt workflow

Python

from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent, HRPO
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot
from openai import OpenAI

class TwoStepAgent(OptimizableAgent):
    def __init__(self, model: str = "gpt-4o-mini"):
        super().__init__()
        self.model = model
        self.client = OpenAI()

    def invoke_agent(
        self,
        prompts: dict[str, ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # First step
        step1_prompt = prompts["step1"]
        step1_messages = step1_prompt.get_messages(dataset_item)
        step1_response = self.client.chat.completions.create(
            model=self.model,
            messages=step1_messages,
            seed=seed,
        )
        step1_result = step1_response.choices[0].message.content

        # Second step uses result from first step
        step2_prompt = prompts["step2"]
        step2_context = {**dataset_item, "step1_result": step1_result}
        step2_messages = step2_prompt.get_messages(step2_context)
        step2_response = self.client.chat.completions.create(
            model=self.model,
            messages=step2_messages,
            seed=seed,
        )
        
        return step2_response.choices[0].message.content

# Define multi-prompt workflow
prompts = {
    "step1": ChatPrompt(
        system="Analyze the question and identify key information.",
        user="{question}",
        model="gpt-4o-mini"
    ),
    "step2": ChatPrompt(
        system="Answer the question based on the analysis.",
        user="Question: {question}\n\nAnalysis: {step1_result}",
        model="gpt-4o-mini"
    ),
}

dataset = hotpot(count=300)

def metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"], 
        output=llm_output
    )

optimizer = HRPO(
    model="openai/gpt-4o-mini",
    n_threads=2,
    max_parallel_batches=3
)

result = optimizer.optimize_prompt(
    prompt=prompts,
    agent_class=TwoStepAgent,
    dataset=dataset,
    metric=metric,
    max_trials=5,
    n_samples=50
)

result.display()

Next steps

Explore optimization algorithms to choose the right optimizer
Learn about defining datasets and metrics
Check framework-specific examples in sdks/opik_optimizer/scripts/llm_frameworks/