Optimize agents
Connect the Agent Optimizer to Agent Frameworks.
The Opik Agent Optimizer can optimize both simple prompts and complex agent workflows. For most use cases, you can optimize prompts directly using ChatPrompt. When you need multi-prompt workflows, agent orchestration, or custom execution logic, you'll use OptimizableAgent to create a custom agent class.
When to use OptimizableAgent vs ChatPrompt
Section titled “When to use OptimizableAgent vs ChatPrompt”Use ChatPrompt directly (default approach):
- Single-prompt optimization - optimizing one prompt template
- Most common use case
- No custom execution logic needed
Use OptimizableAgent when you need:
- Multi-prompt workflows - orchestrating multiple prompts in sequence
- Agent framework integration - connecting to ADK, LangGraph, CrewAI, etc.
- Custom execution logic - special tool handling, async workflows, etc.
Optimizers work seamlessly with both approaches. The optimizer calls your agent's invoke_agent() method repeatedly during optimization, passing different prompt candidates to evaluate.
Single-prompt optimization
Section titled “Single-prompt optimization”For most optimization tasks, you can use ChatPrompt directly without creating a custom agent. The optimizer uses a default LiteLLM-based agent under the hood.
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot
dataset = hotpot(count=300)
def levenshtein_ratio(dataset_item, llm_output):
return LevenshteinRatio().score(
reference=dataset_item["answer"],
output=llm_output
)
prompt = ChatPrompt(
system="You are a helpful assistant.",
user="{question}",
model="openai/gpt-4o-mini"
)
optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
prompt=prompt,
dataset=dataset,
metric=levenshtein_ratio,
max_trials=5,
n_samples=50
)
result.display()Custom agent for framework integration
Section titled “Custom agent for framework integration”When integrating with specific agent frameworks (Google ADK, LangGraph, CrewAI, etc.), you'll create a custom OptimizableAgent subclass. This allows the optimizer to work with your framework's execution model.
Here's an example for Google ADK:
from typing import Any, TYPE_CHECKING
from opik_optimizer import OptimizableAgent
if TYPE_CHECKING:
from opik_optimizer.api_objects import chat_prompt
class ADKAgent(OptimizableAgent):
project_name = "adk-agent"
def invoke_agent(
self,
prompts: dict[str, chat_prompt.ChatPrompt],
dataset_item: dict[str, Any],
allow_tool_use: bool = False,
seed: int | None = None,
) -> str:
# Single-prompt agents extract the prompt from the dict
if len(prompts) > 1:
raise ValueError("ADKAgent only supports single-prompt optimization.")
prompt = list(prompts.values())[0]
messages = prompt.get_messages(dataset_item)
# Your framework-specific execution logic here
# ... create ADK agent, run it, return response ...
return responseThe key points:
- Extract the single prompt from
promptsdict:prompt = list(prompts.values())[0] - Get formatted messages:
messages = prompt.get_messages(dataset_item) - Execute using your framework and return the response string
Multi-prompt optimization
Section titled “Multi-prompt optimization”For multi-step agent workflows, you must use OptimizableAgent because ChatPrompt only handles a single prompt. Multi-prompt optimization allows you to optimize multiple prompts that work together in a pipeline.
When to use multi-prompt optimization
Section titled “When to use multi-prompt optimization”- Sequential reasoning workflows (analyze → respond)
- Multi-hop retrieval pipelines
- Agent orchestration with multiple steps
- Any workflow where one prompt's output feeds into another
Implementing a multi-prompt agent
Section titled “Implementing a multi-prompt agent”Here's a simple example of a two-step workflow that analyzes input and then generates a response:
from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent
from openai import OpenAI
class AnalyzeRespondAgent(OptimizableAgent):
"""Two-step agent: analyze input, then respond based on analysis."""
def __init__(self, model: str = "gpt-4o-mini"):
super().__init__()
self.model = model
self.client = OpenAI()
def invoke_agent(
self,
prompts: dict[str, ChatPrompt],
dataset_item: dict[str, Any],
allow_tool_use: bool = False,
seed: int | None = None,
) -> str:
# Step 1: Analyze the input
analyze_prompt = prompts["analyze"]
analyze_messages = analyze_prompt.get_messages(dataset_item)
analyze_response = self.client.chat.completions.create(
model=self.model,
messages=analyze_messages,
seed=seed,
)
analysis = analyze_response.choices[0].message.content
# Step 2: Generate response based on analysis
respond_prompt = prompts["respond"]
# Pass analysis result to the respond prompt
respond_context = {**dataset_item, "analysis": analysis}
respond_messages = respond_prompt.get_messages(respond_context)
respond_response = self.client.chat.completions.create(
model=self.model,
messages=respond_messages,
seed=seed,
)
return respond_response.choices[0].message.contentUsing the multi-prompt agent
Section titled “Using the multi-prompt agent”When optimizing, pass a dictionary of prompts instead of a single prompt:
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
# Define both prompts in the workflow
prompts = {
"analyze": ChatPrompt(
system="You are an analysis assistant. Extract key information from the input.",
user="{text}",
model="gpt-4o-mini"
),
"respond": ChatPrompt(
system="You are a response assistant. Generate a helpful response based on the analysis.",
user="Analysis: {analysis}\n\nOriginal question: {text}",
model="gpt-4o-mini"
),
}
optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
prompt=prompts, # Pass dict of prompts
agent_class=AnalyzeRespondAgent, # Use your custom agent
dataset=dataset,
metric=levenshtein_ratio,
max_trials=5,
n_samples=50
)
result.display()The optimizer will optimize both prompts in the dictionary, trying different combinations to improve performance.
Key implementation details
Section titled “Key implementation details”invoke_agent() method signature
Section titled “invoke_agent() method signature”All OptimizableAgent subclasses must implement invoke_agent():
def invoke_agent(
self,
prompts: dict[str, ChatPrompt],
dataset_item: dict[str, Any],
allow_tool_use: bool = False,
seed: int | None = None,
) -> str:
# Your implementation here
return response_stringParameters:
prompts: Dictionary mapping prompt names toChatPromptobjectsdataset_item: Dataset row used to format prompt messagesallow_tool_use: Whether tools may be executed (for tool-calling prompts)seed: Optional random seed for reproducibility
Returns: A single string output that will be scored by your metric function
Extracting messages from prompts
Section titled “Extracting messages from prompts”Use ChatPrompt.get_messages() to format the prompt with dataset values:
messages = prompt.get_messages(dataset_item)
# Returns list of message dicts: [{"role": "system", "content": "..."}, ...]For multi-prompt workflows, pass additional context when calling get_messages():
# Pass intermediate results to subsequent prompts
context = {**dataset_item, "intermediate_result": some_value}
messages = prompt.get_messages(context)Best practices
Section titled “Best practices”- Error handling: Return meaningful error messages if execution fails
- Model parameters: Respect
prompt.modelandprompt.model_kwargsfor consistency - Reproducibility: Use the
seedparameter when making LLM calls - Opik tracing: The base class handles tracing automatically, but you can add custom metadata via
self.trace_metadata
Complete examples
Section titled “Complete examples”Single-prompt with ChatPrompt (default)
Section titled “Single-prompt with ChatPrompt (default)”from opik_optimizer import ChatPrompt, EvolutionaryOptimizer
from opik_optimizer.datasets import hotpot
from opik.evaluation.metrics import LevenshteinRatio
dataset = hotpot(count=300)
def metric(dataset_item, llm_output):
return LevenshteinRatio().score(
reference=dataset_item["answer"],
output=llm_output
)
prompt = ChatPrompt(
system="You are a helpful assistant.",
user="{question}",
model="openai/gpt-4o-mini"
)
optimizer = EvolutionaryOptimizer(
model="openai/gpt-4o-mini",
population_size=5,
num_generations=3
)
result = optimizer.optimize_prompt(
prompt=prompt,
dataset=dataset,
metric=metric,
n_samples=50
)
result.display()Multi-prompt workflow
Section titled “Multi-prompt workflow”from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent, HRPO
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot
from openai import OpenAI
class TwoStepAgent(OptimizableAgent):
def __init__(self, model: str = "gpt-4o-mini"):
super().__init__()
self.model = model
self.client = OpenAI()
def invoke_agent(
self,
prompts: dict[str, ChatPrompt],
dataset_item: dict[str, Any],
allow_tool_use: bool = False,
seed: int | None = None,
) -> str:
# First step
step1_prompt = prompts["step1"]
step1_messages = step1_prompt.get_messages(dataset_item)
step1_response = self.client.chat.completions.create(
model=self.model,
messages=step1_messages,
seed=seed,
)
step1_result = step1_response.choices[0].message.content
# Second step uses result from first step
step2_prompt = prompts["step2"]
step2_context = {**dataset_item, "step1_result": step1_result}
step2_messages = step2_prompt.get_messages(step2_context)
step2_response = self.client.chat.completions.create(
model=self.model,
messages=step2_messages,
seed=seed,
)
return step2_response.choices[0].message.content
# Define multi-prompt workflow
prompts = {
"step1": ChatPrompt(
system="Analyze the question and identify key information.",
user="{question}",
model="gpt-4o-mini"
),
"step2": ChatPrompt(
system="Answer the question based on the analysis.",
user="Question: {question}\n\nAnalysis: {step1_result}",
model="gpt-4o-mini"
),
}
dataset = hotpot(count=300)
def metric(dataset_item, llm_output):
return LevenshteinRatio().score(
reference=dataset_item["answer"],
output=llm_output
)
optimizer = HRPO(
model="openai/gpt-4o-mini",
n_threads=2,
max_parallel_batches=3
)
result = optimizer.optimize_prompt(
prompt=prompts,
agent_class=TwoStepAgent,
dataset=dataset,
metric=metric,
max_trials=5,
n_samples=50
)
result.display()Next steps
Section titled “Next steps”- Explore optimization algorithms to choose the right optimizer
- Learn about defining datasets and metrics
- Check framework-specific examples in
sdks/opik_optimizer/scripts/llm_frameworks/