Tool Optimization (MCP & Function Calling)

Tool optimization is a specialized feature that allows you to optimize prompts that use external tools and the Model Context Protocol (MCP). This capability is supported by all optimizers except FewShotBayesianOptimizer, ParameterOptimizer, and GepaOptimizer.

Tool optimization extends traditional prompt optimization to handle prompts that include:

• MCP tools - Model Context Protocol tools for external integrations
• Tool schemas - Structured tool definitions and parameters
• Multi-step workflows - Complex agent workflows involving multiple tools

Tool optimization does not change tool names or schemas. It updates:

• Tool descriptions
• Tool parameter descriptions

Many optimizers can optimize agents that use function calling, but this is different from true tool optimization. Here's an example from the GEPA optimizer:

from opik_optimizer import GepaOptimizer, ChatPrompt

# GEPA example: optimizing an agent with function calling
prompt = ChatPrompt(
    system="You are a helpful assistant. Use the search_wikipedia tool when needed.",
    user="{question}",
    tools=[
        {
            "type": "function",
            "function": {
                "name": "search_wikipedia",
                "description": "This function searches Wikipedia abstracts.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "query": {
                            "type": "string",
                            "description": "The term or phrase to search for."
                        }
                    },
                    "required": ["query"]
                }
            }
        }
    ],
    function_map={
        "search_wikipedia": lambda query: search_wikipedia(query, use_api=True)
    }
)

# GEPA optimizes the agent's prompt, not the tools themselves
optimizer = GepaOptimizer(model="gpt-5-nano")
result = optimizer.optimize_prompt(prompt=prompt, dataset=dataset, metric=metric)

True tool optimization is available for MCP tools and function-calling tools across supported optimizers. MCP tools provide standardized interfaces for external integrations:

# MCP tool optimization example
# See scripts/litellm_metaprompt_context7_remote_example.py for a working example

from opik_optimizer import MetaPromptOptimizer

# MCP tools are configured as OpenAI-style entries (local or remote)
optimizer = MetaPromptOptimizer(model="gpt-5-nano")

# Any supported optimizer works here (e.g., HRPO, Evolutionary, MetaPrompt).

Supported optimizers handle tool-enabled prompts through a specialized optimization process:

The optimizer analyzes:

• Tool schemas - Understanding available functions and their parameters
• Tool usage patterns - How tools are typically invoked in the prompt
• Tool dependencies - Relationships between different tools
• Context requirements - What information tools need to function effectively

The optimizer can improve:

• Tool selection logic - Better instructions for when to use which tools
• Parameter formatting - Clearer guidance on how to structure tool inputs
• Error handling - Instructions for handling tool failures or edge cases
• Tool chaining - Optimizing multi-step tool workflows

Tool optimization also improves:

• Input validation - Better prompts for validating tool inputs
• Output processing - Instructions for handling tool outputs
• Fallback strategies - Alternative approaches when tools are unavailable

Let's see how tool optimization works with a research assistant that uses multiple tools:

from opik_optimizer import MetaPromptOptimizer, ChatPrompt
from opik.evaluation.metrics import LevenshteinRatio

# Define a research assistant prompt with tools
research_prompt = ChatPrompt(
    messages=[
        {
            "role": "system",
            "content": """You are a research assistant. When given a research question:
1. Search for relevant information using the search tool
2. Analyze the results using the analysis tool
3. Provide a comprehensive answer based on your findings

Always cite your sources and be thorough in your research."""
        },
        {
            "role": "user",
            "content": "{research_question}"
        }
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "search_academic_database",
                "description": "Search academic papers and research",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "query": {"type": "string"},
                        "year_range": {"type": "string"},
                        "max_results": {"type": "integer"}
                    }
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "analyze_text",
                "description": "Analyze and summarize text content",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "text": {"type": "string"},
                        "analysis_type": {"type": "string"}
                    }
                }
            }
        }
    ]
)

# Initialize the optimizer
optimizer = MetaPromptOptimizer(
    model="openai/gpt-5-nano"
)

# Define evaluation metric
def research_quality_metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item['expected_answer'],
        output=llm_output
    )

# Run optimization
result = optimizer.optimize_prompt(
    prompt=research_prompt,
    dataset=research_dataset,
    metric=research_quality_metric,
    n_samples=100,
    max_trials=5
)

print("Optimized prompt with tools:")
print(result.prompt)

• Clear descriptions - Provide detailed descriptions for each tool
• Comprehensive parameters - Include all necessary parameters with types
• Example usage - Add examples in tool descriptions when helpful
• Error handling - Define expected error conditions and responses

• Tool introduction - Clearly explain available tools to the model
• Usage guidelines - Provide specific instructions on when and how to use tools
• Output formatting - Specify how tool outputs should be processed
• Fallback instructions - Define what to do when tools fail

• Tool usage metrics - Measure not just final output quality but tool usage effectiveness
• Multi-step evaluation - Evaluate each step in tool-based workflows
• Error rate tracking - Monitor tool failure rates and recovery strategies
• Context preservation - Ensure important context is maintained across tool calls

• Optimizer coverage - Tool optimization is not available in FewShotBayesianOptimizer or ParameterOptimizer
• Tool Complexity - Very complex tool workflows may require manual optimization
• Tool Availability - Optimization assumes tools are available during evaluation
• Schema Changes - Tool schema modifications may require re-optimization

• Evaluation Cost - Tool-enabled prompts require more LLM calls for evaluation
• Tool Latency - External tool calls can slow down optimization
• Resource Usage - Complex tool workflows may require significant computational resources

Tool optimization is an active area of development. Planned improvements include:

• Tool-specific metrics - Specialized evaluation metrics for tool usage
• Automated tool discovery - Automatic detection and optimization of tool patterns
• Tool performance optimization - Optimizing not just prompts but tool usage efficiency

To start optimizing tool-enabled prompts:

1. Choose a supported optimizer - Any optimizer except FewShotBayesianOptimizer and ParameterOptimizer
2. Define your tools - Create clear tool schemas with comprehensive descriptions
3. Structure your prompt - Include clear instructions for tool usage
4. Prepare evaluation data - Ensure your dataset includes tool usage scenarios
5. Run optimization - Use the standard optimization process with tool-enabled prompts

• MetaPrompt Optimizer Documentation
• Optimize tools (MCP)
• Model Context Protocol Specification