Configuring LLM Providers

The Opik Agent Optimizer uses LiteLLM under the hood, giving you access to 100+ LLM providers with a unified interface. This guide shows you how to configure different providers for both your ChatPrompt (the model that runs your prompt) and the Optimizer (the model that improves your prompt).

Understanding the Two Model Types

When using the Opik Optimizer, there are two distinct models to configure:

Model Type	Purpose	Recommendation
ChatPrompt model	The model that executes your prompt during evaluation	Use the same model as your production application
Optimizer model	The model that analyzes failures and generates improved prompts	Use the most capable model available for best results

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

# ChatPrompt model - this is the model your prompt runs on
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"  # Your production model
)

# Optimizer model - this is the model that improves your prompt
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o"  # Use a powerful model for optimization
)

LiteLLM Model Format

All models use the LiteLLM format: provider/model-name

Python

# Examples of the LiteLLM model format
model="openai/gpt-4o"                          # OpenAI
model="anthropic/claude-3-5-sonnet-20241022"   # Anthropic
model="gemini/gemini-2.0-flash"                # Google Gemini
model="azure/my-deployment-name"               # Azure OpenAI
model="ollama/llama3"                          # Ollama (local)
model="openrouter/google/gemini-2.0-flash"     # OpenRouter

Provider Configuration

OpenAI

Environment Variable:

Bash

export OPENAI_API_KEY="sk-..."

Available Models:

openai/gpt-4o - Most capable model
openai/gpt-4o-mini - Fast and cost-effective
openai/gpt-4-turbo - Previous generation
openai/o1 - Reasoning model
openai/o3-mini - Efficient reasoning model

Example:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openai/gpt-4o-mini"
)

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")

Anthropic (Claude)

Environment Variable:

Bash

export ANTHROPIC_API_KEY="sk-ant-..."

Available Models:

anthropic/claude-3-5-sonnet-20241022 - Best balance of speed and capability
anthropic/claude-3-opus-20240229 - Most capable
anthropic/claude-3-haiku-20240307 - Fastest

Example:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="anthropic/claude-3-5-sonnet-20241022"
)

optimizer = MetaPromptOptimizer(model="anthropic/claude-3-5-sonnet-20241022")

Google Gemini

Environment Variable:

Bash

export GOOGLE_API_KEY="..."
# or
export GEMINI_API_KEY="..."

Available Models:

gemini/gemini-2.0-flash - Latest fast model
gemini/gemini-1.5-pro - Most capable
gemini/gemini-1.5-flash - Fast and efficient
gemini/gemini-1.5-flash-8b - Lightweight model

Example:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"
)

optimizer = MetaPromptOptimizer(model="gemini/gemini-1.5-pro")

Complete Gemini Example:

Python

import os
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
import opik

# Set your API key
os.environ["GOOGLE_API_KEY"] = "your-api-key-here"

# Initialize Opik client and create dataset
client = opik.Opik()
dataset = client.get_or_create_dataset(name="gemini-optimization-demo", project_name="my-project")
dataset.insert([
    {"question": "What is machine learning?", "answer": "Machine learning is a subset of AI that enables systems to learn from data."},
    {"question": "What is Python?", "answer": "Python is a high-level programming language known for its readability."},
])

# Define metric
def answer_quality(dataset_item, llm_output):
    metric = LevenshteinRatio()
    return metric.score(reference=dataset_item["answer"], output=llm_output)

# Configure prompt with Gemini
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "Answer the question concisely."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"
)

# Configure optimizer (can use same or different provider)
optimizer = MetaPromptOptimizer(
    model="gemini/gemini-1.5-pro",
    n_threads=4
)

# Run optimization
result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=answer_quality,
    max_trials=5,
    n_samples=2
)

result.display()

Azure OpenAI

Environment Variables:

Bash

export AZURE_API_KEY="..."
export AZURE_API_BASE="https://your-resource.openai.azure.com"
export AZURE_API_VERSION="2024-02-15-preview"

Model Format:

Python

# Use your deployment name
model="azure/your-deployment-name"

Example:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="azure/gpt-4o-deployment"  # Your Azure deployment name
)

optimizer = MetaPromptOptimizer(model="azure/gpt-4o-deployment")

Ollama (Local Models)

Setup:

Install Ollama from ollama.ai
Pull a model: ollama pull llama3
Ollama runs on http://localhost:11434 by default

Environment Variable (optional):

Bash

export OLLAMA_API_BASE="http://localhost:11434"

Available Models:

ollama/llama3 - Meta's Llama 3
ollama/mistral - Mistral 7B
ollama/codellama - Code-focused Llama
ollama/phi3 - Microsoft Phi-3

Example:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="ollama/llama3"
)

# Note: Local models may be slower for optimization
optimizer = MetaPromptOptimizer(model="ollama/llama3")

OpenRouter

OpenRouter provides access to multiple providers through a single API.

Environment Variable:

Bash

export OPENROUTER_API_KEY="sk-or-..."

Model Format:

Python

# Format: openrouter/provider/model
model="openrouter/google/gemini-2.0-flash"
model="openrouter/anthropic/claude-3-5-sonnet"
model="openrouter/meta-llama/llama-3-70b-instruct"

Example:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openrouter/google/gemini-2.0-flash"
)

optimizer = MetaPromptOptimizer(model="openrouter/anthropic/claude-3-5-sonnet")

Environment Variables Reference

Provider	Environment Variable	How to Get
OpenAI	`OPENAI_API_KEY`	platform.openai.com/api-keys
Anthropic	`ANTHROPIC_API_KEY`	console.anthropic.com
Google Gemini	`GOOGLE_API_KEY` or `GEMINI_API_KEY`	aistudio.google.com/apikey
Azure OpenAI	`AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`	Azure Portal
OpenRouter	`OPENROUTER_API_KEY`	openrouter.ai/keys
Ollama	None required (local)	ollama.ai

Model Parameters

You can pass additional parameters to control model behavior:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

# Configure model parameters for the ChatPrompt
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="openai/gpt-4o-mini",
    model_parameters={
        "temperature": 0.7,
        "max_tokens": 1000,
        "top_p": 0.9
    }
)

# Configure model parameters for the Optimizer
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o",
    model_parameters={
        "temperature": 0.1,  # Lower temperature for more consistent optimization
        "max_tokens": 4096
    }
)

Mixing Providers

You can use different providers for the ChatPrompt and Optimizer:

Python

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

# Use a cost-effective model for prompt evaluation
prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "{question}"}
    ],
    model="gemini/gemini-2.0-flash"  # Fast and affordable
)

# Use a powerful model for optimization reasoning
optimizer = MetaPromptOptimizer(
    model="openai/gpt-4o"  # Best reasoning capabilities
)

Troubleshooting

AuthenticationError: Invalid API key

Ensure your API key is correctly set in the environment:

Bash

# Check if the key is set
echo $OPENAI_API_KEY

# Set it if missing
export OPENAI_API_KEY="sk-..."

Model not found

Verify the model name follows the LiteLLM format provider/model-name:

Python

# ✅ Correct
model="openai/gpt-4o"
model="gemini/gemini-2.0-flash"

# ❌ Incorrect
model="gpt-4o"  # Missing provider prefix
model="google/gemini-2.0-flash"  # Wrong provider name (use 'gemini')

Rate limiting errors

If you encounter rate limits, try:

Reducing n_threads in the optimizer
Using a model with higher rate limits
Adding delays between API calls

Next Steps

Learn about Optimization Concepts
Explore different Optimization Algorithms
Check out the API Reference