Few-Shot Bayesian Optimizer

The FewShotBayesianOptimizer is a sophisticated prompt optimization tool adds relevant examples from your sample questions to the system prompt using Bayesian optimization techniques.

The FewShotBayesianOptimizer uses Bayesian optimization to find the optimal set and number of few-shot examples to include with your base instruction prompt for chat models. It Uses Optuna, a hyperparameter optimization framework, to guide the search for the optimal set and number of few-shot examples.

You can use the FewShotBayesianOptimizer to optimize a prompt by following these steps:

from opik_optimizer import FewShotBayesianOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer import datasets, ChatPrompt

# Initialize optimizer
optimizer = FewShotBayesianOptimizer(
    model="openai/gpt-4",
    model_parameters={
        "temperature": 0.1,
        "max_tokens": 5000
    },
)

# Prepare dataset
dataset = datasets.hotpot(count=300)

# Define metric and prompt (see docs for more options)
def levenshtein_ratio(dataset_item, llm_output):
    return LevenshteinRatio().score(reference=dataset_item["answer"], output=llm_output)

prompt = ChatPrompt(
    messages=[
        {"role": "system", "content": "Provide an answer to the question."},
        {"role": "user", "content": "{question}"}
    ]
)

# Run optimization
results = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=levenshtein_ratio,
    n_samples=100
)

# Access results
results.display()

The optimizer has the following parameters:

• model (str, default: openai/gpt-5-nano) — LiteLLM model name for optimizer's internal reasoning (generating few-shot templates)

• model_parameters (dict[str, typing.Any] | None) — Optional dict of LiteLLM parameters for optimizer's internal LLM calls. Common params: temperature, max_tokens, max_completion_tokens, top_p.

• min_examples (int, default: 2) — Minimum number of examples to include in the prompt

• max_examples (int, default: 8) — Maximum number of examples to include in the prompt

• n_threads (int, default: 8) — Number of threads for parallel evaluation

• verbose (int, default: 1) — Controls internal logging/progress bars (0=off, 1=on)

• seed (int, default: 42) — Random seed for reproducibility

The optimize_prompt method has the following parameters:

• prompt (ChatPrompt) — The prompt to optimize

• dataset (Dataset) — Opik Dataset to optimize on

• metric (Callable) — Metric function to evaluate on

• experiment_config (dict | None) — Optional configuration for the experiment, useful to log additional metadata

• n_samples (int | float | str | None) — Number of dataset items to use per evaluation. Use counts (e.g., 50), fractions (e.g., 0.1), percentages (e.g., "10%"), or "all"/"full"/None for the full dataset.

• n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches (defaults to n_samples).

• n_samples_strategy (str | None) — Sampling strategy for subsampling (default: "random_sorted").

• auto_continue (bool, default: False) — Whether to auto-continue optimization

• agent_class (type[opik_optimizer.optimizable_agent.OptimizableAgent] | None) — Optional agent class to use

• project_name (str, default: Optimization) — Opik project name for logging traces (default: "Optimization")

• max_trials (int, default: 10) — Number of trials for Bayesian Optimization (default: 10)

• args (Any) —

• kwargs (Any) —

There are two models to consider when using the FewShotBayesianOptimizer:

• FewShotBayesianOptimizer.model: The model used to generate the few-shot template and placeholder.
• ChatPrompt.model: The model used to evaluate the prompt.

The model parameter accepts any LiteLLM-supported model string (e.g., "gpt-4o", "azure/gpt-4", "anthropic/claude-3-opus", "gemini/gemini-1.5-pro"). You can also pass in extra model parameters using the model_parameters parameter:

optimizer = FewShotBayesianOptimizer(
    model="anthropic/claude-3-opus-20240229",
    model_parameters={
        "temperature": 0.7,
        "max_tokens": 4096
    }
)

1. Explore specific Optimizers for algorithm details.
2. Refer to the FAQ for common questions and troubleshooting.
3. Refer to the API Reference for detailed configuration options.