Skip to main content
Opik Documentation
current

Search documentation

Type to search this documentation.

On this pageOverview

Opik Agent Optimizer API Reference

Technical SDK reference guide

The Opik Agent Optimizer SDK provides a comprehensive set of tools for optimizing LLM prompts and agents. This reference guide documents the standardized API that all optimizers follow, ensuring consistency and interoperability across different optimization algorithms.

  • Standardized API: All optimizers follow the same interface for optimize_prompt() methods
  • Multiple Algorithms: Support for various optimization strategies including evolutionary, few-shot, meta-prompt, and GEPA
  • MCP Support: Built-in support for Model Context Protocol tool calling and optimization
  • Consistent Results: All optimizers return standardized OptimizationResult objects
  • Counter Tracking: Built-in LLM and tool call counters for monitoring usage
  • Backward Compatibility: All original parameters preserved through kwargs extraction
  • Deprecation Warnings: Clear warnings for deprecated parameters with migration guidance

The SDK provides several optimizer classes that all inherit from BaseOptimizer and implement the same standardized interface:

  • ParameterOptimizer: Optimizes LLM call parameters (temperature, top_p, etc.) using Bayesian optimization
  • FewShotBayesianOptimizer: Uses few-shot learning with Bayesian optimization
  • MetaPromptOptimizer: Employs meta-prompting techniques for optimization
  • EvolutionaryOptimizer: Uses genetic algorithms for prompt evolution
  • GepaOptimizer: Leverages GEPA (Genetic-Pareto) optimization approach
  • HRPO (Hierarchical Reflective Prompt Optimizer): Uses hierarchical root cause analysis for targeted prompt refinement

All optimizers implement these core methods with identical signatures:

Python
def optimize_prompt(
    self,
    prompt: ChatPrompt | dict[str, ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    **kwargs: Any,
) -> OptimizationResult

The following parameters are deprecated and will be removed in future versions:

  • num_threads in optimizer constructors: Use n_threads instead
Python
# ❌ Deprecated
optimizer = FewShotBayesianOptimizer(
    model="gpt-4o-mini",
    num_threads=16,  # Deprecated
)

# ✅ Correct
optimizer = FewShotBayesianOptimizer(
    model="gpt-4o-mini",
    n_threads=16,  # Use n_threads instead
)
Python
FewShotBayesianOptimizer(
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    min_examples: int = 2,
    max_examples: int = 8,
    n_threads: int = 12,
    verbose: int = 1,
    seed: int = 42,
    name: str | None = None,
    enable_columnar_selection: bool = True,
    enable_diversity: bool = True,
    enable_multivariate_tpe: bool = True,
    enable_optuna_pruning: bool = True,
    prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95
)

Parameters:

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • min_examples (int, default: 2) —

  • max_examples (int, default: 8) —

  • n_threads (int, default: 12) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • name (str | None) —

  • enable_columnar_selection (bool, default: True) —

  • enable_diversity (bool, default: True) —

  • enable_multivariate_tpe (bool, default: True) —

  • enable_optuna_pruning (bool, default: True) —

  • prompt_overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool = False,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool, default: False) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_optimizer_metadata()
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    optimize_prompts: bool | str | list[str] | None = 'system',
    optimize_tools: bool | dict[str, bool] | None = None,
    args: Any,
    kwargs: Any
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt to optimize (single ChatPrompt or dict of prompts)

  • dataset (Dataset) — Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset_training. For now, it serves as the training dataset parameter.

  • metric (MetricFunction) — A metric function with signature (dataset_item, llm_output) -> float

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional agent for prompt execution (defaults to LiteLLMAgent)

  • experiment_config (dict | None) — Optional configuration for the experiment

  • n_samples (int | float | str | None) — Number of samples to use for evaluation

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • auto_continue (bool, default: False) — Whether to continue optimization automatically

  • project_name (str | None) — Opik project name for logging traces (defaults to OPIK_PROJECT_NAME env or "Optimization")

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset for ranking candidates

  • max_trials (int, default: 10) — Maximum number of optimization trials

  • allow_tool_use (bool, default: True) — Whether tools may be executed during evaluation (default True)

  • optimize_prompts (bool | str | list[str] | None, default: system) — Which prompt roles to allow for optimization

  • optimize_tools (bool | dict[str, bool] | None) — Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support.

  • args (Any) —

  • kwargs (Any) —

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
run_optimization(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context with prompts, dataset, metric, etc.
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
GepaOptimizer(
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    n_threads: int = 12,
    verbose: int = 1,
    seed: int = 42,
    name: str | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95,
    prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None
)

Parameters:

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • n_threads (int, default: 12) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • name (str | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

  • prompt_overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool | None = None,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool | None) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_optimizer_metadata()
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    optimize_prompts: bool | str | list[str] | None = 'system',
    optimize_tools: bool | dict[str, bool] | None = None,
    args: Any,
    kwargs: Any
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt to optimize (single ChatPrompt or dict of prompts)

  • dataset (Dataset) — Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset_training. For now, it serves as the training dataset parameter.

  • metric (MetricFunction) — A metric function with signature (dataset_item, llm_output) -> float

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional agent for prompt execution (defaults to LiteLLMAgent)

  • experiment_config (dict | None) — Optional configuration for the experiment

  • n_samples (int | float | str | None) — Number of samples to use for evaluation

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • auto_continue (bool, default: False) — Whether to continue optimization automatically

  • project_name (str | None) — Opik project name for logging traces (defaults to OPIK_PROJECT_NAME env or "Optimization")

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset for ranking candidates

  • max_trials (int, default: 10) — Maximum number of optimization trials

  • allow_tool_use (bool, default: True) — Whether tools may be executed during evaluation (default True)

  • optimize_prompts (bool | str | list[str] | None, default: system) — Which prompt roles to allow for optimization

  • optimize_tools (bool | dict[str, bool] | None) — Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support.

  • args (Any) —

  • kwargs (Any) —

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
run_optimization(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context with prompts, dataset, metric, etc.
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
MetaPromptOptimizer(
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    prompts_per_round: int = 4,
    enable_context: bool = True,
    num_task_examples: int = 5,
    task_context_columns: list[str] | None = None,
    n_threads: int = 12,
    verbose: int = 1,
    seed: int = 42,
    name: str | None = None,
    use_hall_of_fame: bool = True,
    prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95
)

Parameters:

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • prompts_per_round (int, default: 4) —

  • enable_context (bool, default: True) —

  • num_task_examples (int, default: 5) —

  • task_context_columns (list[str] | None) —

  • n_threads (int, default: 12) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • name (str | None) —

  • use_hall_of_fame (bool, default: True) —

  • prompt_overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool | None = None,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool | None) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_optimizer_metadata()
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    optimize_prompts: bool | str | list[str] | None = 'system',
    optimize_tools: bool | dict[str, bool] | None = None,
    args: Any,
    kwargs: Any
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt to optimize (single ChatPrompt or dict of prompts)

  • dataset (Dataset) — Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset_training. For now, it serves as the training dataset parameter.

  • metric (MetricFunction) — A metric function with signature (dataset_item, llm_output) -> float

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional agent for prompt execution (defaults to LiteLLMAgent)

  • experiment_config (dict | None) — Optional configuration for the experiment

  • n_samples (int | float | str | None) — Number of samples to use for evaluation

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • auto_continue (bool, default: False) — Whether to continue optimization automatically

  • project_name (str | None) — Opik project name for logging traces (defaults to OPIK_PROJECT_NAME env or "Optimization")

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset for ranking candidates

  • max_trials (int, default: 10) — Maximum number of optimization trials

  • allow_tool_use (bool, default: True) — Whether tools may be executed during evaluation (default True)

  • optimize_prompts (bool | str | list[str] | None, default: system) — Which prompt roles to allow for optimization

  • optimize_tools (bool | dict[str, bool] | None) — Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support.

  • args (Any) —

  • kwargs (Any) —

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
run_optimization(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context with prompts, dataset, metric, etc.
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
EvolutionaryOptimizer(
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    population_size: int = 30,
    num_generations: int = 15,
    mutation_rate: float = 0.2,
    crossover_rate: float = 0.8,
    tournament_size: int = 4,
    elitism_size: int = 3,
    adaptive_mutation: bool = True,
    enable_moo: bool = True,
    enable_llm_crossover: bool = True,
    enable_semantic_crossover: bool = False,
    output_style_guidance: str | None = None,
    infer_output_style: bool = False,
    n_threads: int = 12,
    verbose: int = 1,
    seed: int = 42,
    name: str | None = None,
    prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95
)

Parameters:

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • population_size (int, default: 30) —

  • num_generations (int, default: 15) —

  • mutation_rate (float, default: 0.2) —

  • crossover_rate (float, default: 0.8) —

  • tournament_size (int, default: 4) —

  • elitism_size (int, default: 3) —

  • adaptive_mutation (bool, default: True) —

  • enable_moo (bool, default: True) —

  • enable_llm_crossover (bool, default: True) —

  • enable_semantic_crossover (bool, default: False) —

  • output_style_guidance (str | None) —

  • infer_output_style (bool, default: False) —

  • n_threads (int, default: 12) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • name (str | None) —

  • prompt_overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool | None = None,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool | None) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_optimizer_metadata()
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    optimize_prompts: bool | str | list[str] | None = 'system',
    optimize_tools: bool | dict[str, bool] | None = None,
    args: Any,
    kwargs: Any
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt to optimize (single ChatPrompt or dict of prompts)

  • dataset (Dataset) — Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset_training. For now, it serves as the training dataset parameter.

  • metric (MetricFunction) — A metric function with signature (dataset_item, llm_output) -> float

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional agent for prompt execution (defaults to LiteLLMAgent)

  • experiment_config (dict | None) — Optional configuration for the experiment

  • n_samples (int | float | str | None) — Number of samples to use for evaluation

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • auto_continue (bool, default: False) — Whether to continue optimization automatically

  • project_name (str | None) — Opik project name for logging traces (defaults to OPIK_PROJECT_NAME env or "Optimization")

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset for ranking candidates

  • max_trials (int, default: 10) — Maximum number of optimization trials

  • allow_tool_use (bool, default: True) — Whether tools may be executed during evaluation (default True)

  • optimize_prompts (bool | str | list[str] | None, default: system) — Which prompt roles to allow for optimization

  • optimize_tools (bool | dict[str, bool] | None) — Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support.

  • args (Any) —

  • kwargs (Any) —

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
run_optimization(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context with prompts, dataset, metric, etc.
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
HierarchicalReflectiveOptimizer(
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    reasoning_model: str | None = None,
    reasoning_model_parameters: dict[str, typing.Any] | None = None,
    max_parallel_batches: int = 5,
    batch_size: int = 25,
    convergence_threshold: float = 0.01,
    n_threads: int = 12,
    verbose: int = 1,
    seed: int = 42,
    name: str | None = None,
    prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95
)

Parameters:

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • reasoning_model (str | None) —

  • reasoning_model_parameters (dict[str, typing.Any] | None) —

  • max_parallel_batches (int, default: 5) —

  • batch_size (int, default: 25) —

  • convergence_threshold (float, default: 0.01) —

  • n_threads (int, default: 12) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • name (str | None) —

  • prompt_overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool | None = None,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool | None) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_optimizer_metadata()
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    optimize_prompts: bool | str | list[str] | None = 'system',
    optimize_tools: bool | dict[str, bool] | None = None,
    args: Any,
    kwargs: Any
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt to optimize (single ChatPrompt or dict of prompts)

  • dataset (Dataset) — Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset_training. For now, it serves as the training dataset parameter.

  • metric (MetricFunction) — A metric function with signature (dataset_item, llm_output) -> float

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional agent for prompt execution (defaults to LiteLLMAgent)

  • experiment_config (dict | None) — Optional configuration for the experiment

  • n_samples (int | float | str | None) — Number of samples to use for evaluation

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • auto_continue (bool, default: False) — Whether to continue optimization automatically

  • project_name (str | None) — Opik project name for logging traces (defaults to OPIK_PROJECT_NAME env or "Optimization")

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset for ranking candidates

  • max_trials (int, default: 10) — Maximum number of optimization trials

  • allow_tool_use (bool, default: True) — Whether tools may be executed during evaluation (default True)

  • optimize_prompts (bool | str | list[str] | None, default: system) — Which prompt roles to allow for optimization

  • optimize_tools (bool | dict[str, bool] | None) — Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support.

  • args (Any) —

  • kwargs (Any) —

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
run_optimization(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context with prompts, dataset, metric, etc.
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
ParameterOptimizer(
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    default_n_trials: int = 20,
    local_search_ratio: float = 0.3,
    local_search_scale: float = 0.2,
    n_threads: int = 12,
    verbose: int = 1,
    seed: int = 42,
    name: str | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95
)

Parameters:

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • default_n_trials (int, default: 20) —

  • local_search_ratio (float, default: 0.3) —

  • local_search_scale (float, default: 0.2) —

  • n_threads (int, default: 12) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • name (str | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool | None = None,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool | None) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_optimizer_metadata()
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_parameter(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    parameter_space: opik_optimizer.algorithms.parameter_optimizer.ops.search_ops.ParameterSearchSpace | collections.abc.Mapping[str, typing.Any],
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    experiment_config: dict | None = None,
    max_trials: int | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    project_name: str = 'Optimization',
    sampler: optuna.samplers._base.BaseSampler | None = None,
    callbacks: list[collections.abc.Callable[[optuna.study.study.Study, optuna.trial._frozen.FrozenTrial], None]] | None = None,
    timeout: float | None = None,
    local_trials: int | None = None,
    local_search_scale: float | None = None,
    optimization_id: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt or dict of prompts to evaluate with tuned parameters. When a dict is provided, parameters are optimized independently for each prompt.

  • dataset (Dataset) — Dataset providing evaluation examples

  • metric (MetricFunction) — Objective function to maximize

  • parameter_space (opik_optimizer.algorithms.parameter_optimizer.ops.search_ops.ParameterSearchSpace | collections.abc.Mapping[str, typing.Any]) — Definition of the search space for tunable parameters. For multi-prompt, params without a prefix are expanded per prompt. Params already prefixed (e.g., 'analyze.temperature') are kept as-is.

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset. Note: Due to the internal implementation of ParameterOptimizer, this parameter is currently not fully utilized and we recommend not using it for this optimizer.

  • experiment_config (dict | None) — Optional experiment metadata

  • max_trials (int | None) — Total number of trials (if None, uses default_n_trials)

  • n_samples (int | float | str | None) — Number of dataset samples to evaluate per trial (None for all)

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional custom agent instance to execute evaluations

  • project_name (str, default: Optimization) — Opik project name for logging traces (default: "Optimization")

  • sampler (optuna.samplers._base.BaseSampler | None) — Optuna sampler to use (default: TPESampler with seed)

  • callbacks (list[collections.abc.Callable[[optuna.study.study.Study, optuna.trial._frozen.FrozenTrial], None]] | None) — List of callback functions for Optuna study

  • timeout (float | None) — Maximum time in seconds for optimization

  • local_trials (int | None) — Number of trials for local search (overrides local_search_ratio)

  • local_search_scale (float | None) — Scale factor for local search narrowing (0.0-1.0)

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run; when provided it must be a valid UUIDv7 string.

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
ParameterSearchSpace(
    parameters: list[opik_optimizer.algorithms.parameter_optimizer.ops.search_ops.ParameterSpec] = PydanticUndefined
)

Parameters:

  • parameters (list[opik_optimizer.algorithms.parameter_optimizer.ops.search_ops.ParameterSpec], default: PydanticUndefined) —
Python
ParameterSpec(
    name: <class 'str'>,
    description: str | None = None,
    distribution: <enum 'ParameterType'>,
    low: float | None = None,
    high: float | None = None,
    step: float | None = None,
    scale: Literal['linear', 'log'] = 'linear',
    choices: list[Any] | None = None,
    target: str | collections.abc.Sequence[str] | None = None,
    default: Any | None = None
)

Parameters:

  • name (<class 'str'>, default: PydanticUndefined) —

  • description (str | None) —

  • distribution (<enum 'ParameterType'>, default: PydanticUndefined) —

  • low (float | None) —

  • high (float | None) —

  • step (float | None) —

  • scale (Literal['linear', 'log'], default: linear) —

  • choices (list[Any] | None) —

  • target (str | collections.abc.Sequence[str] | None) —

  • default (Any | None) —

Python
ParameterType(
    args: Any,
    kwds: Any
)

Parameters:

  • args (Any) —

  • kwds (Any) —

Python
BaseOptimizer(
    model: str,
    verbose: int = 1,
    seed: int = 42,
    model_parameters: dict[str, typing.Any] | None = None,
    reasoning_model: str | None = None,
    reasoning_model_parameters: dict[str, typing.Any] | None = None,
    name: str | None = None,
    skip_perfect_score: bool = True,
    perfect_score: float = 0.95,
    prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None,
    display: opik_optimizer.utils.display.run.RunDisplay | None = None
)

Parameters:

  • model (str) —

  • verbose (int, default: 1) —

  • seed (int, default: 42) —

  • model_parameters (dict[str, typing.Any] | None) —

  • reasoning_model (str | None) —

  • reasoning_model_parameters (dict[str, typing.Any] | None) —

  • name (str | None) —

  • skip_perfect_score (bool, default: True) —

  • perfect_score (float, default: 0.95) —

  • prompt_overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) —

  • display (opik_optimizer.utils.display.run.RunDisplay | None) —

Python
begin_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
cleanup()
Python
evaluate(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) — Optimization context for this run.

  • prompts (dict) — Dict of named prompts to evaluate (e.g., {"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry.

  • experiment_config (dict[str, typing.Any] | None) — Optional experiment configuration.

  • sampling_tag (str | None) — Optional sampling tag for deterministic subsampling per candidate.

Python
evaluate_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    n_threads: int | None = None,
    verbose: int = 1,
    dataset_item_ids: list[str] | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    seed: int | None = None,
    return_evaluation_result: bool = False,
    allow_tool_use: bool | None = None,
    use_evaluate_on_dict_items: bool | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) —

  • dataset (Dataset) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • n_threads (int | None) —

  • verbose (int, default: 1) —

  • dataset_item_ids (list[str] | None) —

  • experiment_config (dict | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • seed (int | None) —

  • return_evaluation_result (bool, default: False) —

  • allow_tool_use (bool | None) —

  • use_evaluate_on_dict_items (bool | None) —

  • sampling_tag (str | None) —

Python
evaluate_with_result(
    context: OptimizationContext,
    prompts: dict,
    experiment_config: dict[str, typing.Any] | None = None,
    empty_score: float | None = None,
    n_samples: int | float | str | None = None,
    n_samples_strategy: str | None = None,
    sampling_tag: str | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • experiment_config (dict[str, typing.Any] | None) —

  • empty_score (float | None) —

  • n_samples (int | float | str | None) —

  • n_samples_strategy (str | None) —

  • sampling_tag (str | None) —

Python
finish_candidate(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
finish_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
get_config(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_default_prompt(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
get_history_entries()
Python
get_history_rounds()
Python
get_metadata(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
get_prompt(
    key: str,
    fmt: Any
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (Any) —

Python
list_prompts()
Python
on_trial(
    context: OptimizationContext,
    prompts: dict,
    score: float,
    prev_best_score: float | None = None
)

Parameters:

  • context (OptimizationContext) —

  • prompts (dict) —

  • score (float) —

  • prev_best_score (float | None) —

Python
optimize_mcp(
    args: Any,
    kwargs: Any
)

Parameters:

  • args (Any) —

  • kwargs (Any) —

Python
optimize_prompt(
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    dataset: Dataset,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None,
    experiment_config: dict | None = None,
    n_samples: int | float | str | None = None,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str | None = None,
    auto_continue: bool = False,
    project_name: str | None = None,
    optimization_id: str | None = None,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None,
    max_trials: int = 10,
    allow_tool_use: bool = True,
    optimize_prompts: bool | str | list[str] | None = 'system',
    optimize_tools: bool | dict[str, bool] | None = None,
    args: Any,
    kwargs: Any
)

Parameters:

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt]) — The prompt to optimize (single ChatPrompt or dict of prompts)

  • dataset (Dataset) — Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset_training. For now, it serves as the training dataset parameter.

  • metric (MetricFunction) — A metric function with signature (dataset_item, llm_output) -> float

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) — Optional agent for prompt execution (defaults to LiteLLMAgent)

  • experiment_config (dict | None) — Optional configuration for the experiment

  • n_samples (int | float | str | None) — Number of samples to use for evaluation

  • n_samples_minibatch (int | None) — Optional number of samples for inner-loop minibatches

  • n_samples_strategy (str | None) — Sampling strategy name (default "random_sorted")

  • auto_continue (bool, default: False) — Whether to continue optimization automatically

  • project_name (str | None) — Opik project name for logging traces (defaults to OPIK_PROJECT_NAME env or "Optimization")

  • optimization_id (str | None) — Optional ID to use when creating the Opik optimization run

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) — Optional validation dataset for ranking candidates

  • max_trials (int, default: 10) — Maximum number of optimization trials

  • allow_tool_use (bool, default: True) — Whether tools may be executed during evaluation (default True)

  • optimize_prompts (bool | str | list[str] | None, default: system) — Which prompt roles to allow for optimization

  • optimize_tools (bool | dict[str, bool] | None) — Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support.

  • args (Any) —

  • kwargs (Any) —

Python
post_baseline(
    context: OptimizationContext,
    score: float
)

Parameters:

  • context (OptimizationContext) —

  • score (float) —

Python
post_optimize(
    context: OptimizationContext,
    result: OptimizationResult
)

Parameters:

  • context (OptimizationContext) —

  • result (OptimizationResult) —

Python
post_round(
    round_handle: Any,
    context: opik_optimizer.core.state.OptimizationContext | None = None,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    dataset_split: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None
)

Parameters:

  • round_handle (Any) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • dataset_split (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

Python
post_trial(
    context: OptimizationContext,
    candidate_handle: Any,
    score: float | None,
    metrics: dict[str, typing.Any] | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    trial_index: int | None = None,
    timestamp: str | None = None,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate_handle (Any) —

  • score (float | None) —

  • metrics (dict[str, typing.Any] | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • trial_index (int | None) —

  • timestamp (str | None) —

  • round_handle (typing.Any | None) —

Python
pre_baseline(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) —
Python
pre_optimize(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context
Python
pre_round(
    context: OptimizationContext,
    extras: Any
)

Parameters:

  • context (OptimizationContext) —

  • extras (Any) —

Python
pre_trial(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
record_candidate_entry(
    prompt_or_payload: Any,
    score: float | None = None,
    id: str | None = None,
    metrics: dict[str, typing.Any] | None = None,
    notes: str | None = None,
    extra: dict[str, typing.Any] | None = None,
    context: opik_optimizer.core.state.OptimizationContext | None = None
)

Parameters:

  • prompt_or_payload (Any) —

  • score (float | None) —

  • id (str | None) —

  • metrics (dict[str, typing.Any] | None) —

  • notes (str | None) —

  • extra (dict[str, typing.Any] | None) —

  • context (opik_optimizer.core.state.OptimizationContext | None) —

Python
run_optimization(
    context: OptimizationContext
)

Parameters:

  • context (OptimizationContext) — The optimization context with prompts, dataset, metric, etc.
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_candidate(
    context: OptimizationContext,
    candidate: Any,
    round_handle: typing.Any | None = None
)

Parameters:

  • context (OptimizationContext) —

  • candidate (Any) —

  • round_handle (typing.Any | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
ChatPrompt(
    name: str = 'chat-prompt',
    system: str | None = None,
    user: str | None = None,
    messages: list[dict[str, typing.Any]] | None = None,
    tools: list[dict[str, typing.Any]] | collections.abc.Mapping[str, typing.Any] | None = None,
    function_map: collections.abc.Mapping[str, collections.abc.Callable[..., typing.Any]] | None = None,
    model: str = 'openai/gpt-5-nano',
    model_parameters: dict[str, typing.Any] | None = None,
    model_kwargs: dict[str, typing.Any] | None = None,
    kwargs: Any
)

Parameters:

  • name (str, default: chat-prompt) —

  • system (str | None) — the system prompt

  • user (str | None) —

  • messages (list[dict[str, typing.Any]] | None) — a list of dictionaries with role/content, with a content containing {input-dataset-field}

  • tools (list[dict[str, typing.Any]] | collections.abc.Mapping[str, typing.Any] | None) —

  • function_map (collections.abc.Mapping[str, collections.abc.Callable[..., typing.Any]] | None) —

  • model (str, default: openai/gpt-5-nano) —

  • model_parameters (dict[str, typing.Any] | None) —

  • model_kwargs (dict[str, typing.Any] | None) —

  • kwargs (Any) —

Python
copy()
Python
get_messages(
    dataset_item: dict[str, typing.Any] | None = None
)

Parameters:

  • dataset_item (dict[str, typing.Any] | None) —
Python
replace_in_messages(
    messages: list,
    label: str,
    value: str
)

Parameters:

  • messages (list) —

  • label (str) —

  • value (str) —

Python
set_messages(
    messages: list
)

Parameters:

  • messages (list) —
Python
to_dict()
Python
AlgorithmResult(
    best_prompts: dict,
    best_score: float,
    history: Sequence = <factory>,
    metadata: dict = <factory>
)

Parameters:

  • best_prompts (dict) —

  • best_score (float) —

  • history (Sequence, default: ) —

  • metadata (dict, default: ) —

Python
OptimizationResult(
    schema_version: <class 'str'> = 'v1',
    details_version: <class 'str'> = 'v1',
    optimizer: <class 'str'> = 'Optimizer',
    prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt],
    score: <class 'float'>,
    metric_name: <class 'str'>,
    optimization_id: str | None = None,
    dataset_id: str | None = None,
    initial_prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt] | None = None,
    initial_score: float | None = None,
    details: dict[str, Any] = PydanticUndefined,
    history: list[dict[str, Any]] = [],
    llm_calls: int | None = None,
    llm_calls_tools: int | None = None,
    llm_cost_total: float | None = None,
    llm_token_usage_total: dict[str, int] | None = None
)

Parameters:

  • schema_version (<class 'str'>, default: v1) —

  • details_version (<class 'str'>, default: v1) —

  • optimizer (<class 'str'>, default: Optimizer) —

  • prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], default: PydanticUndefined) —

  • score (<class 'float'>, default: PydanticUndefined) —

  • metric_name (<class 'str'>, default: PydanticUndefined) —

  • optimization_id (str | None) —

  • dataset_id (str | None) —

  • initial_prompt (opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt] | None) —

  • initial_score (float | None) —

  • details (dict[str, Any], default: PydanticUndefined) —

  • history (list[dict[str, Any]], default: []) —

  • llm_calls (int | None) —

  • llm_calls_tools (int | None) —

  • llm_cost_total (float | None) —

  • llm_token_usage_total (dict[str, int] | None) —

Python
OptimizationContext(
    prompts: dict,
    initial_prompts: dict,
    is_single_prompt_optimization: bool,
    dataset: Dataset,
    evaluation_dataset: Dataset,
    validation_dataset: opik.api_objects.dataset.dataset.Dataset | None,
    metric: MetricFunction,
    agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None,
    optimization: opik.api_objects.optimization.optimization.Optimization | None,
    optimization_id: str | None,
    experiment_config: dict[str, typing.Any] | None,
    n_samples: int | float | str | None,
    max_trials: int,
    project_name: str,
    n_samples_minibatch: int | None = None,
    n_samples_strategy: str = 'random_sorted',
    allow_tool_use: bool = True,
    baseline_score: float | None = None,
    extra_params: dict = <factory>,
    trials_completed: int = 0,
    should_stop: bool = False,
    finish_reason: Optional = None,
    current_best_score: float | None = None,
    current_best_prompt: dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt] | None = None,
    dataset_split: str | None = None,
    scoring_health: dict[str, int] | None = None
)

Parameters:

  • prompts (dict) —

  • initial_prompts (dict) —

  • is_single_prompt_optimization (bool) —

  • dataset (Dataset) —

  • evaluation_dataset (Dataset) —

  • validation_dataset (opik.api_objects.dataset.dataset.Dataset | None) —

  • metric (MetricFunction) —

  • agent (opik_optimizer.agents.optimizable_agent.OptimizableAgent | None) —

  • optimization (opik.api_objects.optimization.optimization.Optimization | None) —

  • optimization_id (str | None) —

  • experiment_config (dict[str, typing.Any] | None) —

  • n_samples (int | float | str | None) —

  • max_trials (int) —

  • project_name (str) —

  • n_samples_minibatch (int | None) —

  • n_samples_strategy (str, default: random_sorted) —

  • allow_tool_use (bool, default: True) —

  • baseline_score (float | None) —

  • extra_params (dict, default: ) —

  • trials_completed (int, default: 0) —

  • should_stop (bool, default: False) —

  • finish_reason (Optional) —

  • current_best_score (float | None) —

  • current_best_prompt (dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt] | None) —

  • dataset_split (str | None) —

  • scoring_health (dict[str, int] | None) —

Python
OptimizationHistoryState(
    context: Any = None
)

Parameters:

  • context (Any) —
Python
clear()
Python
end_round(
    round_handle: Any,
    best_score: float | None = None,
    best_candidate: typing.Any | None = None,
    best_prompt: typing.Any | None = None,
    stop_reason: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    pareto_front: list[dict[str, typing.Any]] | None = None,
    selection_meta: dict[str, typing.Any] | None = None,
    dataset_split: str | None = None
)

Parameters:

  • round_handle (Any) —

  • best_score (float | None) —

  • best_candidate (typing.Any | None) —

  • best_prompt (typing.Any | None) —

  • stop_reason (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • pareto_front (list[dict[str, typing.Any]] | None) —

  • selection_meta (dict[str, typing.Any] | None) —

  • dataset_split (str | None) —

Python
finalize_stop(
    stop_reason: str | None = None
)

Parameters:

  • stop_reason (str | None) —
Python
get_entries()
Python
get_rounds()
Python
record_trial(
    round_handle: Any,
    score: float | None,
    candidate: typing.Any | None = None,
    trial_index: int | None = None,
    metrics: dict[str, typing.Any] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    timestamp: str | None = None,
    stop_reason: str | None = None,
    candidate_id_prefix: str | None = None
)

Parameters:

  • round_handle (Any) —

  • score (float | None) —

  • candidate (typing.Any | None) —

  • trial_index (int | None) —

  • metrics (dict[str, typing.Any] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • timestamp (str | None) —

  • stop_reason (str | None) —

  • candidate_id_prefix (str | None) —

Python
set_context(
    context: Any
)

Parameters:

  • context (Any) —
Python
set_default_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
set_pareto_front(
    pareto_front: list[dict[str, typing.Any]] | None
)

Parameters:

  • pareto_front (list[dict[str, typing.Any]] | None) —
Python
set_selection_meta(
    selection_meta: dict[str, typing.Any] | None
)

Parameters:

  • selection_meta (dict[str, typing.Any] | None) —
Python
start_round(
    round_index: int | None = None,
    extras: dict[str, typing.Any] | None = None,
    timestamp: str | None = None
)

Parameters:

  • round_index (int | None) —

  • extras (dict[str, typing.Any] | None) —

  • timestamp (str | None) —

Python
with_dataset_split(
    dataset_split: str | None
)

Parameters:

  • dataset_split (str | None) —
Python
OptimizationRound(
    round_index: int,
    trials: list = <factory>,
    best_score: float | None = None,
    best_so_far: float | None = None,
    best_prompt: typing.Any | None = None,
    best_candidate: typing.Any | None = None,
    candidates: list[dict[str, typing.Any]] | None = None,
    generated_prompts: list[dict[str, typing.Any]] | None = None,
    stop_reason: str | None = None,
    stopped: bool | None = None,
    dataset_split: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    timestamp: str = <factory>
)

Parameters:

  • round_index (int) —

  • trials (list, default: ) —

  • best_score (float | None) —

  • best_so_far (float | None) —

  • best_prompt (typing.Any | None) —

  • best_candidate (typing.Any | None) —

  • candidates (list[dict[str, typing.Any]] | None) —

  • generated_prompts (list[dict[str, typing.Any]] | None) —

  • stop_reason (str | None) —

  • stopped (bool | None) —

  • dataset_split (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • timestamp (str, default: ) —

Python
to_dict()
Python
OptimizationTrial(
    trial_index: int | None,
    score: float | None,
    candidate: Any,
    metrics: dict[str, typing.Any] | None = None,
    dataset: str | None = None,
    dataset_split: str | None = None,
    candidate_id: str | None = None,
    extras: dict[str, typing.Any] | None = None,
    timestamp: str = <factory>
)

Parameters:

  • trial_index (int | None) —

  • score (float | None) —

  • candidate (Any) —

  • metrics (dict[str, typing.Any] | None) —

  • dataset (str | None) —

  • dataset_split (str | None) —

  • candidate_id (str | None) —

  • extras (dict[str, typing.Any] | None) —

  • timestamp (str, default: ) —

Python
to_dict()
Python
OptimizableAgent(
    prompt: Any = None,
    project_name: Any = None,
    kwargs: Any
)

Parameters:

  • prompt (Any) —

  • project_name (Any) —

  • kwargs (Any) —

Python
init_agent(
    prompt: Any
)

Parameters:

  • prompt (Any) —
Python
init_llm()
Python
invoke(
    messages: list,
    seed: int | None = None
)

Parameters:

  • messages (list) — List of message dictionaries

  • seed (int | None) — Optional seed for reproducibility

Python
invoke_agent(
    prompts: Any,
    dataset_item: Any,
    allow_tool_use: Any = False,
    seed: Any = None
)

Parameters:

  • prompts (Any) —

  • dataset_item (Any) —

  • allow_tool_use (Any, default: False) —

  • seed (Any) —

Python
invoke_agent_candidates(
    prompts: Any,
    dataset_item: Any,
    allow_tool_use: Any = False,
    seed: Any = None
)

Parameters:

  • prompts (Any) — Mapping of prompt name to ChatPrompt.

  • dataset_item (Any) — Dataset row used to render the prompt messages.

  • allow_tool_use (Any, default: False) — Whether tool execution is allowed in this invocation.

  • seed (Any) — Optional seed for reproducibility.

Python
invoke_dataset_item(
    dataset_item: dict
)

Parameters:

  • dataset_item (dict) —
Python
invoke_prompt(
    prompt: Any,
    dataset_item: Any,
    allow_tool_use: Any = False,
    seed: Any = None
)

Parameters:

  • prompt (Any) —

  • dataset_item (Any) —

  • allow_tool_use (Any, default: False) —

  • seed (Any) —

Python
llm_invoke(
    query: str | None = None,
    messages: list[dict[str, str]] | None = None,
    seed: int | None = None,
    allow_tool_use: bool | None = False
)

Parameters:

  • query (str | None) —

  • messages (list[dict[str, str]] | None) —

  • seed (int | None) —

  • allow_tool_use (bool | None, default: False) —

Python
MultiMetricObjective(
    metrics: list,
    weights: list[float] | None = None,
    name: str = 'multi_metric_objective',
    reason: str | None = None,
    reason_builder: collections.abc.Callable[[list[_opik._score_result.ScoreResult], list[float], float], str | None] | None = None
)

Parameters:

  • metrics (list) —

  • weights (list[float] | None) —

  • name (str, default: multi_metric_objective) —

  • reason (str | None) —

  • reason_builder (collections.abc.Callable[[list[_opik._score_result.ScoreResult], list[float], float], str | None] | None) —

Python
ScoringFailedError(
    failed: int,
    total: int,
    objective_metric_name: str | None = None,
    message: str | None = None
)

Parameters:

  • failed (int) —

  • total (int) —

  • objective_metric_name (str | None) —

  • message (str | None) —

Python
PromptLibrary(
    defaults: dict,
    overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None
)

Parameters:

  • defaults (dict) — Dictionary of default prompt templates

  • overrides (dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None) — Optional dict or callable to customize prompts

Python
get(
    key: str,
    fmt: object
)

Parameters:

  • key (str) — The prompt key to retrieve

  • fmt (object) —

Python
get_default(
    key: str
)

Parameters:

  • key (str) — The prompt key to retrieve
Python
keys()
Python
set(
    key: str,
    value: str
)

Parameters:

  • key (str) — The prompt key to set

  • value (str) — The new prompt template

Python
update(
    overrides: dict
)

Parameters:

  • overrides (dict) — Dictionary of key-value pairs to update
Suggest an edit

Propose a replacement for this page. The site team reviews it before applying any changes.

Export
Documentation menu