Skip to main content
Opik Documentation

Search documentation

Type to search this documentation.

On this pageOverview

evaluatePrompt Function

The evaluatePrompt function provides a streamlined way to evaluate prompt templates against a dataset. It automatically formats message templates with dataset variables, generates LLM responses, and evaluates the results using specified metrics.

Overview

Section titled “Overview”

evaluatePrompt is a convenience wrapper around the evaluate function that handles:

  • Template formatting: Automatically formats message templates with dataset item variables
  • Model invocation: Generates LLM responses using your specified model
  • Experiment tracking: Creates experiments linked to specific prompt versions
  • Metric evaluation: Scores outputs using the specified metrics

This is particularly useful for prompt engineering workflows where you want to quickly test different prompt templates against a dataset.

Function Signature

Section titled “Function Signature”
TypeScript
function evaluatePrompt(
  options: EvaluatePromptOptions
): Promise<EvaluationResult>;

EvaluatePromptOptions

Section titled “EvaluatePromptOptions”
TypeScript
interface EvaluatePromptOptions extends Omit<EvaluateOptions, "task"> {
  // Required parameters
  dataset: Dataset;
  messages: OpikMessage[];

  // Optional parameters
  model?: SupportedModelId | LanguageModel | OpikBaseModel;
  templateType?: "mustache" | "jinja2";
  scoringMetrics?: BaseMetric[];
  experimentName?: string;
  experimentConfig?: Record<string, unknown>;
  prompts?: Prompt[];
  projectName?: string;
  nbSamples?: number;
  scoringKeyMapping?: Record<string, string>;
}

Parameters

Section titled “Parameters”

Required Parameters

Section titled “Required Parameters”

dataset

Section titled “dataset”
  • Type: Dataset
  • Description: The dataset to evaluate prompts against. Each dataset item will be used to format the message templates and generate responses.
TypeScript
const dataset = await client.getOrCreateDataset("my-dataset", "Evaluation dataset", "my-project");

messages

Section titled “messages”
  • Type: OpikMessage[]
  • Description: Array of message templates with {{placeholders}} that will be formatted with dataset variables.
TypeScript
messages: [
  { role: "system", content: "You are a helpful assistant" },
  { role: "user", content: "Translate to {{language}}: {{text}}" },
];

Optional Parameters

Section titled “Optional Parameters”

model

Section titled “model”
  • Type: SupportedModelId | LanguageModel | OpikBaseModel
  • Default: "gpt-5-nano"
  • Description: The language model to use for generation. Can be:
    • Model ID string (e.g., "gpt-5-nano", "claude-3-5-sonnet-latest", "gemini-2.0-flash")
    • Pre-configured LanguageModel instance from Vercel AI SDK
    • Custom OpikBaseModel implementation
TypeScript
// Using model ID string
model: "gpt-5-nano";

// Using LanguageModel instance
import { openai } from "@ai-sdk/openai";
const customModel = openai("gpt-5-nano");
model: customModel;

templateType

Section titled “templateType”
  • Type: "mustache" | "jinja2"
  • Default: "mustache"
  • Description: Template engine to use for variable substitution in message content.
TypeScript
// Mustache syntax (default)
templateType: "mustache";
messages: [{ role: "user", content: "Hello {{name}}" }];

// Jinja2 syntax
templateType: "jinja2";
messages: [{ role: "user", content: "Hello {{ name }}" }];

scoringMetrics

Section titled “scoringMetrics”
  • Type: BaseMetric[]
  • Description: Array of metrics to evaluate the generated outputs. Can include both heuristic and LLM Judge metrics.
TypeScript
import { ExactMatch, Hallucination } from "opik";

scoringMetrics: [new ExactMatch(), new Hallucination()];

experimentName

Section titled “experimentName”
  • Type: string
  • Description: Name for the experiment. If not provided, a name will be auto-generated.
TypeScript
experimentName: "Prompt Evaluation - Translation Task";

experimentConfig

Section titled “experimentConfig”
  • Type: Record<string, unknown>
  • Description: Additional metadata to store with the experiment. The function automatically adds prompt_template and model to this configuration.
TypeScript
experimentConfig: {
  temperature: 0.7,
  max_tokens: 1000,
  version: "v2",
};

prompts

Section titled “prompts”
  • Type: Prompt[]
  • Description: Array of Opik Prompt objects to link to this experiment. Useful for tracking which prompt versions were used.
TypeScript
const prompt = await client.createPrompt({
  name: "translation-prompt",
  prompt: "Translate to {{language}}: {{text}}",
});

prompts: [prompt];

projectName

Section titled “projectName”
  • Type: string
  • Description: Name of the Opik project to log traces to.
TypeScript
projectName: "prompt-engineering";

nbSamples

Section titled “nbSamples”
  • Type: number
  • Description: Maximum number of dataset items to evaluate. Useful for quick testing.
TypeScript
nbSamples: 10; // Only evaluate first 10 items

scoringKeyMapping

Section titled “scoringKeyMapping”
  • Type: Record<string, string>
  • Description: Maps metric parameter names to dataset/output field names when they don't match.
TypeScript
scoringKeyMapping: {
  input: "question", // Map 'input' param to 'question' field
  expected: "reference_answer", // Map 'expected' param to 'reference_answer' field
};

Return Value

Section titled “Return Value”

Returns a Promise<EvaluationResult> containing:

TypeScript
interface EvaluationResult {
  experimentId: string; // ID of created experiment
  experimentName: string; // Name of experiment
  testResults: TestResult[]; // Results for each dataset item
}

Examples

Section titled “Examples”

Basic Usage

Section titled “Basic Usage”

Simple prompt evaluation with default settings:

TypeScript
import { Opik, evaluatePrompt } from "opik";

const client = new Opik();
const dataset = await client.getOrCreateDataset("qa-dataset", "Evaluation dataset", "my-project");

await dataset.insert([
  {
    question: "What is the capital of France?",
    expected_answer: "Paris",
  },
  {
    question: "How do you calculate the area of a circle?",
    expected_answer: "π × radius²",
  },
]);

const result = await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "system",
      content:
        "You are a helpful assistant. Answer questions accurately and concisely.",
    },
    { role: "user", content: "{{question}}" },
  ],
  model: "gpt-5-nano",
  projectName: "my-project",
});

console.log(`Experiment ID: ${result.experimentId}`);
console.log(`Evaluated ${result.testResults.length} items`);

With Scoring Metrics

Section titled “With Scoring Metrics”

Evaluate prompts with automatic scoring:

TypeScript
import { evaluatePrompt } from "opik";
import { Hallucination, ExactMatch } from "opik";

// Create dataset with expected answers
const dataset = await client.getOrCreateDataset("geography-qa", "Geography evaluation dataset", "my-project");
await dataset.insert([
  {
    country: "France",
    expected_answer: "Paris",
  },
  {
    country: "Japan",
    expected_answer: "Tokyo",
  },
]);

await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "user",
      content: "What is the capital of {{country}}?",
    },
  ],
  model: "gpt-5-nano",
  scoringMetrics: [
    new ExactMatch(), // Check exact match with expected output
    new Hallucination(), // Check for hallucinations
  ],
  experimentName: "Geography Quiz Evaluation",
  projectName: "my-project",
});

Using LanguageModel Instances

Section titled “Using LanguageModel Instances”

Use LanguageModel instances for provider-specific features:

TypeScript
import { openai } from "@ai-sdk/openai";
import { evaluatePrompt } from "opik";

// Create model instance
const customModel = openai("gpt-5-nano");

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "Summarize: {{text}}" }],
  model: customModel,
  experimentConfig: {
    model_provider: "openai",
    model_name: "gpt-5-nano",
  },
  projectName: "my-project",
});

Multi-Provider Model Support

Section titled “Multi-Provider Model Support”

The function supports models from multiple providers:

TypeScript
// OpenAI
model: "gpt-5-nano";

// Anthropic
model: "claude-3-5-sonnet-latest";

// Google Gemini
model: "gemini-2.0-flash";

// Or use provider-specific LanguageModel instances
import { anthropic } from "@ai-sdk/anthropic";
const claude = anthropic("claude-3-5-sonnet-latest");
model: claude;

Linking to Prompt Versions

Section titled “Linking to Prompt Versions”

Track which prompt versions are used in evaluations:

TypeScript
import { Opik, evaluatePrompt } from "opik";

const client = new Opik();

// Create or get a prompt
const prompt = await client.createPrompt({
  name: "customer-support-prompt",
  prompt: "{{system_message}}\n\nUser: {{user_query}}",
});

// Link the prompt to the evaluation
await evaluatePrompt({
  dataset,
  messages: [
    { role: "system", content: "{{system_message}}" },
    { role: "user", content: "{{user_query}}" },
  ],
  model: "gpt-5-nano",
  prompts: [prompt], // Link to prompt
  experimentName: "Customer Support - v2.1",
  projectName: "my-project",
});

Template Types

Section titled “Template Types”

Mustache Templates (Default)

Section titled “Mustache Templates (Default)”
TypeScript
await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "user",
      content: "Hello {{name}}, your order #{{order_id}} is ready.",
    },
  ],
  templateType: "mustache", // This is the default
  projectName: "my-project",
});

Jinja2 Templates

Section titled “Jinja2 Templates”
TypeScript
await evaluatePrompt({
  dataset,
  messages: [
    {
      role: "user",
      content: "Hello {{ name }}, your order #{{ order_id }} is ready.",
    },
  ],
  templateType: "jinja2",
  projectName: "my-project",
});

Scoring Key Mapping

Section titled “Scoring Key Mapping”

Map dataset fields to metric parameter names:

TypeScript
// Dataset has: { question: "...", reference_answer: "..." }
// Metric expects: { input: "...", expected: "..." }

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{question}}" }],
  scoringMetrics: [new ExactMatch()],
  scoringKeyMapping: {
    input: "question",
    expected: "reference_answer",
  },
  projectName: "my-project",
});

Subset Evaluation

Section titled “Subset Evaluation”

Evaluate only a subset of the dataset for quick iteration:

TypeScript
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{prompt}}" }],
  nbSamples: 5, // Only evaluate first 5 items
  experimentName: "Quick Test",
  projectName: "my-project",
});

How It Works

Section titled “How It Works”

When you call evaluatePrompt, the following happens:

  1. Template Formatting: For each dataset item, message templates are formatted with item variables
  2. Model Invocation: The formatted messages are sent to the specified model to generate a response
  3. Experiment Creation: An experiment is created (or updated) with metadata
  4. Metric Scoring: If metrics are provided, each output is scored
  5. Result Aggregation: Results are collected and returned

Experiment Configuration

Section titled “Experiment Configuration”

The function automatically enriches experiment configuration with:

  • prompt_template: The message templates used
  • model: The model identifier (name or type)

You can add additional metadata via experimentConfig:

TypeScript
experimentConfig: {
  // Auto-added by evaluatePrompt:
  // prompt_template: [{ role: 'user', content: '...' }]
  // model: 'gpt-5-nano'

  // Your custom metadata:
  temperature: 0.7,
  version: "v2.0",
  author: "team-ai",
  description: "Testing improved prompt structure",
};

Best Practices

Section titled “Best Practices”

1. Start Simple, Then Add Metrics

Section titled “1. Start Simple, Then Add Metrics”

Begin with basic prompt evaluation, then add metrics as needed:

TypeScript
// Step 1: Basic evaluation to see outputs
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  projectName: "my-project",
});

// Step 2: Add metrics after reviewing outputs
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  scoringMetrics: [new Hallucination()],
  projectName: "my-project",
});

2. Use Descriptive Experiment Names

Section titled “2. Use Descriptive Experiment Names”

Make it easy to find and compare experiments:

TypeScript
experimentName: "Translation - GPT-4o - v2.3 - 2025-01-15";

3. Version Your Prompts

Section titled “3. Version Your Prompts”

Link evaluations to prompt versions for better tracking:

TypeScript
const prompt = await client.createPrompt({
  name: "qa-prompt",
  prompt: "Answer: {{question}}",
  version: "v2.3",
});

await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "Answer: {{question}}" }],
  prompts: [prompt],
  projectName: "my-project",
});

4. Start with Small Samples

Section titled “4. Start with Small Samples”

Use nbSamples for quick iteration before full evaluation:

TypeScript
// Quick test with 10 samples
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  nbSamples: 10,
  projectName: "my-project",
});

// Full evaluation once satisfied
await evaluatePrompt({
  dataset,
  messages: [{ role: "user", content: "{{input}}" }],
  projectName: "my-project",
  // Evaluate entire dataset
});

5. Include Context in System Messages

Section titled “5. Include Context in System Messages”

Structure your prompts with clear system messages:

TypeScript
messages: [
  {
    role: "system",
    content:
      "You are an expert {{domain}} assistant. Provide accurate, concise answers.",
  },
  {
    role: "user",
    content: "{{question}}",
  },
];

Error Handling

Section titled “Error Handling”

The function validates inputs and throws errors for common issues:

TypeScript
try {
  await evaluatePrompt({
    dataset,
    messages: [],
  });
} catch (error) {
  console.error(error.message);
  // Error: Messages array is required and cannot be empty
}

Common validation errors:

  • Missing required dataset parameter
  • Empty messages array
  • Invalid experimentConfig (must be plain object)
  • Invalid templateType (must be 'mustache' or 'jinja2')

See Also

Section titled “See Also”
Suggest an edit

Propose a replacement for this page. The site team reviews it before applying any changes.

Export
Documentation menu