Test Suites

Test suites are pre-configured regression tests for your LLM application. They let you validate that prompt changes, model updates, or code modifications don't break existing functionality — codifying real production failures as reusable test cases.

A test suite pairs:

• Items — a dataset of inputs (and any context they need), each with optional item-level assertions and execution policy overrides
• Global assertions — natural-language checks (evaluated by an LLM judge) applied to every item
• An execution policy — how many times to run each item, and how many of those runs must pass

For a walkthrough of building suites via Ollie, the UI, or the SDK, see Building Test Suites. This page is the SDK API reference.

The TypeScript SDK provides several methods to create and manage test suites through the OpikClient class.

import { Opik } from "opik";

const opik = new Opik();

// Create a new test suite with suite-level assertions
const suite = await opik.createTestSuite({
  name: "customer-support-qa",
  description: "Regression tests for the support agent",
  projectName: "my-project",
  globalAssertions: [
    "The response is grounded in the provided documentation context",
    "The response directly addresses the user's question",
  ],
  globalExecutionPolicy: { runsPerItem: 2, passThreshold: 2 },
});

// Get an existing test suite, or create it if it doesn't exist
const suite2 = await opik.getOrCreateTestSuite({
  name: "customer-support-qa",
  projectName: "my-project",
});

// Get an existing test suite by name
const existing = await opik.getTestSuite("customer-support-qa", "my-project");

// List test suites
const suites = await opik.getTestSuites(100, "my-project");

// Delete a test suite
await opik.deleteTestSuite("customer-support-qa", "my-project");

// Insert items — item-level assertions/execution policy are optional
// and add to (not replace) the suite-level ones
await suite.insert([
  {
    data: { question: "How do I reset my password?" },
  },
  {
    data: { question: "Can I use this with Kubernetes?" },
    assertions: ["The response does NOT claim Kubernetes is supported"],
    executionPolicy: { runsPerItem: 3, passThreshold: 2 },
  },
]);

// Update existing items (each item must include an `id`)
await suite.update([
  { id: "item-1", assertions: ["The response is under 3 sentences"] },
]);

// Retrieve items, with item-level and suite-level policy merged
const items = await suite.getItems();

// Retrieve items with the raw, unmerged stored payload — evaluators as
// EvaluatorItemWrite[] rather than decoded assertion strings, and the
// item-level executionPolicy only (not merged with the suite default)
const rawItems = await suite.getRawItems();

// Delete specific items
await suite.delete(["item-1", "item-2"]);

// Delete all items in the suite
await suite.clear();

await suite.updateItemAssertions("item-1", [
  "The response cites a specific step from the provided context",
]);

await suite.updateItemExecutionPolicy("item-1", {
  runsPerItem: 5,
  passThreshold: 4,
});

// Or update both at once
await suite.updateItem("item-1", {
  assertions: ["The response is polite"],
  executionPolicy: { runsPerItem: 3, passThreshold: 2 },
});

Execution policies control how many times each item is run and how many of those runs must pass — useful for handling non-deterministic LLM outputs.

• A run passes if all its assertions pass
• An item passes if runsPassed >= passThreshold
• The suite's pass rate is the ratio of passed items to total items

// Read the suite-level defaults
const assertions = await suite.getGlobalAssertions();
const policy = await suite.getGlobalExecutionPolicy();

// Update suite-level assertions and/or execution policy
await suite.updateTestSettings({
  globalAssertions: [
    "The response is grounded in the provided context",
    "The response is concise",
  ],
  globalExecutionPolicy: { runsPerItem: 5, passThreshold: 3 },
});

runTests is the primary entry point for executing a suite: it runs your task against every item, scores results using the suite's configured assertions, and returns pass/fail results.

import { runTests } from "opik";

const result = await runTests({
  testSuite: suite,
  task: async (item) => ({
    input: item,
    output: await myLLM(item.question),
  }),
});

console.log(result.allItemsPassed, result.passRate);

Each run creates a separate experiment in Opik, so you can compare multiple runs (e.g. two prompt versions) against the same suite in the dashboard. evaluateTestSuite is the lower-level function runTests builds on, useful if you need the raw EvaluationResult rather than the suite-shaped TestSuiteResult.

Like datasets, test suites are versioned. See Datasets for the general versioning model — the same DatasetVersion-style methods are available on TestSuite:

const currentVersion = await suite.getCurrentVersionName();
const versionInfo = await suite.getVersionInfo();
const v2 = await suite.getVersionView("v2");

Creates a new test suite.

Arguments:

• options: CreateTestSuiteOptions - { name, description?, globalAssertions?, globalExecutionPolicy?, tags?, projectName? }

Returns: Promise<TestSuite> - The created test suite

Retrieves an existing test suite by name.

Arguments:

• name: string - The name of the test suite to retrieve
• projectName?: string - Optional project name to scope the lookup. If not provided, uses the client's configured project.

Returns: Promise<TestSuite>

Throws: DatasetNotFoundError if the test suite doesn't exist

Retrieves an existing test suite by name, or creates it if it doesn't exist.

Arguments:

• options: CreateTestSuiteOptions - Same shape as createTestSuite

Returns: Promise<TestSuite> - The existing or newly created test suite

Deletes a test suite by name.

Arguments:

• name: string - The name of the test suite to delete
• projectName?: string - Optional project name to scope the lookup. If not provided, uses the client's configured project.

Returns: Promise<void>

Returns all test suites up to the specified limit.

Arguments:

• maxResults?: number - Maximum number of test suites to return (default: 1000)
• projectName?: string - Optional project name to filter by. If not provided, uses the client's configured project.

Returns: Promise<TestSuite[]>

Retrieves all experiments associated with a test suite.

Arguments:

• name: string - The name of the test suite
• maxResults?: number - Maximum number of experiments to return (default: 100)
• projectName?: string - Optional project name to scope the suite lookup. If not provided, uses the client's configured project.

Returns: Promise<TestSuiteExperiment[]> - Each entry carries the suite-specific assertion aggregates (passRate, passedCount, totalCount, assertionScores) populated by the backend

Throws: DatasetNotFoundError if the test suite doesn't exist

Inserts new items into the test suite.

Arguments:

• items: TestSuiteItem[] - { data, assertions?, description?, executionPolicy? }[]

Returns: Promise<void>

Updates existing items. Each item must include an id.

Arguments:

• items: UpdateTestSuiteItem[] - { id, data?, assertions?, description?, executionPolicy? }[]

Returns: Promise<void>

Throws: Error if any item is missing an id

Retrieves items with item-level assertions decoded and the item-level execution policy merged with the suite-level default.

Arguments:

• nbSamples?: number - Max items to retrieve (default: all)
• lastRetrievedId?: string - Opaque cursor for pagination

Returns: Promise<Array<{ id, data, description?, assertions, executionPolicy }>>

Retrieves items with the stored payload preserved verbatim — evaluators as raw EvaluatorItemWrite[] (not decoded to assertion strings) and executionPolicy as the item-level value only (not merged with the suite default). Use this when you need to inspect or forward the stored config as-is.

Arguments:

• nbSamples?: number - Max items to retrieve (default: all)
• lastRetrievedId?: string - Opaque cursor for pagination

Returns: Promise<RawTestSuiteItem[]>

Deletes items from the test suite.

Arguments:

• itemIds: string[] - List of item IDs to delete

Returns: Promise<void>

Deletes all items from the test suite.

Returns: Promise<void>

Returns: Promise<string[]> - The suite-level assertions

Returns: Promise<Required<ExecutionPolicy>> - The suite-level execution policy ({ runsPerItem, passThreshold })

Updates suite-level assertions and/or execution policy. Fields you omit retain their current value.

Arguments:

• options: UpdateTestSuiteOptions - { globalAssertions?, globalExecutionPolicy? } — at least one is required

Returns: Promise<void>

Updates a single item's assertions and/or execution policy.

Arguments:

• itemId: string
• options: { assertions?: string[]; executionPolicy?: ExecutionPolicy } - At least one is required

Returns: Promise<void>

Shorthand for updateItem(itemId, { assertions }).

Arguments:

• itemId: string
• assertions: string[]

Returns: Promise<void>

Shorthand for updateItem(itemId, { executionPolicy }).

Arguments:

• itemId: string
• executionPolicy: ExecutionPolicy

Returns: Promise<void>

Returns: Promise<string[]>

Returns: Promise<number | undefined>

Returns: Promise<string | undefined> - The latest version name (e.g. "v1"), or undefined if no versions exist

Returns: Promise<DatasetVersionPublic | undefined>

Arguments:

• versionName: string - e.g. "v1", "v2"

Returns: Promise<DatasetVersion>

Throws: DatasetVersionNotFoundError if the version doesn't exist

The primary entry point for running a test suite.

async function runTests(options: RunTestsOptions): Promise<TestSuiteResult>;

Returns: Promise<TestSuiteResult>

The lower-level function runTests is built on. Runs a test suite using the evaluators and execution policy stored in the suite's dataset version metadata, and returns the raw EvaluationResult rather than a suite-shaped TestSuiteResult.

async function evaluateTestSuite(
  options: EvaluateTestSuiteOptions
): Promise<EvaluationResult>;

Arguments: same shape as evaluate, plus dataset (the suite's dataset).

Returns: Promise<EvaluationResult>

Result of running a test suite. Returned by runTests.

class TestSuiteResult {
  readonly allItemsPassed: boolean;
  readonly itemsPassed: number;
  readonly itemsTotal: number;
  readonly passRate: number | undefined;
  readonly itemResults: Map<string, ItemResult>;
  readonly experimentId: string;
  readonly experimentName?: string;
  readonly experimentUrl?: string;
  readonly suiteName?: string;
  readonly totalTime?: number;

  // Converts the result to a plain report object (camelCase keys), grouping
  // test results by trial and including per-item pass/fail and assertion detail
  toReportDict(): Record<string, unknown>;

  // Alias for toReportDict()
  toDict(): Record<string, unknown>;
}

Result for a single test suite item, keyed by datasetItemId in TestSuiteResult.itemResults.

type ItemResult = {
  datasetItemId: string;
  passed: boolean;
  // Whether this item had at least one assertion evaluated across any of its runs
  hasAssertions: boolean;
  runsPassed: number;
  runsTotal: number;
  // Configured runsPerItem from the execution policy
  configuredRunsPerItem: number;
  passThreshold: number;
  testResults: EvaluationTestResult[];
};

Represents an experiment run against a test suite. Extends Experiment (see Experiments) with the aggregate assertion statistics the backend populates only for test suite experiments (undefined for regular dataset experiments).

class TestSuiteExperiment extends Experiment {
  readonly passRate?: number;
  readonly passedCount?: number;
  readonly totalCount?: number;
  readonly assertionScores?: AssertionScoreAveragePublic[];
}

interface CreateTestSuiteOptions {
  name: string;
  description?: string;
  globalAssertions?: string[];
  globalExecutionPolicy?: ExecutionPolicy;
  tags?: string[];
  projectName?: string;
}

interface ExecutionPolicy {
  runsPerItem?: number;
  passThreshold?: number;
}