Troubleshooting

This guide covers common troubleshooting scenarios for self-hosted Opik deployments.

Common Issues

ClickHouse Migration Failures: Missing Cluster Macro

Problem Description

Opik requires ClickHouse to be configured with cluster macros, even for single-node deployments. Opik migrations use the ON CLUSTER '{cluster}' clause to ensure DDL operations execute consistently across all nodes in a cluster.

If the {cluster} macro is not configured in your ClickHouse instance, migrations will fail with the following error:

Code: 139. DB::Exception: No macro 'cluster' in config.

Symptoms:

Backend fails to start or enters CrashLoopBackOff state
Migration errors appear in backend logs
Error message: Code: 139. DB::Exception: No macro 'cluster' in config.

Automatic Configuration

Opik Helm Chart and Docker Compose deployments automatically configure the required cluster macros. If you're using Opik's provided deployment configurations, you should not encounter this issue.

Manual Configuration Required

If you're running your own ClickHouse instance (not using Opik's Helm chart or Docker Compose), you need to configure the cluster macros yourself.

1. Add Cluster Macro Configuration

Add the {cluster} macro to your ClickHouse configuration file. The location depends on your ClickHouse installation:

For standard ClickHouse installations:

Add the macros to /etc/clickhouse-server/config.d/macros.xml (or your equivalent config directory):

xml

<clickhouse>
    <macros>
        <cluster>single_node_cluster</cluster>
        <shard>1</shard>
        <replica>clickhouse</replica>
    </macros>
</clickhouse>

2. Restart ClickHouse

After adding the configuration, restart ClickHouse for the changes to take effect:

3. Verify Configuration

You can verify the macro is configured by connecting to ClickHouse and running:

SQL

SELECT * FROM system.macros WHERE macro = 'cluster';

You should see a row with the macro name and value.

4. Retry Migrations

After restarting ClickHouse, retry the backend deployment or migration. The backend should automatically retry after ClickHouse is ready.

Backend Not Ready: `clickhouse-traces-topology` / `clickhouse-spans-topology`

Problem Description

The backend refuses readiness and the clickhouse-traces-topology or clickhouse-spans-topology health check reports unhealthy with a message about databaseAnalyticsDataModel.tracesDistributedWrapEnabled or databaseAnalyticsDataModel.spansDistributedWrapEnabled.

This check failing is intentional, not a bug. It means the setting disagrees with the actual shape of the table it describes — traces or spans — in ClickHouse. Deletion of those rows cannot work in that state, so the backend takes itself out of rotation at startup instead of accepting traffic and failing on the first delete.

The two probes are independent and each reads only its own tables, because the two cutovers are applied separately: an install can legitimately be wrapped on traces and not on spans. Whichever probe is red names the setting to change.

Symptoms:

Readiness probe (/health-check?name=all&type=ready) returns 503; pods never become Ready
GET /health-check?name=clickhouse-traces-topology (or ...=clickhouse-spans-topology) reports "healthy": false
Backend logs: A critical dependency is now unhealthy: name=clickhouse-traces-topology, type=READY

Cause

The setting tells the backend where to send deletions for its table. It has to match the table:

Setting	Required engine
`tracesDistributedWrapEnabled: false` (the default)	`traces` is any MergeTree-family engine
`tracesDistributedWrapEnabled: true`	`traces` is `Distributed`, with a `traces_local` table present
`spansDistributedWrapEnabled: false` (the default)	`spans` is any MergeTree-family engine
`spansDistributedWrapEnabled: true`	`spans` is `Distributed`, with a `spans_local` table present

"MergeTree-family" means any engine whose name ends in MergeTree — the check matches on that suffix, because that is what marks an engine as taking deletions directly. On a standard install both tables are ReplicatedReplacingMergeTree, which qualifies; so do MergeTree, ReplicatedMergeTree and ClickHouse Cloud's SharedMergeTree.

A default self-hosted or open-source install has both settings false and the ReplicatedReplacingMergeTree tables the migrations create, so both checks pass and there is nothing to do. One only fails if its setting was turned on, or if its table was converted to a Distributed wrapper without turning it on.

Resolution

Start from the health check message — it names both the setting and the engine it actually found. The readiness endpoint does not carry it (/health-check returns the verdict only), so read it from the admin connector, which listens on SERVER_ADMIN_PORT (8081 by default):

Bash

# Kubernetes — the Helm chart's Service does not expose the admin port, so forward it.
# Substitute your namespace; every command below assumes the same one. The workload is named after
# the chart rather than the release, so it stays `opik-backend` unless you set `nameOverride`.
kubectl -n <namespace> port-forward deploy/opik-backend 8081:8081
curl -s localhost:8081/healthcheck | jq '.["clickhouse-spans-topology"], .["clickhouse-traces-topology"]'

# Docker Compose — published on the host only when you start with the override file
# (docker compose -f docker-compose.yaml -f docker-compose.override.yaml ...)
curl -s localhost:8081/healthcheck

# Docker Compose — otherwise, from inside the container
docker exec <backend-container> curl -s localhost:8081/healthcheck

Then make the setting and the table agree.

Check what the tables really are. Substitute <database_name> with your ANALYTICS_DB_DATABASE_NAME — opik is only the default, and querying the wrong database returns no rows, which looks like the tables are missing:

SQL

SELECT name, engine FROM system.tables
WHERE database = '<database_name>'
  AND name IN ('traces', 'traces_local', 'spans', 'spans_local');

SQL

-- Same lookup, every replica. Compare down each table separately: one table's engine must be
-- the same on every host. A table and its `_local` shard are expected to differ from each
-- other: after the wrap `traces` is `Distributed` and `traces_local` is a
-- `(Replicated)MergeTree`, which is the healthy shape, not a mismatch. So is `traces` being
-- wrapped while `spans` is not — the two cutovers are independent.
SELECT name, hostName() AS host, engine
FROM clusterAllReplicas('{cluster}', system.tables)
WHERE database = '<database_name>'
  AND name IN ('traces', 'traces_local', 'spans', 'spans_local')
ORDER BY name, host;

Substitute {cluster} here as well — your cluster macro, from SELECT * FROM system.macros; on a single-node install drop the clusterAllReplicas(...) wrapper instead. If a single table's engine differs between hosts, an ON CLUSTER DDL has not finished propagating or failed on a host — let it settle, or re-run it there. Changing the flag would only move the failure to the other replicas.

Then set the flag to match, for whichever table is out of step:

traces is a MergeTree-family engine → set databaseAnalyticsDataModel.tracesDistributedWrapEnabled: false (Helm) or ANALYTICS_DB_DATA_MODEL_TRACES_DISTRIBUTED_WRAP_ENABLED=false (Docker Compose or plain environment), then restart the backend.
traces is Distributed and traces_local exists → set the same key to true and restart.
spans is a MergeTree-family engine → set databaseAnalyticsDataModel.spansDistributedWrapEnabled: false (Helm) or ANALYTICS_DB_DATA_MODEL_SPANS_DISTRIBUTED_WRAP_ENABLED=false, then restart the backend.
spans is Distributed and spans_local exists → set the same key to true and restart.

On Helm, an explicit env entry wins over the chart value. The chart derives ANALYTICS_DB_DATA_MODEL_TRACES_DISTRIBUTED_WRAP_ENABLED / ANALYTICS_DB_DATA_MODEL_SPANS_DISTRIBUTED_WRAP_ENABLED from databaseAnalyticsDataModel.tracesDistributedWrapEnabled / databaseAnalyticsDataModel.spansDistributedWrapEnabled only when that key is not already set under component.backend.env — an explicit entry there takes precedence. So if you change the top-level value, restart, and the check still reports the old expectation, you have a stale override: update or remove the entry under component.backend.env. Confirm what the backend actually receives — both keys at once — with kubectl -n <namespace> get configmap opik-backend -o jsonpath='{.data.ANALYTICS_DB_DATA_MODEL_TRACES_DISTRIBUTED_WRAP_ENABLED}{"\n"}{.data.ANALYTICS_DB_DATA_MODEL_SPANS_DISTRIBUTED_WRAP_ENABLED}{"\n"}' (the ConfigMap is named after the chart too).

The check re-evaluates on every probe, so readiness returns on its own once the two sides agree — no further action beyond the restart that picks up the new setting.

If the table does not exist at all, the check says so instead: the analytics migrations have not run. See the migration sections above.

If you are applying the wrap: what the transition looks like

Applying the Distributed wrap means changing two things that cannot land at the same instant — the setting (a config push plus a rolling restart) and the table (one DDL statement). Expect a window where they disagree, and expect this check to report it. The behaviour is the same for traces and spans:

The probe is a critical readiness dependency, so a pod whose setting disagrees with the table it finds never becomes Ready. During a rolling update that means the new pod stays unready and Kubernetes keeps the old one serving. The deployment looks stuck; traffic does not stop.
The probe re-reads the topology on every probe but holds the setting from startup, so the two orderings recover differently:
- Setting first (recommended): the pod has already restarted, so it clears itself within one probe interval of the DDL landing.
- DDL first: existing pods keep their old setting and stay unready until the config rollout restarts them. There the restart is the fix, not a workaround.
Either way the window announces itself, and it is self-clearing. Keep it short, and run it inside a declared maintenance window.

Because the probe reads one replica per probe, a partly-propagated ON CLUSTER DDL shows up as pods flapping rather than the whole fleet going unready. That is expected inside the window — use the cluster-wide query above to confirm the DDL reached every replica.

Fresh Multi-Replica Install Migration Failures

Problem Description

A brand-new Opik install on a ClickHouse cluster with 2 or more replicas cannot complete its analytics migrations. The backend never starts (CrashLoopBackOff), and the migration step fails with errors such as:

Code: 60. DB::Exception: Could not find table: <table_name>. (UNKNOWN_TABLE)
Code: 81. DB::Exception: Database opik does not exist. (UNKNOWN_DATABASE)

opik is the default analytics database name (ANALYTICS_DB_DATABASE_NAME); substitute your own if you overrode it.

Symptoms:

Fresh install only (no existing data); backend in CrashLoopBackOff
Migration errors reference a table or the opik database that is missing on one replica
One replica holds the opik database and tables while another has few or none

Cause

Opik's earliest analytics migrations predate cluster-aware DDL — they do not use ON CLUSTER '{cluster}' — so they create the opik database and base tables on a single replica only. Later, cluster-aware migrations fan out to every replica and fail on the one that never received that base schema. The ClickHouse operator only copies schema to a replica when it joins the cluster (a scale-up), not during a migration.

Resolution

Install High-Availability deployments in two phases so the schema exists before the extra replicas join:

Install (or reset to) clickhouse.replicasCount: 1.
Wait until opik-backend is Ready — all migrations are applied on the single replica.
Raise clickhouse.replicasCount (e.g. to 2) and upgrade. The operator provisions the new replica and copies the fully-migrated schema to it.

Lost Liquibase Changelog with an Intact Schema

Problem Description

The opik tables are all present and the application data is intact, but the Liquibase changelog table (default.DATABASECHANGELOG) is empty or missing rows. A newly started opik-backend replica then treats every migration as pending and replays it against tables that already exist, failing with errors such as:

Code: 15. DB::Exception: Cannot add column `created_by`: column with this name already exists. (DUPLICATE_COLUMN)

Symptoms:

Existing deployment with real data; the schema looks complete
New replicas or rolling updates fail with Init:CrashLoopBackOff while already-running replicas keep serving
Migration errors say an object already exists, rather than that one is missing
SELECT count() FROM default.DATABASECHANGELOG returns 0, or far fewer rows than there are migration files

Cause

This usually follows a recovery that restored the data but not the bookkeeping — for example a volume delete/recreate where the opik.* tables were restored from backup while the default database holding the changelog was not.

Liquibase decides what to run purely from the changelog table. With no rows, it concludes nothing has ever been applied and starts from the first changeset. Replaying migrations against a schema that already has them is not safe: early changesets add columns that are already present, and later ones convert tables to ReplicatedMergeTree and move partitions between them. Migrations cannot be edited to make a replay safe — once released, a changeset is immutable, because changing it alters its checksum and every already-migrated deployment would then refuse to start.

The fix is therefore to repair the ledger, not to re-run the migrations.

Resolution

Re-baseline the changelog: record the pending changesets as applied without executing them. Opik ships a script for this in the backend image.

Only do this when the schema genuinely matches the changelog. Re-baselining asserts "everything up to here is already applied" — if a migration truly has not run, it is marked as done and never will, leaving the schema permanently behind a ledger that claims otherwise.

"Matches the changelog" means more than the tables existing. Before continuing, confirm all of the following:

The image version you are running against is the one the schema was migrated to. Re-baselining with a newer jar marks that version's extra migrations as applied without running them.
Every changeset in the pending list corresponds to an object that already exists — spot-check the most recent few, not just the first.
For ClickHouse (dbAnalytics, the default) on a multi-replica cluster, check every replica, not just the one you are connected to. A partially restored replica can look complete at the table level while its replicated tables are stale. system.replicas only reflects the node you are connected to, so query it cluster-wide:
SQL
```
-- Any replica that cannot accept writes is not settled.
SELECT hostName() AS host, database, table, is_readonly, absolute_delay
FROM clusterAllReplicas('{cluster}', system.replicas)
WHERE database = '<database_name>' AND is_readonly;

-- A stuck queue entry means replication is not merely behind, it is failing.
SELECT hostName() AS host, database, table, type, num_tries, last_exception
FROM clusterAllReplicas('{cluster}', system.replication_queue)
WHERE database = '<database_name>' AND num_tries > 10;
```
Substitute {cluster} (your cluster macro) and <database_name> (your ANALYTICS_DB_DATABASE_NAME, opik by default); on a single-node install drop the clusterAllReplicas(...) wrapper and query system.replicas directly. Both queries should return no rows. A nonzero absolute_delay on its own is normal replication lag and is not a reason to stop — what matters is that no replica is read-only and no queue entry is failing repeatedly.
For MySQL (--database db) there is no replication view to check — confirm instead that the tables and columns named by the pending changesets already exist, and that the deployment has not been upgraded past the schema you are recovering.

If any of these is uncertain, stop and restore the changelog from a backup of the same version instead. --yes skips the prompt, not these checks — only use it in a runbook that has already performed them.

For ClickHouse the script enforces the coarse version of this itself: it counts the tables in the analytics database and refuses when the schema is plainly not built (an empty or nearly empty database against a full pending list), including under --yes. That catches the worst case — re-baselining a database that has no schema at all — but it cannot judge the finer conditions above, such as a stale replica or a newer jar. Treat it as a backstop, not a substitute for the checklist.

Get a shell in an opik-backend container, in its working directory (/opt/opik).
If a replica is still running (the usual case — existing pods keep serving while only new ones fail), exec into it. If every replica is in CrashLoopBackOff, there is nothing to exec into; start a one-shot container from the same image instead, so the jar matches the deployed schema version.
On Kubernetes:
Bash
```
NS=<your namespace>

# Take the image from the failing deployment rather than using a floating tag —
# the recovery must run against the version the schema was migrated to.
IMAGE=$(kubectl get deploy opik-backend -n "$NS" \
  -o jsonpath='{.spec.template.spec.containers[0].image}')

# --image is required by kubectl even though the override also sets it, so both
# read from $IMAGE to keep them from drifting apart.
kubectl run opik-rebaseline --rm -it --restart=Never -n "$NS" \
  --image="$IMAGE" \
  --overrides="$(cat <<JSON
{"spec":{"containers":[{"name":"opik-rebaseline","image":"$IMAGE","command":["bash"],"stdin":true,"tty":true,
  "envFrom":[{"secretRef":{"name":"<your opik-backend secret>"}},{"configMapRef":{"name":"<your opik-backend configmap>"}}]}]}}
JSON
)"
```
On Docker Compose:
Bash
```
docker compose run --rm --entrypoint bash opik-backend
```
On Compose the service already pins its image, so no extra step is needed there.
envFrom carries environment variables but not files. If your deployment terminates TLS to MySQL or ClickHouse — anything setting MYSQL_TRUSTSTORE_URL / MYSQL_CLIENTSTORE_URL, or relying on the chart's ca-cert-injection init container — the one-shot pod will start without those certificates and fail to connect before it reaches the re-baseline. Copy the relevant volumes, volumeMounts and init container from the live spec (kubectl get deploy opik-backend -n "$NS" -o yaml) into the overrides as well. Exec'ing into a surviving replica avoids this entirely, so prefer it whenever one is available.
Review what would change, without writing anything:
Bash
```
./rebaseline_db_changelog.sh --dry-run
```
This prints the pending changesets. Confirm they correspond to schema objects that already exist in the database.
Re-baseline the analytics changelog:
Bash
```
./rebaseline_db_changelog.sh
```
The script shows the pending list again, asks for confirmation, records the changesets as applied, and re-runs the status check so you can see the ledger is now clean. Only changelog rows are written — no DDL runs and no data is touched.
Restart the failing replicas. They now skip the recorded changesets instead of replaying them.

Pass --yes to skip the confirmation prompt in an automated runbook; the schema check still runs and still aborts.

--database db re-baselines the MySQL (state) changelog instead of the ClickHouse (analytics) one. There is no MySQL client in the backend image, so the schema check cannot run there and the script refuses unless you add --force-unverified, which asserts you have confirmed the schema yourself. The same flag overrides every one of the ClickHouse checks: a probe that could not reach the server, and a probe that reports an implausibly low table count against a schema you have confirmed is at head. Reach for it only after working through the checklist above — it turns a refusal back into the silent-bricking path.

The same applies to --config. The re-baseline connects through the config file you pass, while the schema check reads the ANALYTICS_DB_MIGRATIONS_* environment variables — these describe the same database only for the packaged config.yml, which resolves from exactly those variables. With any other config the check could verify a different database than the one being written, so the script refuses unless you add --force-unverified.

None of these refusals apply to --dry-run, which reports without writing and so needs no verification: it prints the pending changesets and exits, on either database, and never contacts ClickHouse for a table count. That is what makes the inspection step above workable on MySQL, where the schema cannot be verified at all.

If you prefer to run the underlying commands directly, these are the Dropwizard migration commands the script drives:

Bash

java -jar opik-backend-$OPIK_VERSION.jar dbAnalytics status --verbose config.yml
java -jar opik-backend-$OPIK_VERSION.jar dbAnalytics fast-forward --all config.yml

Prevention

Include the default database in ClickHouse backups, not just opik. The changelog table is small but a restore without it leaves the deployment unable to start new replicas.

ClickHouse Zookeeper Metadata Loss

Problem Description

If Zookeeper loses the metadata paths for ClickHouse tables, you will see coordination exceptions in the ClickHouse logs and potentially in the opik-backend service logs. These errors indicate that Zookeeper cannot find table metadata paths.

Symptoms:

Error messages appearing in ClickHouse logs and propagating to opik-backend service:

Code: 999. Coordination::Exception: Coordination error: No node, path /clickhouse/tables/0/default/DATABASECHANGELOG/log. (KEEPER_EXCEPTION)

This indicates that Zookeeper has lost the metadata paths for one or more ClickHouse tables.

Resolution Steps

Follow these steps to restore ClickHouse table metadata in Zookeeper:

1. Clean Zookeeper Paths (If Needed)

If only some table paths are missing in Zookeeper, you'll need to delete the existing paths manually. Connect to the Zookeeper pod and use the Zookeeper CLI:

Bash

# Connect to Zookeeper pod
kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181

# Delete all ClickHouse table paths
deleteall /clickhouse/tables

2. Restart ClickHouse

Restart the ClickHouse pods so they become aware that Zookeeper no longer has the metadata:

Bash

kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0

3. Restore Replica Definitions

Connect to the first ClickHouse replica and restore the replica definitions for each table:

Bash

# Connect to the first ClickHouse replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client

Run the SYSTEM RESTORE REPLICA command for each table:

SQL

-- Restore system tables
SYSTEM RESTORE REPLICA default.DATABASECHANGELOG;
SYSTEM RESTORE REPLICA default.DATABASECHANGELOGLOCK;

-- Verify your Opik database name
SHOW DATABASES;

-- List all Opik tables (replace 'opik' with your actual schema name if different)
USE opik;
SHOW TABLES;

-- Restore each Opik table
SYSTEM RESTORE REPLICA opik.attachments;
SYSTEM RESTORE REPLICA opik.automation_rule_evaluator_logs;
SYSTEM RESTORE REPLICA opik.comments;
SYSTEM RESTORE REPLICA opik.dataset_items;
SYSTEM RESTORE REPLICA opik.experiment_items;
SYSTEM RESTORE REPLICA opik.experiments;
SYSTEM RESTORE REPLICA opik.feedback_scores;
SYSTEM RESTORE REPLICA opik.guardrails;
SYSTEM RESTORE REPLICA opik.optimizations;
SYSTEM RESTORE REPLICA opik.project_configurations;
SYSTEM RESTORE REPLICA opik.spans;
SYSTEM RESTORE REPLICA opik.traces;
SYSTEM RESTORE REPLICA opik.trace_threads;
SYSTEM RESTORE REPLICA opik.workspace_configurations;

4. Restart ClickHouse Again

Restart ClickHouse again to ensure it:

Re-establishes connections to Zookeeper
Verifies and synchronizes the newly restored metadata
Automatically resumes normal replication operations

Bash

kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0

5. Validate the Recovery

After the restart completes, verify that the replica status is healthy:

SQL

-- Check table creation
SHOW CREATE TABLE opik.attachments;

-- Verify replica status
SELECT table, is_readonly, replica_is_active, zookeeper_exception
FROM system.replicas;

Expected Results:

is_readonly = 0 (table is writable)
replica_is_active = 1 (replica is active)
zookeeper_exception = '' (no exceptions)

You can also verify from the Zookeeper side:

Bash

# Connect to Zookeeper CLI
kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181

# List tables (example path - adjust for your database name)
ls /clickhouse/tables/0/<database_name>/<table_name>

ClickHouse `TOO_MANY_PARTS` Errors and Stuck Merges

Problem Description

Under sustained high-volume ingestion, span/trace batch inserts may start failing with HTTP 500s while ClickHouse rejects new parts:

Code: 252. DB::Exception: Too many parts (N) in table 'opik.spans'.
Merges are processing significantly slower than inserts:
While executing WaitForAsyncInsert. (TOO_MANY_PARTS)

Clients calling POST /api/v1/private/spans/batch (or /traces/batch) receive 500s. The active part count for the table has exceeded ClickHouse's parts_to_throw_insert threshold (default 3000) and is not draining.

Symptoms:

500s on the span/trace batch endpoints; TOO_MANY_PARTS (Code: 252) in the opik-backend logs
A high and still-growing active-part count for spans and/or traces
If it persists, the opik-backend ClickHouse connection pool can exhaust (ConnectionRequestTimeoutException), amplifying the impact

Diagnose first

SQL

-- Active parts per node and table: is the count high and growing?
SELECT hostName() AS host, table, count() AS active_parts
FROM clusterAllReplicas('{cluster}', system.parts)
WHERE active AND database = '<database_name>'
GROUP BY host, table ORDER BY active_parts DESC;

-- Is a merge stuck? Look for a queue entry with a rising num_tries and a repeating exception.
SELECT hostName() AS host, database, table, type, num_tries, new_part_name, last_exception
FROM clusterAllReplicas('{cluster}', system.replication_queue)
WHERE database = '<database_name>'
ORDER BY num_tries DESC;

-- How many merges are actually running per node vs. the pool size?
SELECT hostName() AS host, count() AS running_merges
FROM clusterAllReplicas('{cluster}', system.merges)
GROUP BY host;

-- Replica health (rules out the ZooKeeper-metadata-loss scenario below).
SELECT hostName() AS host, table, is_readonly, replica_is_active, zookeeper_exception
FROM clusterAllReplicas('{cluster}', system.replicas);

Cause A — fragmentation (merges can't keep up). Merges complete normally and quickly, running_merges is healthy, but parts are created faster than they merge. system.replication_queue shows no entry stuck with a high num_tries. This is driven by async-insert flush frequency under high concurrency.
Cause B — a stuck merge blocking the queue. running_merges is near zero (despite a large background_pool_size) and one replication_queue entry has a high, climbing num_tries with a repeating last_exception — commonly Code: 76 ... CANNOT_OPEN_FILE on corrupt/missing source-part files. That one poison entry jams the scheduler so nothing else merges and parts cannot drain, regardless of CPU or disk.

Cause A resolution — tune async insert

Opik applies its async-insert settings on the ClickHouse connection (via custom_http_params in ANALYTICS_DB_QUERY_PARAMETERS), so they apply to every insert. The shipped defaults favor freshness over batching:

async_insert=1
wait_for_async_insert=1
async_insert_busy_timeout_min_ms=100
async_insert_busy_timeout_max_ms=250
async_insert_use_adaptive_busy_timeout=1

With async_insert=1, each server-side flush becomes a new part. A short busy-timeout window (100–250 ms) means frequent flushes, and under high concurrency this creates many small parts that merges must keep up with.

To reduce fragmentation for high-volume deployments, widen the flush window (fewer, larger parts) via these opik-backend environment variables:

Bash

ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MAX_MS=2000   # ceiling of the adaptive flush window (ms); e.g. 1000–3000
ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MIN_MS=1000   # floor of the adaptive flush window (ms); keep below max
ANALYTICS_DB_ASYNC_INSERT_MAX_DATA_SIZE=52428800     # buffered bytes that force a flush; larger = fewer parts

Widening the flush window trades a little ingestion latency and buffer memory — rows become queryable up to max_ms later — for far fewer parts, a good trade for high-volume observability data. Under sustained high load, flushes are size-triggered anyway, so most of the added latency falls on quieter periods. After changing these, restart the opik-backend and watch the active part count and system.asynchronous_inserts, dialing max_ms between 1000–3000 ms to taste. New inserts will fragment less; an existing backlog still needs to merge down (it will, once the insert rate no longer outpaces merges).

Cause B resolution — unblock the stuck merge

Work from least to most invasive: the threshold bump (step 1) and diagnosis (steps 2–3) are non-destructive; DETACH (step 4) is recoverable but data-affecting; the ZooKeeper edit (step 6) is the last resort. Run the commands against the table your diagnosis flagged — replace <table> with spans or traces.

Restore ingestion immediately (reversible, no data loss). Temporarily raise the throw threshold so inserts succeed while you fix the root cause:
SQL
```
ALTER TABLE <database_name>.<table> ON CLUSTER '{cluster}' MODIFY SETTING parts_to_throw_insert = 20000, parts_to_delay_insert = 20000;
```
On a Distributed deployment, target the underlying per-shard ReplacingMergeTree table, not the Distributed proxy. This is a safe, reversible stopgap — no data is deleted — so revert to the defaults once the backlog has drained.
Confirm replicas are healthy (is_readonly = 0, replica_is_active = 1, zookeeper_exception = ''). If you see Keeper No node errors, switch to ClickHouse Zookeeper Metadata Loss.
Identify the failing merge and its corrupt source parts from the system.replication_queue.last_exception values (the CANNOT_OPEN_FILE messages name the offending part directories).
Try to recover the corrupt source parts. If a part is still intact on another replica, DETACH the corrupt local copy (never DROP) — ClickHouse then re-fetches the good copy from a healthy replica, which lets the blocked merge complete. Detached parts are preserved under the table's detached/ directory.
SQL
```
ALTER TABLE <database_name>.<table> DETACH PART 'all_1_100_5';
-- repeat for each corrupt part named in the exception
```
Data-affecting, expert operation — do it with ClickHouse expertise on hand. On a ReplicatedMergeTree the queued merge entry references its source parts by name, so DETACH does not by itself remove that entry: it only helps when the part is intact elsewhere and can be re-fetched. If the part is corrupt on all replicas there is nothing to re-fetch — the entry keeps retrying and you must proceed to step 6 to clear it (and removing the part also drops its rows, a small bounded window for observability data). Always DETACH (recoverable), never DROP.
Re-check the queue. If the stuck entry is gone and merges resume, the backlog will drain on its own. A SYSTEM RESTART REPLICA <database_name>.<table> can help a node re-read its queue.
Last resort — remove the stuck queue entry from ZooKeeper. When re-fetch/DETACH can't clear it (steps 4–5), removing the poison entry directly is the reliable fix — but it is dangerous, so do it only after 4–5 and ideally with ClickHouse expertise on hand. Delete the specific stuck queue-XXXXXXXXXX node under every replica path, then restart the replicas:
```
/clickhouse/tables/<shard>/<database_name>/<table>/replicas/<replica>/queue/queue-XXXXXXXXXX
```
Editing ZooKeeper directly is dangerous and can corrupt replication if the wrong node is removed. Delete only the exact stuck entry, on all replicas, and prefer the DETACH PART path above whenever possible.
Verify recovery and revert. Watch the active-part count fall, then restore parts_to_throw_insert / parts_to_delay_insert to their defaults.

Prevention

High-volume ingestion: raise async_insert_busy_timeout_max_ms (Cause A above) before scaling ingestion up, and monitor part counts.
ClickHouse major-version upgrades: a replica can transiently go read-only during a rolling upgrade of replicated tables, which may leave part files inconsistent and schedule a merge that later fails. Quiesce or throttle ingestion during the upgrade, and confirm all replicas are healthy (system.replicas) and system.replication_queue is clean before ramping ingestion back up.
Monitoring: alert on the cluster-wide active part count per table and on system.replication_queue entries with a rising num_tries, so a stuck merge is caught before it becomes TOO_MANY_PARTS.

Diagnostic Commands

Connecting to ClickHouse

Connect directly to ClickHouse pods for diagnostics:

Bash

# Connect to first replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client

# Connect to second replica (if running multiple replicas)
kubectl exec -it chi-opik-clickhouse-cluster-0-1-0 -- clickhouse-client

Connecting to Zookeeper

Connect directly to Zookeeper pods:

Bash

# Connect to Zookeeper pod
kubectl exec -it cometml-production-opik-zookeeper-0 -- bash

# Run Zookeeper client commands
zkCli.sh -server localhost:2181

Common Zookeeper commands:

Bash

# List tables in Zookeeper
kubectl exec -it cometml-production-opik-zookeeper-0 -- \
  zkCli.sh -server localhost:2181 ls /clickhouse/tables/0/opik

# Remove a specific table from Zookeeper
kubectl exec -it cometml-production-opik-zookeeper-0 -- \
  zkCli.sh -server localhost:2181 \
  deleteall /clickhouse/tables/0/opik/optimizations

Prevention and Best Practices

To avoid Zookeeper metadata loss issues:

Regular Backups: Implement regular backups of ClickHouse data. See the Advanced ClickHouse Backup guide for details.
Monitoring: Set up monitoring for Zookeeper health and ClickHouse replica status. Alert on zookeeper_exception in system.replicas.
Resource Allocation: Ensure Zookeeper has adequate resources (CPU, memory, disk) to maintain metadata reliably.
Persistent Storage: Use persistent volumes for Zookeeper to prevent data loss during pod restarts.
Replica Validation: Regularly check replica status with the diagnostic queries above.
Check the Changelog Before Upgrades: Review the self-host changelog for breaking and critical changes before you upgrade your deployment.
Back Up the Liquibase Ledger: Include ClickHouse's default database in backups alongside opik. It holds Liquibase's DATABASECHANGELOG table — the migration ledger, unrelated to the self-host changelog page above; restoring data without it leaves the deployment unable to start new replicas. See Lost Liquibase Changelog with an Intact Schema.

Getting Help

If you continue to experience issues after following this guide:

Check the Opik GitHub Issues for similar problems
Review ClickHouse and Zookeeper logs for additional error details
Open a new issue on GitHub with:
- Opik versions:
  - Backend version (opik-backend)
  - Frontend version (opik-frontend)
  - Helm chart version (if deployed via Helm)
- ClickHouse version
- Zookeeper version
- Error logs from all services (ClickHouse, Zookeeper, opik-backend)
- Steps taken to reproduce the issue

Common Issues

ClickHouse Migration Failures: Missing Cluster Macro

Problem Description

Automatic Configuration

Manual Configuration Required

1. Add Cluster Macro Configuration

2. Restart ClickHouse

3. Verify Configuration

4. Retry Migrations

Backend Not Ready: clickhouse-traces-topology / clickhouse-spans-topology

Problem Description

Cause

Resolution

If you are applying the wrap: what the transition looks like

Fresh Multi-Replica Install Migration Failures

Problem Description

Cause

Resolution

Lost Liquibase Changelog with an Intact Schema

Problem Description

Cause

Resolution

Prevention

ClickHouse Zookeeper Metadata Loss

Problem Description

Resolution Steps

1. Clean Zookeeper Paths (If Needed)

2. Restart ClickHouse

3. Restore Replica Definitions

4. Restart ClickHouse Again

5. Validate the Recovery

ClickHouse TOO_MANY_PARTS Errors and Stuck Merges

Problem Description

Diagnose first

Cause A resolution — tune async insert

Cause B resolution — unblock the stuck merge

Prevention

Diagnostic Commands

Connecting to ClickHouse

Connecting to Zookeeper

Prevention and Best Practices

Getting Help

Backend Not Ready: `clickhouse-traces-topology` / `clickhouse-spans-topology`

ClickHouse `TOO_MANY_PARTS` Errors and Stuck Merges