Large CSV uploads

Opik accepts CSV file uploads of up to 2GB for dataset creation. For self-hosted deployments that need to process large CSV files, apply the additional configuration below.

CSV upload supports:

• CSV files up to 2GB in size
• Asynchronous processing - files are processed in the background after upload

Large file uploads require more time to transfer. Increase the server idle timeout:

SERVER_IDLE_TIMEOUT: 10m

The default timeout is 30 seconds, which is insufficient for large file uploads. We recommend setting it to 10 minutes for files up to 2GB.

If you're using the Helm chart deployment, add the following configuration to your values.yaml:

component:
  frontend:
    # Increase client body size limit to 2GB
    clientMaxBodySize: "2g"
    
    # Increase proxy timeouts for large file uploads
    upstreamConfig:
      proxy_read_timeout: 600s
      proxy_connect_timeout: 600s
      proxy_send_timeout: 600s
      client_max_body_size: 2g

The backend service temporarily buffers uploaded CSV files to disk before processing them. Ensure your backend pods/containers have:

• Minimum 50GB of disk space available
• Sufficient IOPS for concurrent file operations

You can optionally configure the batch size for CSV processing:

BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE: 1000

The default batch size is 1000 rows per batch. Adjust this based on your:

• Available memory
• Row complexity (number of columns, data size)
• Desired processing speed

For Docker Compose deployments, the configuration is slightly different:

Add the environment variables to the backend service:

services:
  backend:
    environment:
      - SERVER_IDLE_TIMEOUT=10m
      - BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE=1000

The nginx configuration files already include the 2GB limit for local deployments. No additional changes are needed for nginx_default_local.conf or nginx_local_be_local.conf.

Here's a complete example for Helm chart deployments:

# values.yaml
component:
  backend:
    env:
      SERVER_IDLE_TIMEOUT: "10m"
      BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE: "1000"
    
    # Ensure adequate disk space
    persistence:
      enabled: true
      size: 100Gi  # Adjust based on your needs
  
  frontend:
    clientMaxBodySize: "2g"
    
    upstreamConfig:
      proxy_read_timeout: 600s
      proxy_connect_timeout: 600s
      proxy_send_timeout: 600s
      client_max_body_size: 2g

Then upgrade your Helm release:

helm upgrade opik opik/opik -n opik -f values.yaml

After applying the configuration:

1. Restart services to apply the changes
2. Test with a small CSV first (< 100MB) to verify the feature works
3. Monitor logs during upload to ensure proper processing:

# Kubernetes
kubectl logs -n opik deployment/opik-backend -f | grep CSV

# Docker Compose
docker-compose logs -f backend | grep CSV

You should see log messages like:

CSV upload request for dataset 'xxx' on workspaceId 'xxx'
CSV upload accepted for dataset 'xxx' on workspaceId 'xxx', processing asynchronously
Starting asynchronous CSV processing for dataset 'xxx' on workspaceId 'xxx'
CSV processing completed for dataset 'xxx', total items: 'xxx'

Problem: HTTP 413 Request Entity Too Large

Solution: Verify nginx configuration includes client_max_body_size: 2g at the server level, not just in location blocks.

Problem: File uploads successfully but items don't appear in the dataset

Solution:

1. Check backend logs for processing errors
2. Verify adequate disk space is available
3. Check memory limits - large CSV files require sufficient memory for processing

Problem: Upload times out before completing

Solution:

1. Increase SERVER_IDLE_TIMEOUT further (e.g., to 15m or 20m)
2. Increase nginx proxy timeouts in upstreamConfig
3. Check network bandwidth between client and server

Problem: Backend service crashes or restarts during processing

Solution:

1. Reduce BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE to process smaller batches
2. Increase backend service memory limits
3. Process smaller CSV files or split large files into multiple uploads

• Scaling Opik - General scaling guidelines
• Kubernetes Deployment - Helm chart documentation
• Troubleshooting - Common issues and solutions