Observability for Portkey with Opik

Portkey is an enterprise-grade AI gateway that provides a unified interface to access 200+ LLMs with advanced features like smart routing, automatic fallbacks, load balancing, and comprehensive observability.

Gateway Overview

Portkey provides enterprise-grade features for managing LLM API access, including:

250+ AI Models: Single consistent API to connect with models from OpenAI, Anthropic, Google, Azure, AWS, and more
Multi-Modal Support: Language, vision, audio, and image models
Advanced Routing: Fallbacks, load balancing, conditional routing based on metadata, and provider weights
Smart Caching: Simple and semantic caching to reduce latency and cost
Security & Governance: Guardrails, secure key management (virtual keys), role-based access control
Compliance: SOC2, HIPAA, GDPR compliant with data privacy controls
Observability: Request/response logging, latency tracking, cost metrics, error rates, and throughput monitoring

Account Setup

Comet provides a hosted version of the Opik platform. Simply create an account and grab your API Key.

You can also run the Opik platform locally, see the installation guide for more information.

Getting Started

Installation

First, ensure you have opik, openai, and portkey-ai packages installed:

Bash

pip install opik openai portkey-ai

Configuring Opik

Configure the Opik Python SDK for your deployment type. See the Python SDK Configuration guide for detailed instructions on:

CLI configuration: opik configure
Code configuration: opik.configure()
Self-hosted vs Cloud vs Enterprise setup
Configuration files and environment variables

Configuring Portkey

You'll need a Portkey API key and virtual keys for your LLM providers. You can get these from the Portkey dashboard.

Set your API keys as environment variables:

Bash

export PORTKEY_API_KEY="YOUR_PORTKEY_API_KEY"
export PORTKEY_VIRTUAL_KEY="YOUR_PORTKEY_VIRTUAL_KEY"

Or set them programmatically:

Python

import os
import getpass

if "PORTKEY_API_KEY" not in os.environ:
    os.environ["PORTKEY_API_KEY"] = getpass.getpass("Enter your Portkey API key: ")

if "PORTKEY_VIRTUAL_KEY" not in os.environ:
    os.environ["PORTKEY_VIRTUAL_KEY"] = getpass.getpass("Enter your Portkey virtual key: ")

Logging LLM Calls

Since Portkey provides an OpenAI-compatible API, we can use the Opik OpenAI SDK wrapper to automatically log Portkey calls as generations in Opik.

Simple LLM Call

Python

import os
from opik.integrations.openai import track_openai
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key=os.environ["PORTKEY_API_KEY"],
        provider="@OPENAI_PROVIDER"
    )
)

# Wrap the client with Opik tracking
client = track_openai(client, project_name="portkey-integration-demo")

# Make a chat completion request
response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are a knowledgeable AI assistant."},
        {"role": "user", "content": "What is the largest city in France?"}
    ]
)

# Print the assistant's reply
print(response.choices[0].message.content)

Advanced Usage

Using with the `@track` decorator

If you have multiple steps in your LLM pipeline, you can use the @track decorator to log the traces for each step. If Portkey is called within one of these steps, the LLM call will be associated with that corresponding step:

Python

import os
from opik import track
from opik.integrations.openai import track_openai
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

# Create an OpenAI client configured for Portkey
client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key=os.environ["PORTKEY_API_KEY"],
        provider="@OPENAI_PROVIDER"
    )
)

# Wrap the client with Opik tracking
client = track_openai(client, project_name="portkey-integration-demo")

@track
def generate_response(prompt: str):
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a knowledgeable AI assistant."},
            {"role": "user", "content": prompt}
        ]
    )
    return response.choices[0].message.content

@track
def refine_response(initial_response: str):
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "You enhance and polish text responses."},
            {"role": "user", "content": f"Please improve this response: {initial_response}"}
        ]
    )
    return response.choices[0].message.content

@track(project_name="portkey-integration-demo")
def generate_and_refine(prompt: str):
    # First LLM call: Generate initial response
    initial = generate_response(prompt)
    
    # Second LLM call: Refine the response
    refined = refine_response(initial)
    
    return refined

# Example usage
result = generate_and_refine("Explain quantum computing in simple terms.")

The trace will show nested LLM calls with hierarchical spans.

Further Improvements

If you have suggestions for improving the Portkey integration, please let us know by opening an issue on GitHub.

Gateway Overview

Account Setup

Getting Started

Installation

Configuring Opik

Configuring Portkey

Logging LLM Calls

Simple LLM Call

Advanced Usage

Using with the @track decorator

Further Improvements

Using with the `@track` decorator