# Datasets
Source: https://docs.zeroeval.com/evaluations/datasets
Create, version, and manage datasets with the ZeroEval Python SDK.
## Why datasets?
**Datasets** are named, versioned collections of rows. Each row is just a Python `dict`. Use them to store test cases for your model and share them across experiments.
## Quick example
```python
import zeroeval as ze
ze.init() # 1οΈβ£ pick up API key from `zeroeval setup`
capitals = ze.Dataset(
name="Capitals",
description="Country β capital mapping",
data=[
{"input": "Colombia", "output": "BogotΓ‘"},
{"input": "Peru", "output": "Lima"},
],
)
capitals.push() # π creates version 1 in your workspace
capitals = ze.Dataset.pull("Capitals") # later, fetch it back
print(capitals[0]) # {'input': 'Colombia', 'output': 'BogotΓ‘'}
```
## Adding rows
```python
capitals.add_rows([{"input": "USA", "output": "Washington, DC"}])
capitals.push() # automatically becomes version 2
```
### Multimodal fields
Images, audio, video or plain URLs are all welcome:
```python
medical = ze.Dataset("Chest_X_Ray", [{"symptoms": "shortness of breath"}])
medical.add_image(0, "scan", "sample_images/patient01.jpg")
medical.add_audio(0, "dictation", "sample_audio/doctor_note.wav")
medical.add_media_url(0, "external_report",
"https://example.com/report.pdf", media_type="image")
medical.push()
```
### Inspecting & slicing
```python
len(capitals) # number of rows
capitals.columns # ['input', 'output']
sample = capitals[:5] # standard list slicing for quick smoke-tests
```
# Experiments
Source: https://docs.zeroeval.com/evaluations/experiments
Run tasks and evaluators on your datasets β locally or in the cloud.
## TL;DR
1. Pull (or create) a dataset
2. Write a **task** β any Python function that maps a row β model output
3. Optionally write **evaluators** β functions that score `(row, output)`
4. Wrap them in `ze.Experiment` and call `.run()`
## Minimal example
```python
import zeroeval as ze
ze.init()
dataset = ze.Dataset.pull("Capitals")
def task(row):
# imagine calling an LLM here
return row["input"].upper()
def exact_match(row, output):
return row["output"].upper() == output
exp = ze.Experiment(
dataset=dataset,
task=task,
evaluators=[exact_match],
name="Capitals-baseline"
)
results = exp.run() # uploads outputs + scores
```
`results` is a list of `ExperimentResult` objects and also appears in the ZeroEval web console.
## Running just the task
Need to iterate fast? Run the task first, add evaluators later:
```python
results = exp.run_task() # only model calls
exp.run_evaluators([exact_match], results) # score later (or change evaluator)
```
## Subsets & slices
```python
exp.run(dataset[:100]) # only the first 100 rows
```
## Tracing and spans
If you have the tracing decorators installed, any spans inside your task are automatically picked up:
```python
import zeroeval as ze
@ze.span(name="model_call")
def task(row):
# autotraced
return call_llm(row["input"])
```
## Multimodal experiments
Experiments work with datasets that contain images/audio/video exactly the same way β simply pass those fields to your model API and write the right evaluator.
Full multimodal example lives in `zeroeval-sdk/examples/experiments.2.multimodal.py`.
# Introduction
Source: https://docs.zeroeval.com/llm-gateway/introduction
A unified interface to seamlessly access and switch between various Large Language Models from different providers.
The LLM Gateway is a unified API that lets you access multiple Large Language Models through a single endpoint. Switch between models from different providers with just a parameter change.
## Getting Started
### 1. Get Your API Key
Create an API key from your [Settings β API Keys](https://app.zeroeval.com/settings?section=api-keys) page.
### 2. Use the API
Replace your OpenAI base URL with ZeroEval's gateway and use model names directly:
```python Python
from openai import OpenAI
# Initialize the client with ZeroEval API
client = OpenAI(
api_key="YOUR_API_KEY",
base_url="https://api.zeroeval.com/v1",
)
# Make a completion request
response = client.chat.completions.create(
model="gpt-4o", # Just the model name, no provider prefix
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello, how are you?"},
],
)
print(response.choices[0].message.content)
```
```typescript TypeScript
import OpenAI from 'openai';
// Initialize the client with ZeroEval API
const client = new OpenAI({
apiKey: 'YOUR_API_KEY',
baseURL: 'https://api.zeroeval.com/v1',
});
// Make a completion request
async function generateResponse() {
const response = await client.chat.completions.create({
model: 'gpt-4o', // Just the model name, no provider prefix
messages: [
{ role: 'system', content: 'You are a helpful assistant.' },
{ role: 'user', content: 'Hello, how are you?' },
],
});
console.log(response.choices[0].message.content);
}
generateResponse();
```
```bash cURL
curl -X POST "https://api.zeroeval.com/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{
"model": "gpt-4o",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Hello, how are you?"
}
]
}'
```
## Available Models
Get a list of available models:
```bash
curl -X GET "https://api.zeroeval.com/v1/models" \
-H "Authorization: Bearer YOUR_API_KEY"
```
# Quickstart
Source: https://docs.zeroeval.com/tracing/quickstart
Get started with tracing and observability in ZeroEval
ZeroEval provides comprehensive tracing capabilities to help you monitor, debug, and optimize your AI applications. This guide introduces the core concepts and helps you get started.
Tracing allows you to monitor and debug your AI applications by capturing detailed information about each operation:
* **Spans**: Individual units of work (e.g., an API call, a function execution)
* **Traces**: Complete request flows composed of multiple spans
* **Sessions**: Groups of related traces (e.g., a user conversation)
* **Attributes**: Metadata attached to spans for filtering and analysis
## Getting Started (in 5 minutes)
### Get your API key
Create an API key from your [Settings β API Keys](https://app.zeroeval.com/settings?section=api-keys) page.
### Install the SDK
Choose your SDK to begin integrating ZeroEval:
For Python applications using frameworks like FastAPI, Django, or Flask
For Node.js, Next.js, and browser-based applications
Need help? Check out our [API reference](/api-reference) or reach out on [Discord](https://discord.gg/MuExkGMNVz).
# Integrations
Source: https://docs.zeroeval.com/tracing/sdks/python/integrations
Automatic instrumentation for popular AI/ML frameworks
The [ZeroEval Python SDK](https://pypi.org/project/zeroeval/) automatically traces intruments the supported integrations, meaning the only thing to do is to initialize the SDK before importing the frameworks you want to trace.
## OpenAI
```python
import zeroeval as ze
ze.init()
import openai
client = openai.OpenAI()
# This call is automatically traced
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}]
)
# Streaming is also automatically traced
stream = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
## LangChain
```python
import zeroeval as ze
ze.init()
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
# All components are automatically traced
model = ChatOpenAI()
prompt = ChatPromptTemplate.from_template("Tell me about {topic}")
chain = prompt | model
response = chain.invoke({"topic": "AI"})
```
## LangGraph
```python
import zeroeval as ze
ze.init()
from langgraph.graph import StateGraph, START, END
from langchain_core.messages import HumanMessage
# Define a multi-node graph
workflow = StateGraph(AgentState)
workflow.add_node("reasoning", reasoning_node)
workflow.add_node("agent", agent_node)
workflow.add_node("tools", tool_node)
workflow.add_conditional_edges(
"agent",
should_continue,
{"tools": "tools", "end": END}
)
app = workflow.compile()
# Full graph execution is automatically traced
result = app.invoke({"messages": [HumanMessage(content="Help me plan a trip")]})
# Streaming is also supported
for chunk in app.stream({"messages": [HumanMessage(content="Hello")]}):
print(chunk)
```
## LiveKit
The SDK automatically creates traces for LiveKit agents, including events from the following plugins:
* Cartesia (TTS)
* Deepgram (STT)
* OpenAI (LLM)
```python
import zeroeval as ze
ze.init()
from livekit import agents
from livekit.agents import AgentSession, Agent
from livekit.plugins import openai
async def entrypoint(ctx: agents.JobContext):
await ctx.connect()
# All agent sessions are automatically traced
session = AgentSession(
llm=openai.realtime.RealtimeModel(voice="coral")
)
await session.start(
room=ctx.room,
agent=Agent(instructions="You are a helpful voice AI assistant.")
)
# Agent interactions are automatically captured
await session.generate_reply(
instructions="Greet the user and offer your assistance."
)
if __name__ == "__main__":
agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))
```
Need help? Contact us at [founders@zeroeval.com](mailto:founders@zeroeval.com) or join our [Discord](https://discord.gg/MuExkGMNVz).
# Reference
Source: https://docs.zeroeval.com/tracing/sdks/python/reference
Complete API reference for the Python SDK
## Installation
```bash
pip install zeroeval
```
## Core Functions
### `init()`
Initializes the ZeroEval SDK. Must be called before using any other SDK features.
```python
def init(
api_key: str = None,
workspace_name: str = "Personal Workspace",
debug: bool = False,
api_url: str = "https://api.zeroeval.com"
) -> None
```
**Parameters:**
* `api_key` (str, optional): Your ZeroEval API key. If not provided, uses `ZEROEVAL_API_KEY` environment variable
* `workspace_name` (str, optional): The name of your workspace. Defaults to `"Personal Workspace"`
* `debug` (bool, optional): If True, enables detailed logging for debugging. Can also be enabled by setting `ZEROEVAL_DEBUG=true` environment variable
* `api_url` (str, optional): The URL of the ZeroEval API. Defaults to `"https://api.zeroeval.com"`
**Example:**
```python
import zeroeval as ze
ze.init(
api_key="your-api-key",
workspace_name="My Workspace",
debug=True
)
```
## Decorators
### `@span`
Decorator and context manager for creating spans around code blocks.
```python
@span(
name: str,
session_id: Optional[str] = None,
session: Optional[Union[str, dict[str, str]]] = None,
attributes: Optional[dict[str, Any]] = None,
input_data: Optional[str] = None,
output_data: Optional[str] = None,
tags: Optional[dict[str, str]] = None
)
```
**Parameters:**
* `name` (str): Name of the span
* `session_id` (str, optional): **Deprecated** - Use `session` parameter instead
* `session` (Union\[str, dict], optional): Session information. Can be:
* A string containing the session ID
* A dict with `{"id": "...", "name": "..."}`
* `attributes` (dict, optional): Additional attributes to attach to the span
* `input_data` (str, optional): Manual input data override
* `output_data` (str, optional): Manual output data override
* `tags` (dict, optional): Tags to attach to the span
**Usage as Decorator:**
```python
import zeroeval as ze
@ze.span(name="calculate_sum")
def add_numbers(a: int, b: int) -> int:
return a + b # Parameters and return value automatically captured
# With manual I/O
@ze.span(name="process_data", input_data="manual input", output_data="manual output")
def process():
# Process logic here
pass
# With session
@ze.span(name="user_action", session={"id": "123", "name": "John's Session"})
def user_action():
pass
```
**Usage as Context Manager:**
```python
import zeroeval as ze
with ze.span(name="data_processing") as current_span:
result = process_data()
current_span.set_io(input_data="input", output_data=str(result))
```
### `@experiment`
Decorator that attaches dataset and model information to a function.
```python
@experiment(
dataset: Optional[Dataset] = None,
model: Optional[str] = None
)
```
**Parameters:**
* `dataset` (Dataset, optional): Dataset to use for the experiment
* `model` (str, optional): Model identifier
**Example:**
```python
import zeroeval as ze
dataset = ze.Dataset.pull("my-dataset")
@ze.experiment(dataset=dataset, model="gpt-4")
def my_experiment():
# Experiment logic
pass
```
## Classes
### `Dataset`
A class to represent a named collection of dictionary records.
#### Constructor
```python
Dataset(
name: str,
data: list[dict[str, Any]],
description: Optional[str] = None
)
```
**Parameters:**
* `name` (str): The name of the dataset
* `data` (list\[dict]): A list of dictionaries containing the data
* `description` (str, optional): A description of the dataset
**Example:**
```python
dataset = Dataset(
name="Capitals",
description="Country to capital mapping",
data=[
{"input": "France", "output": "Paris"},
{"input": "Germany", "output": "Berlin"}
]
)
```
#### Methods
##### `push()`
Push the dataset to the backend, creating a new version if it already exists.
```python
def push(self, create_new_version: bool = False) -> Dataset
```
**Parameters:**
* `self`: The Dataset instance
* `create_new_version` (bool, optional): For backward compatibility. This parameter is no longer needed as new versions are automatically created when a dataset name already exists. Defaults to False
**Returns:** Returns self for method chaining
##### `pull()`
Static method to pull a dataset from the backend.
```python
@classmethod
def pull(
cls,
dataset_name: str,
version_number: Optional[int] = None
) -> Dataset
```
**Parameters:**
* `cls`: The Dataset class itself (automatically provided when using `@classmethod`)
* `dataset_name` (str): The name of the dataset to pull from the backend
* `version_number` (int, optional): Specific version number to pull. If not provided, pulls the latest version
**Returns:** A new Dataset instance populated with data from the backend
##### `add_rows()`
Add new rows to the dataset.
```python
def add_rows(self, new_rows: list[dict[str, Any]]) -> None
```
**Parameters:**
* `self`: The Dataset instance
* `new_rows` (list\[dict]): A list of dictionaries representing the rows to add
##### `add_image()`
Add an image to a specific row.
```python
def add_image(
self,
row_index: int,
column_name: str,
image_path: str
) -> None
```
**Parameters:**
* `self`: The Dataset instance
* `row_index` (int): Index of the row to update (0-based)
* `column_name` (str): Name of the column to add the image to
* `image_path` (str): Path to the image file to add
##### `add_audio()`
Add audio to a specific row.
```python
def add_audio(
self,
row_index: int,
column_name: str,
audio_path: str
) -> None
```
**Parameters:**
* `self`: The Dataset instance
* `row_index` (int): Index of the row to update (0-based)
* `column_name` (str): Name of the column to add the audio to
* `audio_path` (str): Path to the audio file to add
##### `add_media_url()`
Add a media URL to a specific row.
```python
def add_media_url(
self,
row_index: int,
column_name: str,
media_url: str,
media_type: str = "image"
) -> None
```
**Parameters:**
* `self`: The Dataset instance
* `row_index` (int): Index of the row to update (0-based)
* `column_name` (str): Name of the column to add the media URL to
* `media_url` (str): URL pointing to the media file
* `media_type` (str, optional): Type of media - "image", "audio", or "video". Defaults to "image"
#### Properties
* `name` (str): The name of the dataset
* `description` (str): The description of the dataset
* `columns` (list\[str]): List of all unique column names
* `data` (list\[dict]): List of the data portion for each row
* `backend_id` (str): The ID in the backend (after pushing)
* `version_id` (str): The version ID in the backend
* `version_number` (int): The version number in the backend
#### Example
```python
import zeroeval as ze
# Create a dataset
dataset = ze.Dataset(
name="Capitals",
description="Country to capital mapping",
data=[
{"input": "France", "output": "Paris"},
{"input": "Germany", "output": "Berlin"}
]
)
# Push to backend
dataset.push()
# Pull from backend
dataset = ze.Dataset.pull("Capitals", version_number=1)
# Add rows
dataset.add_rows([{"input": "Italy", "output": "Rome"}])
# Add multimodal data
dataset.add_image(0, "flag", "flags/france.png")
dataset.add_audio(0, "anthem", "anthems/france.mp3")
dataset.add_media_url(0, "video_url", "https://example.com/video.mp4", "video")
```
### `Experiment`
Represents an experiment that runs a task on a dataset with optional evaluators.
#### Constructor
```python
Experiment(
dataset: Dataset,
task: Callable[[Any], Any],
evaluators: Optional[list[Callable[[Any, Any], Any]]] = None,
name: Optional[str] = None,
description: Optional[str] = None
)
```
**Parameters:**
* `dataset` (Dataset): The dataset to run the experiment on
* `task` (Callable): Function that processes each row and returns output
* `evaluators` (list\[Callable], optional): List of evaluator functions that take (row, output) and return evaluation result
* `name` (str, optional): Name of the experiment. Defaults to task function name
* `description` (str, optional): Description of the experiment. Defaults to task function's docstring
**Example:**
```python
import zeroeval as ze
ze.init()
# Pull dataset
dataset = ze.Dataset.pull("Capitals")
# Define task
def capitalize_task(row):
return row["input"].upper()
# Define evaluator
def exact_match(row, output):
return row["output"].upper() == output
# Create and run experiment
exp = ze.Experiment(
dataset=dataset,
task=capitalize_task,
evaluators=[exact_match],
name="Capital Uppercase Test"
)
results = exp.run()
# Or run task and evaluators separately
results = exp.run_task()
exp.run_evaluators([exact_match], results)
```
#### Methods
##### `run()`
Run the complete experiment (task + evaluators).
```python
def run(
self,
subset: Optional[list[dict]] = None
) -> list[ExperimentResult]
```
**Parameters:**
* `self`: The Experiment instance
* `subset` (list\[dict], optional): Subset of dataset rows to run the experiment on. If None, runs on entire dataset
**Returns:** List of experiment results for each row
##### `run_task()`
Run only the task without evaluators.
```python
def run_task(
self,
subset: Optional[list[dict]] = None,
raise_on_error: bool = False
) -> list[ExperimentResult]
```
**Parameters:**
* `self`: The Experiment instance
* `subset` (list\[dict], optional): Subset of dataset rows to run the task on. If None, runs on entire dataset
* `raise_on_error` (bool, optional): If True, raises exceptions encountered during task execution. If False, captures errors. Defaults to False
**Returns:** List of experiment results for each row
##### `run_evaluators()`
Run evaluators on existing results.
```python
def run_evaluators(
self,
evaluators: Optional[list[Callable[[Any, Any], Any]]] = None,
results: Optional[list[ExperimentResult]] = None
) -> list[ExperimentResult]
```
**Parameters:**
* `self`: The Experiment instance
* `evaluators` (list\[Callable], optional): List of evaluator functions to run. If None, uses evaluators from the Experiment instance
* `results` (list\[ExperimentResult], optional): List of results to evaluate. If None, uses results from the Experiment instance
**Returns:** The evaluated results
### `Span`
Represents a span in the tracing system. Usually created via the `@span` decorator.
#### Methods
##### `set_io()`
Set input and output data for the span.
```python
def set_io(
self,
input_data: Optional[str] = None,
output_data: Optional[str] = None
) -> None
```
**Parameters:**
* `self`: The Span instance
* `input_data` (str, optional): Input data to attach to the span. Will be converted to string if not already
* `output_data` (str, optional): Output data to attach to the span. Will be converted to string if not already
##### `set_tags()`
Set tags on the span.
```python
def set_tags(self, tags: dict[str, str]) -> None
```
**Parameters:**
* `self`: The Span instance
* `tags` (dict\[str, str]): Dictionary of tags to set on the span
##### `set_attributes()`
Set attributes on the span.
```python
def set_attributes(self, attributes: dict[str, Any]) -> None
```
**Parameters:**
* `self`: The Span instance
* `attributes` (dict\[str, Any]): Dictionary of attributes to set on the span
##### `set_error()`
Set error information for the span.
```python
def set_error(
self,
code: str,
message: str,
stack: Optional[str] = None
) -> None
```
**Parameters:**
* `self`: The Span instance
* `code` (str): Error code or exception class name
* `message` (str): Error message
* `stack` (str, optional): Stack trace information
## Context Functions
### `get_current_span()`
Returns the currently active span, if any.
```python
def get_current_span() -> Optional[Span]
```
**Returns:** The currently active Span instance, or None if no span is active
### `get_current_trace()`
Returns the current trace ID.
```python
def get_current_trace() -> Optional[str]
```
**Returns:** The current trace ID, or None if no trace is active
### `get_current_session()`
Returns the current session ID.
```python
def get_current_session() -> Optional[str]
```
**Returns:** The current session ID, or None if no session is active
### `set_tag()`
Sets tags on a span, trace, or session.
```python
def set_tag(
target: Union[Span, str],
tags: dict[str, str]
) -> None
```
**Parameters:**
* `target`: The target to set tags on
* `Span`: Sets tags on the specific span
* `str`: Sets tags on the trace (if valid trace ID) or session (if valid session ID)
* `tags` (dict\[str, str]): Dictionary of tags to set
**Example:**
```python
import zeroeval as ze
# Set tags on current span
current_span = ze.get_current_span()
if current_span:
ze.set_tag(current_span, {"user_id": "12345", "environment": "production"})
# Set tags on trace
trace_id = ze.get_current_trace()
if trace_id:
ze.set_tag(trace_id, {"version": "1.5"})
```
### `set_signal()`
Send a signal to a span, trace, or session.
```python
def set_signal(
target: Union[Span, str],
signals: dict[str, Union[str, bool, int, float]]
) -> bool
```
**Parameters:**
* `target`: The entity to attach signals to
* `Span`: Sends signals to the specific span
* `str`: Sends signals to the trace (if active trace ID) or session
* `signals` (dict): Dictionary of signal names to values
**Returns:** True if signals were sent successfully, False otherwise
**Example:**
```python
import zeroeval as ze
# Send signals to current span
current_span = ze.get_current_span()
if current_span:
ze.set_signal(current_span, {
"accuracy": 0.95,
"is_successful": True,
"error_count": 0
})
# Send signals to trace
trace_id = ze.get_current_trace()
if trace_id:
ze.set_signal(trace_id, {"model_score": 0.85})
```
## CLI Commands
The ZeroEval SDK includes a CLI tool for running experiments and setup.
### `zeroeval run`
Run a Python script containing ZeroEval experiments.
```bash
zeroeval run script.py
```
### `zeroeval setup`
Interactive setup to configure API credentials.
```bash
zeroeval setup
```
## Environment Variables
The SDK uses the following environment variables:
* `ZEROEVAL_API_KEY`: Your ZeroEval API key
* `ZEROEVAL_API_URL`: API endpoint URL (defaults to `https://api.zeroeval.com`)
* `ZEROEVAL_DEBUG`: Set to `true` to enable debug logging
* `ZEROEVAL_DISABLED_INTEGRATIONS`: Comma-separated list of integrations to disable
# Setup
Source: https://docs.zeroeval.com/tracing/sdks/python/setup
Get started with ZeroEval tracing in Python applications
The [ZeroEval Python SDK](https://pypi.org/project/zeroeval/) provides seamless integration with your Python applications through automatic instrumentation and a simple decorator-based API.
## Installation
```bash pip
pip install zeroeval
```
```bash poetry
poetry add zeroeval
```
## Basic Setup
```python
import zeroeval as ze
# Option 1: ZEROEVAL_API_KEY in your environment variable file
ze.init()
# Option 2: Provide API key directly from
# https://app.zeroeval.com/settings?tab=api-keys
ze.init(api_key="YOUR_API_KEY")
```
Run `zeroeval setup` once to save your API key securely to
`~/.config/zeroeval/config.json`
## Patterns
### Decorators
The `@span` decorator is the easiest way to add tracing:
```python
import zeroeval as ze
@ze.span(name="fetch_data")
def fetch_data(user_id: str):
# Function arguments are automatically captured as inputs
# Return values are automatically captured as outputs
return {"user_id": user_id, "name": "John Doe"}
@ze.span(name="process_data", attributes={"version": "1.0"})
def process_data(data: dict):
# Add custom attributes for better filtering
return f"Welcome, {data['name']}!"
```
### Context Manager
For more control over span lifecycles:
```python
import zeroeval as ze
def complex_workflow():
with ze.span(name="data_pipeline") as pipeline_span:
# Fetch stage
with ze.span(name="fetch_stage") as fetch_span:
data = fetch_external_data()
fetch_span.set_io(output_data=str(data))
# Process stage
with ze.span(name="process_stage") as process_span:
processed = transform_data(data)
process_span.set_io(
input_data=str(data),
output_data=str(processed)
)
# Save stage
with ze.span(name="save_stage") as save_span:
result = save_to_database(processed)
save_span.set_io(output_data=f"Saved {result} records")
```
## Advanced Configuration
Fine-tune the tracer behavior:
```python
from zeroeval.observability.tracer import tracer
# Configure tracer settings
tracer.configure(
flush_interval=5.0, # Flush every 5 seconds
max_spans=200, # Buffer up to 200 spans
collect_code_details=True # Capture source code context
)
```
## Context
Access current context information:
```python
# Get the current span
current_span = ze.get_current_span()
# Get the current trace ID
trace_id = ze.get_current_trace()
# Get the current session ID
session_id = ze.get_current_session()
```
## CLI Tooling
The Python SDK includes helpful CLI commands:
```bash
# Save your API key securely
zeroeval setup
# Run scripts with automatic tracing
zeroeval run my_script.py
```
# Integrations
Source: https://docs.zeroeval.com/tracing/sdks/typescript/integrations
Tracing integrations with popular libraries
## OpenAI
```typescript
import { OpenAI } from 'openai';
import * as ze from 'zeroeval';
const openai = ze.wrap(new OpenAI());
const completion = await openai.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: 'Hello!' }]
});
const stream = await openai.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: 'Tell me a story' }],
stream: true
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || '');
}
```
## Vercel AI SDK
```typescript
import * as ai from 'ai';
import { openai } from '@ai-sdk/openai';
import * as ze from 'zeroeval';
const wrappedAI = ze.wrap(ai);
// Text generation
const { text } = await wrappedAI.generateText({
model: openai('gpt-4'),
prompt: 'Write a haiku about coding'
});
// Streaming
const { textStream } = await wrappedAI.streamText({
model: openai('gpt-4'),
messages: [{ role: 'user', content: 'Hello!' }]
});
for await (const delta of textStream) {
process.stdout.write(delta);
}
// Structured output
const { object } = await wrappedAI.generateObject({
model: openai('gpt-4'),
schema: z.object({
name: z.string(),
age: z.number()
}),
prompt: 'Generate a random person'
});
```
## LangChain / LangGraph
```typescript
import {
ZeroEvalCallbackHandler,
setGlobalCallbackHandler
} from 'zeroeval/langchain';
// Set globally, no need to pass on each individual call
setGlobalCallbackHandler(new ZeroEvalCallbackHandler());
// OPTIONAL: alternatively use per-invocation
const handler = new ZeroEvalCallbackHandler();
const result = await chain.invoke(
{ topic: 'AI' },
{ callbacks: [handler] }
);
```
Need help? Check out our [GitHub examples](https://github.com/zeroeval/zeroeval-ts-sdk/tree/main/examples) or reach out on [Discord](https://discord.gg/MuExkGMNVz).
# Reference
Source: https://docs.zeroeval.com/tracing/sdks/typescript/reference
Complete API reference for the TypeScript SDK
## Installation
```bash
npm install zeroeval
```
## Core Functions
### `init()`
Initializes the ZeroEval SDK. Must be called before using any other SDK features.
```typescript
function init(opts?: InitOptions): void
```
#### Parameters
* `opts` (optional): `InitOptions`
* `apiKey` (optional): `string` - Your ZeroEval API key. If not provided, uses `ZEROEVAL_API_KEY` environment variable
* `apiUrl` (optional): `string` - Custom API URL. Defaults to `https://api.zeroeval.com`
* `flushInterval` (optional): `number` - Interval in milliseconds to flush spans
* `maxSpans` (optional): `number` - Maximum number of spans to buffer before flushing
* `collectCodeDetails` (optional): `boolean` - Whether to collect code location details
* `integrations` (optional): `Record` - Enable/disable specific integrations
* `debug` (optional): `boolean` - Enable debug logging
#### Example
```typescript
import * as ze from 'zeroeval';
ze.init({
apiKey: 'your-api-key',
debug: true
});
```
## Wrapper Functions
### `wrap()`
Wraps a supported AI client to automatically trace all API calls.
```typescript
function wrap(client: T): WrappedClient
```
#### Supported Clients
* OpenAI SDK (`openai` package)
* Vercel AI SDK (`ai` package)
#### Examples
```typescript
// OpenAI
import { OpenAI } from 'openai';
import * as ze from 'zeroeval';
const openai = ze.wrap(new OpenAI());
// Vercel AI SDK
import * as ai from 'ai';
import * as ze from 'zeroeval';
const wrappedAI = ze.wrap(ai);
```
## Context Functions
### `getCurrentSpan()`
Returns the currently active span, if any.
```typescript
function getCurrentSpan(): Span | undefined
```
### `getCurrentTrace()`
Returns the current trace ID.
```typescript
function getCurrentTrace(): string | undefined
```
### `getCurrentSession()`
Returns the current session ID.
```typescript
function getCurrentSession(): string | undefined
```
### `setTag()`
Sets tags on a span, trace, or session.
```typescript
function setTag(
target: Span | string | undefined,
tags: Record
): void
```
#### Parameters
* `target`: The target to set tags on
* `Span`: Sets tags on the specific span
* `string`: Sets tags on the trace (if valid trace ID) or session (if valid session ID)
* `undefined`: Sets tags on the current span
* `tags`: Object containing key-value pairs of tags
#### Example
```typescript
// Set tags on current span
ze.setTag(undefined, { user_id: '12345', environment: 'production' });
// Set tags on specific trace
const traceId = ze.getCurrentTrace();
if (traceId) {
ze.setTag(traceId, { feature: 'checkout' });
}
// Set tags on a span object
const span = ze.getCurrentSpan();
if (span) {
ze.setTag(span, { action: 'process_payment' });
}
```
## Spans API
There are two main ways to create spans in the TypeScript SDK:
### `withSpan()`
Wraps a function execution in a span, automatically capturing input/output and timing.
```typescript
function withSpan(
opts: SpanOptions,
fn: () => Promise | T
): Promise | T
```
**Parameters:**
* `opts` (SpanOptions): Configuration for the span
* `name` (string): Name of the span
* `sessionId` (string, optional): Session ID to associate with the span
* `sessionName` (string, optional): Human-readable session name
* `tags` (object, optional): Tags to attach to the span
* `attributes` (object, optional): Additional attributes
* `inputData` (any, optional): Manual input data override
* `outputData` (any, optional): Manual output data override
* `fn` (Function): The function to execute within the span
**Example:**
```typescript
import * as ze from 'zeroeval';
// Basic usage
const result = await ze.withSpan(
{ name: 'fetch-user-data' },
async () => {
const user = await fetchUser(userId);
return user;
}
);
// With session and tags
const data = ze.withSpan(
{
name: 'process-payment',
sessionId: sessionId,
tags: { environment: 'production', version: '1.0' }
},
() => processPayment(amount)
);
```
### `@span` Decorator
Decorator for class methods to automatically create spans. Requires TypeScript with experimental decorators enabled.
```typescript
span(opts: SpanOptions): MethodDecorator
```
**Parameters:**
* `opts` (SpanOptions): Same configuration options as `withSpan()`
**Example:**
```typescript
import * as ze from 'zeroeval';
class UserService {
@ze.span({ name: 'get-user' })
async getUser(id: string): Promise {
// Method implementation
// Input (id) and output (User) are automatically captured
return await db.users.findById(id);
}
@ze.span({
name: 'update-user',
tags: { operation: 'update' }
})
async updateUser(id: string, data: Partial): Promise {
return await db.users.update(id, data);
}
}
```
**Note:** To use decorators, ensure your `tsconfig.json` includes:
```json
{
"compilerOptions": {
"experimentalDecorators": true
}
}
```
## Signals API
### `sendSignal()`
Send a signal to a specific entity.
```typescript
async function sendSignal(
entityType: 'session' | 'trace' | 'span' | 'completion',
entityId: string,
name: string,
value: string | boolean | number,
signalType?: 'boolean' | 'numerical'
): Promise
```
#### Parameters
* `entityType`: Type of entity to attach the signal to
* `entityId`: UUID of the entity
* `name`: Name of the signal
* `value`: Signal value (string, boolean, or number)
* `signalType` (optional): Signal type, auto-detected if not provided
### `sendTraceSignal()`
Send a signal to the current trace.
```typescript
function sendTraceSignal(
name: string,
value: string | boolean | number,
signalType?: 'boolean' | 'numerical'
): void
```
### `sendSessionSignal()`
Send a signal to the current session.
```typescript
function sendSessionSignal(
name: string,
value: string | boolean | number,
signalType?: 'boolean' | 'numerical'
): void
```
### `sendSpanSignal()`
Send a signal to the current span.
```typescript
function sendSpanSignal(
name: string,
value: string | boolean | number,
signalType?: 'boolean' | 'numerical'
): void
```
### `getEntitySignals()`
Retrieve signals for a specific entity.
```typescript
async function getEntitySignals(
entityType: 'session' | 'trace' | 'span' | 'completion',
entityId: string
): Promise
```
## LangChain Integration
### `ZeroEvalCallbackHandler`
A callback handler for integrating with LangChain.
```typescript
class ZeroEvalCallbackHandler extends BaseCallbackHandler
```
#### Constructor
```typescript
constructor(options?: ZeroEvalCallbackHandlerOptions)
```
#### Options
* `debug` (optional): `boolean` - Enable debug logging
* `excludeMetadataProps` (optional): `RegExp` - Pattern for metadata properties to exclude
* `maxConcurrentSpans` (optional): `number` - Maximum concurrent spans. Defaults to 1000
* `spanCleanupIntervalMs` (optional): `number` - Cleanup interval in milliseconds. Defaults to 60000
#### Example
```typescript
import { ZeroEvalCallbackHandler } from 'zeroeval/langchain';
const handler = new ZeroEvalCallbackHandler({
debug: true,
maxConcurrentSpans: 500
});
// Use with LangChain
const chain = new ConversationChain({
callbacks: [handler]
});
```
### `setGlobalCallbackHandler()`
Sets a global callback handler for LangChain.
```typescript
function setGlobalCallbackHandler(handler: ZeroEvalCallbackHandler): void
```
### `getGlobalHandler()`
Gets the current global callback handler.
```typescript
function getGlobalHandler(): BaseCallbackHandler | undefined
```
### `clearGlobalHandler()`
Clears the global callback handler.
```typescript
function clearGlobalHandler(): void
```
## Types
### `InitOptions`
Configuration options for SDK initialization.
```typescript
interface InitOptions {
apiKey?: string;
apiUrl?: string;
workspaceName?: string;
flushInterval?: number;
maxSpans?: number;
collectCodeDetails?: boolean;
integrations?: Record;
debug?: boolean;
}
```
### `SignalCreate`
Structure for creating a new signal.
```typescript
interface SignalCreate {
entity_type: 'session' | 'trace' | 'span' | 'completion';
entity_id: string;
name: string;
value: string | boolean | number;
signal_type?: 'boolean' | 'numerical';
}
```
### `Signal`
Structure representing a signal.
```typescript
interface Signal {
value: string | boolean | number;
type: 'boolean' | 'numerical';
}
```
### `ZeroEvalCallbackHandlerOptions`
Options for the LangChain callback handler.
```typescript
interface ZeroEvalCallbackHandlerOptions {
debug?: boolean;
excludeMetadataProps?: RegExp;
maxConcurrentSpans?: number;
spanCleanupIntervalMs?: number;
}
```
## Environment Variables
The SDK uses the following environment variables:
* `ZEROEVAL_API_KEY`: Your ZeroEval API key
* `ZEROEVAL_API_URL`: API endpoint URL (defaults to `https://api.zeroeval.com`)
* `ZEROEVAL_DEBUG`: Set to `true` to enable debug logging
# Setup
Source: https://docs.zeroeval.com/tracing/sdks/typescript/setup
Get started with ZeroEval tracing in TypeScript and JavaScript applications
The [ZeroEval TypeScript SDK](https://www.npmjs.com/package/zeroeval) provides tracing for Node.js and browser applications through wrapper functions and integration callbacks.
## Installation
```bash npm
npm install zeroeval
```
```bash yarn
yarn add zeroeval
```
```bash pnpm
pnpm add zeroeval
```
## Basic Setup
```ts
import * as ze from 'zeroeval';
// Option 1: ZEROEVAL_API_KEY in your environment variable file
ze.init();
// Option 2: API key
ze.init({ apiKey: 'YOUR_API_KEY' });
// Option 3: With additional configuration
ze.init({
apiKey: 'YOUR_API_KEY',
apiUrl: 'https://api.zeroeval.com', // optional
flushInterval: 10, // seconds
maxSpans: 100,
});
```
## Patterns
The SDK offers two ways to add tracing to your TypeScript/JavaScript code:
### Basic Usage
```ts Function Wrapping
import * as ze from 'zeroeval';
// Wrap synchronous functions
const fetchData = (userId: string) =>
ze.withSpan({ name: 'fetch_data' }, () => ({
userId,
name: 'John Doe'
}));
// Wrap async functions
const processData = async (data: { name: string }) =>
ze.withSpan(
{
name: 'process_data',
attributes: { version: '1.0' }
},
async () => {
const result = await transform(data);
return `Welcome, ${result.name}!`;
}
);
// Complex workflows with nested spans
async function complexWorkflow() {
return ze.withSpan({ name: 'data_pipeline' }, async () => {
const data = await ze.withSpan(
{ name: 'fetch_stage' },
fetchExternalData
);
const processed = await ze.withSpan(
{ name: 'process_stage' },
() => transformData(data)
);
const result = await ze.withSpan(
{ name: 'save_stage' },
() => saveToDatabase(processed)
);
return result;
});
}
```
```ts Decorators
import { span } from 'zeroeval';
class DataService {
@span({
name: 'fetch_user_data',
tags: { service: 'user_api' }
})
async fetchUser(userId: string) {
const response = await fetch(`/api/users/${userId}`);
return response.json();
}
@span({
name: 'process_order',
attributes: { version: '2.0' }
})
processOrder(orderId: string, items: string[]) {
return { orderId, processed: true };
}
}
// TypeScript Configuration Required:
// Add to your tsconfig.json:
// {
// "compilerOptions": {
// "experimentalDecorators": true,
// "emitDecoratorMetadata": true
// }
// }
// When using tsx or ts-node:
// tsx --experimental-decorators your-file.ts
// ts-node --experimental-decorators your-file.ts
```
**Decorators require TypeScript configuration**: Enable `experimentalDecorators` and `emitDecoratorMetadata` in your `tsconfig.json`. When using runtime tools like `tsx` or `ts-node`, pass the `--experimental-decorators` flag.
### Sessions
Group related spans into sessions:
```ts
import { v4 as uuidv4 } from 'uuid';
const sessionId = uuidv4();
async function userJourney(userId: string) {
return ze.withSpan(
{
name: 'user_journey',
sessionId: sessionId,
sessionName: `User ${userId} Session`
},
async () => {
await login(userId);
await browseProducts();
await checkout();
}
);
}
```
## Context
Access current context information:
```ts
import { getCurrentSpan, getCurrentTrace, getCurrentSession } from 'zeroeval';
function myFunction() {
// Get current span
const span = getCurrentSpan();
// Get current trace ID
const traceId = getCurrentTrace();
// Get current session ID
const sessionId = getCurrentSession();
}
```
# Sessions
Source: https://docs.zeroeval.com/tracing/sessions
Group related spans into sessions for better organization and analysis
Sessions provide a powerful way to group related spans together, making it easier to track and analyze complex workflows, user interactions, or multi-step processes. This guide covers everything you need to know about working with sessions.
For complete API documentation, see the [Python SDK Reference](/tracing/sdks/python/reference) or [TypeScript SDK Reference](/tracing/sdks/typescript/reference).
## Creating Sessions
### Basic Session with ID
The simplest way to create a session is by providing a session ID:
```python Python
import uuid
import zeroeval as ze
# Generate a unique session ID
session_id = str(uuid.uuid4())
@ze.span(name="process_request", session=session_id)
def process_request(data):
# This span belongs to the session
return transform_data(data)
```
```typescript TypeScript (Basic)
import { randomUUID } from 'crypto';
import * as ze from 'zeroeval';
// Generate a unique session ID
const sessionId = randomUUID();
function processRequest(data: any) {
return ze.withSpan({
name: "process_request",
sessionId
}, () => {
// This span belongs to the session
return transformData(data);
});
}
```
```typescript TypeScript (Decorators)
import { randomUUID } from 'crypto';
import { span } from 'zeroeval';
// Generate a unique session ID
const sessionId = randomUUID();
class RequestProcessor {
@span({ name: "process_request", sessionId })
processRequest(data: any) {
// This span belongs to the session
return transformData(data);
}
}
```
### Named Sessions
For better organization in the ZeroEval dashboard, you can provide both an ID and a descriptive name:
```python Python
@ze.span(
name="user_interaction",
session={
"id": session_id,
"name": "Customer Support Chat - User #12345"
}
)
def handle_support_chat(user_id, message):
# Process the support request
return generate_response(message)
```
```typescript TypeScript (Basic)
function handleSupportChat(userId: string, message: string) {
return ze.withSpan({
name: "user_interaction",
sessionId: sessionId,
sessionName: "Customer Support Chat - User #12345"
}, () => {
// Process the support request
return generateResponse(message);
});
}
```
```typescript TypeScript (Decorators)
class SupportHandler {
@span({
name: "user_interaction",
sessionId: sessionId,
sessionName: "Customer Support Chat - User #12345"
})
handleSupportChat(userId: string, message: string) {
// Process the support request
return generateResponse(message);
}
}
```
## Session Inheritance
Child spans automatically inherit the session from their parent span:
```python Python
session_info = {
"id": str(uuid.uuid4()),
"name": "Order Processing Pipeline"
}
@ze.span(name="process_order", session=session_info)
def process_order(order_id):
# These nested calls automatically belong to the same session
validate_order(order_id)
charge_payment(order_id)
fulfill_order(order_id)
@ze.span(name="validate_order")
def validate_order(order_id):
# Automatically part of the parent's session
return check_inventory(order_id)
@ze.span(name="charge_payment")
def charge_payment(order_id):
# Also inherits the session
return process_payment(order_id)
```
```typescript TypeScript (Basic)
const sessionInfo = {
id: randomUUID(),
name: "Order Processing Pipeline"
};
function processOrder(orderId: string) {
return ze.withSpan({
name: "process_order",
sessionId: sessionInfo.id,
sessionName: sessionInfo.name
}, () => {
// These nested calls automatically belong to the same session
validateOrder(orderId);
chargePayment(orderId);
fulfillOrder(orderId);
});
}
function validateOrder(orderId: string) {
return ze.withSpan({ name: "validate_order" }, () => {
// Automatically part of the parent's session
return checkInventory(orderId);
});
}
function chargePayment(orderId: string) {
return ze.withSpan({ name: "charge_payment" }, () => {
// Also inherits the session
return processPayment(orderId);
});
}
```
```typescript TypeScript (Decorators)
const sessionInfo = {
id: randomUUID(),
name: "Order Processing Pipeline"
};
class OrderProcessor {
@span({
name: "process_order",
sessionId: sessionInfo.id,
sessionName: sessionInfo.name
})
processOrder(orderId: string) {
// These nested calls automatically belong to the same session
this.validateOrder(orderId);
this.chargePayment(orderId);
this.fulfillOrder(orderId);
}
@span({ name: "validate_order" })
validateOrder(orderId: string) {
// Automatically part of the parent's session
return checkInventory(orderId);
}
@span({ name: "charge_payment" })
chargePayment(orderId: string) {
// Also inherits the session
return processPayment(orderId);
}
fulfillOrder(orderId: string) {
// Not traced
return fulfillOrder(orderId);
}
}
```
## Advanced Session Patterns
### Multi-Agent RAG System
Track complex retrieval-augmented generation workflows with multiple specialized agents:
```python Python
session = {
"id": str(uuid.uuid4()),
"name": "Multi-Agent RAG Pipeline"
}
@ze.span(name="rag_coordinator", session=session)
async def process_query(query):
# Retrieval
docs = await retrieval_agent(query)
# Reranking
ranked = await reranking_agent(query, docs)
# Generation
response = await generation_agent(query, ranked)
return response
@ze.span(name="retrieval_agent")
async def retrieval_agent(query):
# Inherits session from parent
embeddings = await embed(query)
return await vector_search(embeddings)
@ze.span(name="generation_agent")
async def generation_agent(query, context):
return await llm.generate(query, context)
```
```typescript TypeScript (Basic)
const session = {
id: randomUUID(),
name: "Multi-Agent RAG Pipeline"
};
async function processQuery(query: string) {
return ze.withSpan({
name: "rag_coordinator",
sessionId: session.id,
sessionName: session.name
}, async () => {
// Retrieval
const docs = await retrievalAgent(query);
// Reranking
const ranked = await rerankingAgent(query, docs);
// Generation
const response = await generationAgent(query, ranked);
return response;
});
}
async function retrievalAgent(query: string) {
return ze.withSpan({ name: "retrieval_agent" }, async () => {
// Inherits session from parent
const embeddings = await embed(query);
return await vectorSearch(embeddings);
});
}
async function generationAgent(query: string, context: any) {
return ze.withSpan({ name: "generation_agent" }, async () => {
return await llm.generate(query, context);
});
}
```
```typescript TypeScript (Decorators)
const session = {
id: randomUUID(),
name: "Multi-Agent RAG Pipeline"
};
class RAGPipeline {
@span({
name: "rag_coordinator",
sessionId: session.id,
sessionName: session.name
})
async processQuery(query: string) {
// Retrieval
const docs = await this.retrievalAgent(query);
// Reranking
const ranked = await this.rerankingAgent(query, docs);
// Generation
const response = await this.generationAgent(query, ranked);
return response;
}
@span({ name: "retrieval_agent" })
async retrievalAgent(query: string) {
// Inherits session from parent
const embeddings = await embed(query);
return await vectorSearch(embeddings);
}
@span({ name: "generation_agent" })
async generationAgent(query: string, context: any) {
return await llm.generate(query, context);
}
async rerankingAgent(query: string, docs: any[]) {
// Not traced
return await rerank(query, docs);
}
}
```
### Conversational AI Session
Track a complete conversation with an AI assistant:
```python Python
class ChatSession:
def __init__(self, user_id):
self.session = {
"id": f"chat-{user_id}-{uuid.uuid4()}",
"name": f"AI Chat - User {user_id}"
}
self.history = []
@ze.span(name="process_message", session=lambda self: self.session)
async def process_message(self, message):
# Add to history
self.history.append({"role": "user", "content": message})
# Generate response
response = await self.generate_response()
self.history.append({"role": "assistant", "content": response})
return response
@ze.span(name="generate_response", session=lambda self: self.session)
async def generate_response(self):
return await llm.chat(self.history)
```
```typescript TypeScript (Basic)
class ChatSession {
private session: { id: string; name: string };
private history: any[] = [];
constructor(userId: string) {
this.session = {
id: `chat-${userId}-${randomUUID()}`,
name: `AI Chat - User ${userId}`
};
}
async processMessage(message: string) {
return ze.withSpan({
name: "process_message",
sessionId: this.session.id,
sessionName: this.session.name
}, async () => {
// Add to history
this.history.push({ role: "user", content: message });
// Generate response
const response = await this.generateResponse();
this.history.push({ role: "assistant", content: response });
return response;
});
}
async generateResponse() {
return ze.withSpan({
name: "generate_response",
sessionId: this.session.id,
sessionName: this.session.name
}, async () => {
return await llm.chat(this.history);
});
}
}
```
```typescript TypeScript (Decorators)
import { span } from 'zeroeval';
class ChatSession {
private session: { id: string; name: string };
private history: any[] = [];
constructor(userId: string) {
this.session = {
id: `chat-${userId}-${randomUUID()}`,
name: `AI Chat - User ${userId}`
};
}
@span({
name: "process_message",
sessionId: function(this: ChatSession) { return this.session.id; },
sessionName: function(this: ChatSession) { return this.session.name; }
})
async processMessage(message: string) {
// Add to history
this.history.push({ role: "user", content: message });
// Generate response
const response = await this.generateResponse();
this.history.push({ role: "assistant", content: response });
return response;
}
@span({
name: "generate_response",
sessionId: function(this: ChatSession) { return this.session.id; },
sessionName: function(this: ChatSession) { return this.session.name; }
})
async generateResponse() {
return await llm.chat(this.history);
}
}
```
### Batch LLM Processing
Process multiple documents with LLMs in a single session:
```python Python
async def batch_summarize(documents):
session = {
"id": f"batch-{uuid.uuid4()}",
"name": f"Batch Summarization - {len(documents)} docs"
}
@ze.span(name="batch_processor", session=session)
async def process():
summaries = []
for i, doc in enumerate(documents):
with ze.span(name=f"summarize_doc_{i}", session=session) as span:
try:
summary = await llm.summarize(doc)
span.set_io(
input_data=f"Doc: {doc['title']}",
output_data=summary[:100]
)
summaries.append(summary)
except Exception as e:
span.set_error(
code=type(e).__name__,
message=str(e)
)
return summaries
return await process()
```
```typescript TypeScript (Basic)
async function batchSummarize(documents: any[]) {
const session = {
id: `batch-${randomUUID()}`,
name: `Batch Summarization - ${documents.length} docs`
};
return ze.withSpan({
name: "batch_processor",
sessionId: session.id,
sessionName: session.name
}, async () => {
const summaries = [];
for (let i = 0; i < documents.length; i++) {
await ze.withSpan({
name: `summarize_doc_${i}`,
sessionId: session.id,
sessionName: session.name
}, async () => {
try {
const summary = await llm.summarize(documents[i]);
const span = ze.getCurrentSpan();
if (span) {
span.setIO(
`Doc: ${documents[i].title}`,
summary.substring(0, 100)
);
}
summaries.push(summary);
} catch (e: any) {
const span = ze.getCurrentSpan();
if (span) {
span.setError({
code: e.constructor.name,
message: e.message
});
}
}
});
}
return summaries;
});
}
```
```typescript TypeScript (Decorators)
class BatchProcessor {
private session = {
id: `batch-${randomUUID()}`,
name: `Batch Summarization`
};
@span({
name: "batch_processor",
sessionId: function(this: BatchProcessor) { return this.session.id; },
sessionName: function(this: BatchProcessor) { return `${this.session.name} - ${this.documents.length} docs`; }
})
async batchSummarize(documents: any[]) {
const summaries = [];
for (let i = 0; i < documents.length; i++) {
const summary = await this.summarizeDoc(documents[i], i);
if (summary) {
summaries.push(summary);
}
}
return summaries;
}
@span({
name: function(this: BatchProcessor, _doc: any, index: number) { return `summarize_doc_${index}`; },
sessionId: function(this: BatchProcessor) { return this.session.id; },
sessionName: function(this: BatchProcessor) { return this.session.name; }
})
async summarizeDoc(doc: any, index: number) {
try {
const summary = await llm.summarize(doc);
const span = ze.getCurrentSpan();
if (span) {
span.setIO(
`Doc: ${doc.title}`,
summary.substring(0, 100)
);
}
return summary;
} catch (e: any) {
const span = ze.getCurrentSpan();
if (span) {
span.setError({
code: e.constructor.name,
message: e.message
});
}
return null;
}
}
private documents: any[] = [];
}
```
## Context Manager Sessions
You can also use sessions with the context manager pattern:
```python Python
session_info = {
"id": str(uuid.uuid4()),
"name": "Data Pipeline Run"
}
with ze.span(name="etl_pipeline", session=session_info) as pipeline_span:
# Extract phase
with ze.span(name="extract_data") as extract_span:
raw_data = fetch_from_source()
extract_span.set_io(output_data=f"Extracted {len(raw_data)} records")
# Transform phase
with ze.span(name="transform_data") as transform_span:
clean_data = transform_records(raw_data)
transform_span.set_io(
input_data=f"{len(raw_data)} raw records",
output_data=f"{len(clean_data)} clean records"
)
# Load phase
with ze.span(name="load_data") as load_span:
result = save_to_destination(clean_data)
load_span.set_io(output_data=f"Loaded to {result['location']}")
```
```typescript TypeScript (Basic)
const sessionInfo = {
id: randomUUID(),
name: "Data Pipeline Run"
};
await ze.withSpan({
name: "etl_pipeline",
sessionId: sessionInfo.id,
sessionName: sessionInfo.name
}, async () => {
// Extract phase
const rawData = await ze.withSpan({ name: "extract_data" }, async () => {
const data = await fetchFromSource();
const span = ze.getCurrentSpan();
if (span) {
span.setIO(undefined, `Extracted ${data.length} records`);
}
return data;
});
// Transform phase
const cleanData = await ze.withSpan({ name: "transform_data" }, async () => {
const data = transformRecords(rawData);
const span = ze.getCurrentSpan();
if (span) {
span.setIO(
`${rawData.length} raw records`,
`${data.length} clean records`
);
}
return data;
});
// Load phase
await ze.withSpan({ name: "load_data" }, async () => {
const result = await saveToDestination(cleanData);
const span = ze.getCurrentSpan();
if (span) {
span.setIO(undefined, `Loaded to ${result.location}`);
}
});
});
```
```typescript TypeScript (Decorators)
import { span } from 'zeroeval';
import * as ze from 'zeroeval';
class ETLPipeline {
private sessionInfo = {
id: randomUUID(),
name: "Data Pipeline Run"
};
@span({
name: "etl_pipeline",
sessionId: function(this: ETLPipeline) { return this.sessionInfo.id; },
sessionName: function(this: ETLPipeline) { return this.sessionInfo.name; }
})
async runPipeline() {
// Extract phase
const rawData = await this.extractData();
// Transform phase
const cleanData = await this.transformData(rawData);
// Load phase
await this.loadData(cleanData);
}
@span({ name: "extract_data" })
async extractData() {
const data = await fetchFromSource();
const span = ze.getCurrentSpan();
if (span) {
span.setIO(undefined, `Extracted ${data.length} records`);
}
return data;
}
@span({ name: "transform_data" })
transformData(rawData: any[]) {
const data = transformRecords(rawData);
const span = ze.getCurrentSpan();
if (span) {
span.setIO(
`${rawData.length} raw records`,
`${data.length} clean records`
);
}
return data;
}
@span({ name: "load_data" })
async loadData(cleanData: any[]) {
const result = await saveToDestination(cleanData);
const span = ze.getCurrentSpan();
if (span) {
span.setIO(undefined, `Loaded to ${result.location}`);
}
}
}
```
# Signals
Source: https://docs.zeroeval.com/tracing/signals
Capture real-world feedback and metrics to enrich your traces, spans, and sessions.
Signals are any piece of user feedback, behavior, or metric you care about β thumbs-up, a 5-star rating, dwell time, task completion, error rates β¦ you name it. Signals help you understand how your AI system performs in the real world by connecting user outcomes to your traces.
You can attach signals to:
* **Completions** (LLM responses)
* **Spans** (individual operations)
* **Sessions** (user interactions)
* **Traces** (entire request flows)
For complete signals API documentation, see the [Python SDK Reference](/tracing/sdks/python/reference#signals) or [TypeScript SDK Reference](/tracing/sdks/typescript/reference#signals).
## Using signals in code
### With the Python SDK
```python Python
import zeroeval as ze
# Initialize the tracer
ze.init(api_key="your-api-key")
# Start a span and add a signal
with ze.trace("user_query") as span:
# Your AI logic here
response = process_user_query(query)
# Add a signal to the current span
ze.set_signal("user_satisfaction", True)
ze.set_signal("response_quality", 4.5)
ze.set_signal("task_completed", "success")
```
```typescript TypeScript (Basic)
import * as ze from 'zeroeval';
// Initialise the tracer
ze.init({ apiKey: "your-api-key" });
// Start a span and add signals
await ze.withSpan({ name: "user_query" }, async () => {
const response = await processUserQuery(query);
// --- Add signals on the current span ---
await ze.sendSpanSignal("user_satisfaction", true);
await ze.sendSpanSignal("response_quality", 4.5);
// --- Attach to the whole trace / session ---
await ze.sendTraceSignal("task_completed", "success");
await ze.sendSessionSignal("vip_user", true);
});
```
```typescript TypeScript (Decorators)
import { span } from 'zeroeval';
import * as ze from 'zeroeval';
// Initialise the tracer
ze.init({ apiKey: "your-api-key" });
class QueryProcessor {
@span({ name: "user_query" })
async processQuery(query: string) {
const response = await processUserQuery(query);
// --- Add signals on the current span ---
await ze.sendSpanSignal("user_satisfaction", true);
await ze.sendSpanSignal("response_quality", 4.5);
// --- Attach to the whole trace / session ---
await ze.sendTraceSignal("task_completed", "success");
await ze.sendSessionSignal("vip_user", true);
return response;
}
}
```
### Setting signals on different targets
```python Python
# On the current span
ze.set_signal("helpful", True)
# On a specific span
span = ze.current_span()
ze.set_signal(span, {"rating": 5, "category": "excellent"})
# On the current trace
ze.set_trace_signal("conversion", True)
# On the current session
ze.set_session_signal("user_engaged", True)
```
```typescript TypeScript
import * as ze from 'zeroeval';
// On the current span
await ze.sendSpanSignal("helpful", true);
// On a specific span object
const span = ze.getCurrentSpan();
span?.addSignal("rating", 5);
span?.addSignal("category", "excellent");
// On the current trace
await ze.sendTraceSignal("conversion", true);
// On the current session
await ze.sendSessionSignal("user_engaged", true);
```
## API endpoint
For direct API calls, send signals to:
```
POST https://api.zeroeval.com/workspaces//signals
```
Auth is the same bearer API key you use for tracing.
### Payload schema
| field | type | required | notes |
| -------------- | ------------------------------ | -------- | ---------------------------------------------- |
| completion\_id | string | β | **OpenAI completion ID** (for LLM completions) |
| span\_id | string | β | **Span ID** (for specific spans) |
| trace\_id | string | β | **Trace ID** (for entire traces) |
| session\_id | string | β | **Session ID** (for user sessions) |
| name | string | β
| e.g. `user_satisfaction` |
| value | string \| bool \| int \| float | β
| your data β see examples below |
You must provide at least one of: `completion_id`, `span_id`, `trace_id`, or
`session_id`.
## Common signal patterns
Below are some quick copy-pasta snippets for the most common cases.
### 1. Binary feedback (π / π)
```python Python SDK
import zeroeval as ze
# On current span
ze.set_signal("thumbs_up", True)
# On specific span
ze.set_signal(span, {"helpful": False})
```
```typescript TypeScript SDK
import * as ze from 'zeroeval';
// Thumbs-up on the current span
await ze.sendSpanSignal("thumbs_up", true);
// Thumbs-down
await ze.sendSpanSignal("thumbs_up", false);
```
```python API
import requests
payload = {
"span_id": span.id,
"name": "thumbs_up",
"value": True // or False
}
requests.post(
f"https://api.zeroeval.com/workspaces/{WORKSPACE_ID}/signals",
json=payload,
headers={"Authorization": f"Bearer {ZE_API_KEY}"}
)
```
### 2. Star rating (1β5)
```python Python SDK
ze.set_signal("star_rating", 4)
```
```typescript TypeScript SDK
import * as ze from 'zeroeval';
// Star rating (1β5) on the current trace
await ze.sendTraceSignal("star_rating", 4);
```
```js JavaScript API
fetch(`https://api.zeroeval.com/workspaces/${WORKSPACE_ID}/signals`, {
method: "POST",
headers: {
Authorization: `Bearer ${ZE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
trace_id: trace.id,
name: "star_rating",
value: 4, // any integer 1β5
}),
});
```
### 3. Continuous metrics
```python Python SDK
# Response time
ze.set_signal("response_time_ms", 1250.5)
# Task completion time
ze.set_signal("time_on_task_sec", 12.85)
# Accuracy score
ze.set_signal("accuracy", 0.94)
```
```typescript TypeScript SDK
import * as ze from 'zeroeval';
// Response time on the current session
await ze.sendSessionSignal("response_time_ms", 1250.5);
// Accuracy score on current span
await ze.sendSpanSignal("accuracy", 0.94);
```
```python API
payload = {
"session_id": session.id,
"name": "time_on_task_sec",
"value": 12.85 // float works too
}
```
### 4. Categorical outcomes
```python Python SDK
ze.set_signal("task_status", "success")
ze.set_signal("error_type", "timeout")
ze.set_signal("user_intent", "purchase")
```
```typescript TypeScript SDK
import * as ze from 'zeroeval';
// Task status on current span
await ze.sendSpanSignal("task_status", "success");
```
```js JavaScript API
{
completion_id: completion.id,
name: "task_status",
value: "success" // could also be "retry" / "fail"
}
```
### 5. Session-level signals
```python Python
# Track user engagement across an entire session
ze.set_session_signal("pages_visited", 5)
ze.set_session_signal("converted", True)
ze.set_session_signal("user_tier", "premium")
```
```typescript TypeScript
import * as ze from 'zeroeval';
// Track user engagement across the session
await ze.sendSessionSignal("pages_visited", 5);
await ze.sendSessionSignal("converted", true);
await ze.sendSessionSignal("user_tier", "premium");
```
### 6. Trace-level signals
```python Python
# Track outcomes for an entire request flow
ze.set_trace_signal("request_successful", True)
ze.set_trace_signal("total_cost", 0.045)
ze.set_trace_signal("model_used", "gpt-4o")
```
```typescript TypeScript
import * as ze from 'zeroeval';
// Trace-level outcomes
await ze.sendTraceSignal("request_successful", true);
await ze.sendTraceSignal("total_cost", 0.045);
await ze.sendTraceSignal("model_used", "gpt-4o");
```
## Signal types
Signals are automatically categorized based on their values:
* **Boolean**: `true`/`false` values β useful for success/failure, yes/no feedback
* **Numerical**: integers and floats β useful for ratings, scores, durations, costs
* **Categorical**: strings β useful for status, categories, error types
## Putting it all together
```python Python
import zeroeval as ze
# Initialize tracing
ze.init(api_key="your-api-key")
# Start a session for user interaction
with ze.trace("user_chat_session", session_name="Customer Support") as session:
# Process user query
with ze.trace("process_query") as span:
response = llm_client.chat.completions.create(...)
# Signal on the LLM completion
ze.set_signal("response_generated", True)
ze.set_signal("response_length", len(response.choices[0].message.content))
# Capture user feedback
user_rating = get_user_feedback() # Your feedback collection logic
# Signal on the session
ze.set_session_signal("user_rating", user_rating)
ze.set_session_signal("issue_resolved", user_rating >= 4)
# Signal on the entire trace
ze.set_trace_signal("interaction_complete", True)
```
```typescript TypeScript (Basic)
import * as ze from 'zeroeval';
ze.init({ apiKey: "your-api-key" });
// Start a session for user interaction
await ze.withSpan({
name: "user_chat_session",
sessionName: "Customer Support",
}, async () => {
// Process user query
await ze.withSpan({ name: "process_query" }, async () => {
const response = await llmClient.chat.completions.create(...);
// Signal on the LLM completion
await ze.sendSpanSignal("response_generated", true);
await ze.sendSpanSignal("response_length", response.choices[0].message.content.length);
});
// Capture user feedback
const userRating = await getUserFeedback();
// Session-level signals
await ze.sendSessionSignal("user_rating", userRating);
await ze.sendSessionSignal("issue_resolved", userRating >= 4);
// Trace-level signal
await ze.sendTraceSignal("interaction_complete", true);
});
```
```typescript TypeScript (Decorators)
import { span } from 'zeroeval';
import * as ze from 'zeroeval';
ze.init({ apiKey: "your-api-key" });
class SupportChat {
@span({
name: "user_chat_session",
sessionName: "Customer Support"
})
async handleChatSession() {
// Process user query
await this.processQuery();
// Capture user feedback
const userRating = await getUserFeedback();
// Session-level signals
await ze.sendSessionSignal("user_rating", userRating);
await ze.sendSessionSignal("issue_resolved", userRating >= 4);
// Trace-level signal
await ze.sendTraceSignal("interaction_complete", true);
}
@span({ name: "process_query" })
async processQuery() {
const response = await llmClient.chat.completions.create(...);
// Signal on the LLM completion
await ze.sendSpanSignal("response_generated", true);
await ze.sendSpanSignal("response_length", response.choices[0].message.content.length);
return response;
}
}
```
That's it! Your signals will appear in the ZeroEval dashboard, helping you understand how your AI system performs in real-world scenarios.
# Tags
Source: https://docs.zeroeval.com/tracing/tagging
Simple ways to attach rich, query-able tags to your traces.
Tags are keyβvalue pairs that can be attached to any **span**, **trace**, or **session**. They power the facet filters in the console so you can slice-and-dice your telemetry by *user*, *plan*, *model*, *tenant*, or anything else that matters to your business.
For complete tagging API documentation, see the [Python SDK Reference](/tracing/sdks/python/reference#tags) or [TypeScript SDK Reference](/tracing/sdks/typescript/reference#tags).
## 1. Tag once, inherit everywhere
When you add a `tags` dictionary to the **first** span you create, every child span automatically gets the same tags. That means you set them once and they flow down the entire call-stack.
```python Python
import zeroeval as ze
@ze.span(
name="handle_request",
tags={
"user_id": "42", # who triggered the request
"tenant": "acme-corp", # multi-tenant identifier
"plan": "enterprise" # commercial plan
}
)
def handle_request():
authenticate()
fetch_data()
process()
# Two nested child spans β they automatically inherit *all* the tags
with ze.span(name="fetch_data"):
...
with ze.span(name="process", tags={"stage": "post"}):
...
```
```typescript TypeScript (Basic)
import * as ze from 'zeroeval';
function handleRequest() {
ze.withSpan({
name: "handle_request",
tags: {
user_id: "42", // who triggered the request
tenant: "acme-corp", // multi-tenant identifier
plan: "enterprise" // commercial plan
}
}, () => {
authenticate();
fetchData();
process();
});
}
// Two nested child spans β they automatically inherit *all* the tags
function fetchData() {
ze.withSpan({ name: "fetch_data" }, () => {
// ...
});
}
function process() {
ze.withSpan({
name: "process",
tags: { stage: "post" }
}, () => {
// ...
});
}
```
```typescript TypeScript (Decorators)
import { span } from 'zeroeval';
class RequestHandler {
@span({
name: "handle_request",
tags: {
user_id: "42", // who triggered the request
tenant: "acme-corp", // multi-tenant identifier
plan: "enterprise" // commercial plan
}
})
handleRequest() {
this.authenticate();
this.fetchData();
this.process();
}
// Two nested child spans β they automatically inherit *all* the tags
@span({ name: "fetch_data" })
fetchData() {
// ...
}
@span({
name: "process",
tags: { stage: "post" }
})
process() {
// ...
}
authenticate() {
// Not traced
}
}
```
## 2. Tag a single span
If you want to tag only a **single** span (or override a tag inherited from a parent) simply provide the `tags` argument on that specific decorator or context manager.
```python Python
import zeroeval as ze
@ze.span(name="top_level")
def top_level():
# Child span with its own tags β *not* inherited by siblings
with ze.span(name="db_call", tags={"table": "customers", "operation": "SELECT"}):
query_database()
# Another child span without tags β it has no knowledge of the db_call tags
with ze.span(name="render"):
render_template()
```
```typescript TypeScript (Basic)
import * as ze from 'zeroeval';
function topLevel() {
ze.withSpan({ name: "top_level" }, () => {
// Child span with its own tags β *not* inherited by siblings
ze.withSpan({
name: "db_call",
tags: { table: "customers", operation: "SELECT" }
}, () => {
queryDatabase();
});
// Another child span without tags β it has no knowledge of the db_call tags
ze.withSpan({ name: "render" }, () => {
renderTemplate();
});
});
}
```
```typescript TypeScript (Decorators)
import { span } from 'zeroeval';
class DataService {
@span({ name: "top_level" })
topLevel() {
this.performDbCall();
this.renderOutput();
}
@span({
name: "db_call",
tags: { table: "customers", operation: "SELECT" }
})
performDbCall() {
// Child span with its own tags β *not* inherited by siblings
queryDatabase();
}
@span({ name: "render" })
renderOutput() {
// Another child span without tags β it has no knowledge of the db_call tags
renderTemplate();
}
}
```
Under the hood these tags live only on that single span, they are **not** copied to siblings or parents.
## 3. Granular tagging (session, trace, or span)
You can add granular tags at the session, trace, or span level after they've been created:
```python Python
import uuid
from langchain_core.messages import HumanMessage
import zeroeval as ze
DEMO_TAGS = {"example": "langgraph_tags_demo", "project": "zeroeval"}
SESSION_ID = str(uuid.uuid4())
SESSION_INFO = {"id": SESSION_ID, "name": "Tags Demo Session"}
with ze.span(
name="demo.root_invoke",
session=SESSION_INFO,
tags={**DEMO_TAGS, "run": "invoke"},
):
# 1οΈβ£ Tag the *current* span only
current_span = ze.get_current_span()
ze.set_tag(current_span, {"phase": "pre-run"})
# 2οΈβ£ Tag the whole trace β root + all children (past *and* future)
current_trace = ze.get_current_trace()
ze.set_tag(current_trace, {"run_mode": "invoke"})
# 3οΈβ£ Tag the entire session
current_session = ze.get_current_session()
ze.set_tag(current_session, {"env": "local"})
result = app.invoke({"messages": [HumanMessage(content="hello")], "count": 0})
```
```typescript TypeScript (Basic)
import { randomUUID } from 'crypto';
import { HumanMessage } from '@langchain/core/messages';
import * as ze from 'zeroeval';
const DEMO_TAGS = { example: "langgraph_tags_demo", project: "zeroeval" };
const SESSION_ID = randomUUID();
const SESSION_INFO = { id: SESSION_ID, name: "Tags Demo Session" };
ze.withSpan({
name: "demo.root_invoke",
sessionId: SESSION_INFO.id,
sessionName: SESSION_INFO.name,
tags: { ...DEMO_TAGS, run: "invoke" }
}, async () => {
// 1οΈβ£ Tag the *current* span only
const currentSpan = ze.getCurrentSpan();
if (currentSpan) {
ze.setTag(currentSpan, { phase: "pre-run" });
}
// 2οΈβ£ Tag the whole trace β root + all children (past *and* future)
const currentTrace = ze.getCurrentTrace();
if (currentTrace) {
ze.setTag(currentTrace, { run_mode: "invoke" });
}
// 3οΈβ£ Tag the entire session
const currentSession = ze.getCurrentSession();
if (currentSession) {
ze.setTag(currentSession, { env: "local" });
}
const result = await app.invoke({
messages: [new HumanMessage("hello")],
count: 0
});
});
```
```typescript TypeScript (Decorators)
import { randomUUID } from 'crypto';
import { HumanMessage } from '@langchain/core/messages';
import { span } from 'zeroeval';
import * as ze from 'zeroeval';
const DEMO_TAGS = { example: "langgraph_tags_demo", project: "zeroeval" };
class TaggingDemo {
private sessionId = randomUUID();
private sessionInfo = {
id: this.sessionId,
name: "Tags Demo Session"
};
@span({
name: "demo.root_invoke",
sessionId: function(this: TaggingDemo) { return this.sessionInfo.id; },
sessionName: function(this: TaggingDemo) { return this.sessionInfo.name; },
tags: { ...DEMO_TAGS, run: "invoke" }
})
async rootInvoke() {
// 1οΈβ£ Tag the *current* span only
const currentSpan = ze.getCurrentSpan();
if (currentSpan) {
ze.setTag(currentSpan, { phase: "pre-run" });
}
// 2οΈβ£ Tag the whole trace β root + all children (past *and* future)
const currentTrace = ze.getCurrentTrace();
if (currentTrace) {
ze.setTag(currentTrace, { run_mode: "invoke" });
}
// 3οΈβ£ Tag the entire session
const currentSession = ze.getCurrentSession();
if (currentSession) {
ze.setTag(currentSession, { env: "local" });
}
const result = await app.invoke({
messages: [new HumanMessage("hello")],
count: 0
});
return result;
}
}
```