ZeroEval derives prompt optimization suggestions directly from feedback on your production traces. By capturing preferences and correctness signals, we provide concrete prompt edits you can test and use for your agents.
Submitting Feedback
Feedback is the foundation of prompt optimization. You can submit feedback for completions through the ZeroEval dashboard, the Python SDK, or the public API. Feedback helps ZeroEval understand what good and bad outputs look like for your specific use case.
Feedback through the dashboard
The easiest way to provide feedback is through the ZeroEval dashboard. Navigate to your task’s “Suggestions” tab, review incoming completions, and provide thumbs up/down feedback with optional reasons and expected outputs.
Feedback through the SDK
For programmatic feedback submission, use the Python or TypeScript SDK. This is useful when you have automated evaluation systems or want to collect feedback from your application in production.
import zeroeval as ze
ze.init()
# Send feedback for a specific completion
ze.send_feedback(
prompt_slug="support-bot",
completion_id="550e8400-e29b-41d4-a716-446655440000",
thumbs_up=False,
reason="Response was too verbose",
expected_output="A concise 2-3 sentence response"
)
Parameters
| Python | TypeScript | Type | Required | Description |
|---|
prompt_slug | promptSlug | str/string | Yes | The slug/name of your prompt (same as used in ze.prompt()) |
completion_id | completionId | str/string | Yes | The UUID of the completion to provide feedback on |
thumbs_up | thumbsUp | bool/boolean | Yes | True/true for positive, False/false for negative |
reason | reason | str/string | No | Optional explanation of why you gave this feedback |
expected_output | expectedOutput | str/string | No | Optional description of what the expected output should be |
metadata | metadata | dict/object | No | Optional additional metadata to attach to the feedback |
The completion_id is automatically tracked when you use ze.prompt() with automatic tracing enabled. You can access it from the OpenAI response object’s id field, or retrieve it from your traces in the dashboard.
Complete example with feedback
import zeroeval as ze
from openai import OpenAI
ze.init()
client = OpenAI()
# Define your prompt - ZeroEval will automatically use the latest optimized
# version from your dashboard if one exists, falling back to this content
system_prompt = ze.prompt(
name="support-bot",
content="You are a helpful customer support agent."
)
# Make a completion
response = client.chat.completions.create(
model="gpt-4",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": "How do I reset my password?"}
]
)
# Get the completion ID and text
completion_id = response.id
completion_text = response.choices[0].message.content
# Evaluate the response (manually or automatically)
is_good_response = evaluate_response(completion_text)
# Send feedback based on evaluation
ze.send_feedback(
prompt_slug="support-bot",
completion_id=completion_id,
thumbs_up=is_good_response,
reason="Clear step-by-step instructions" if is_good_response else "Missing link to reset page",
expected_output=None if is_good_response else "Should include direct link: https://app.example.com/reset"
)
Auto-optimization: When you use ze.prompt() with content, ZeroEval automatically fetches the latest optimized version from your dashboard if one exists. Your content serves as a fallback for initial setup. This means your prompts improve automatically as you tune them, without any code changes.If you need to test the hardcoded content specifically (e.g., for debugging or A/B testing), use from_="explicit" (Python) or from: "explicit" (TypeScript):# Bypass auto-optimization and always use this exact content
prompt = ze.prompt(
name="support-bot",
from_="explicit",
content="You are a helpful customer support agent."
)
Feedback through the API
For integration with non-Python systems or direct API access, you can submit feedback using the public HTTP API.
Endpoint
POST /v1/prompts/{prompt_slug}/completions/{completion_id}/feedback
Authentication
Requires API key authentication via the Authorization header:
Authorization: Bearer YOUR_API_KEY
Request body
{
"thumbs_up": false,
"reason": "Response was inaccurate",
"expected_output": "The correct answer should mention X, Y, and Z",
"metadata": {
"evaluated_by": "automated_system",
"evaluation_score": 0.45
}
}
Response
{
"id": "fb123e45-67f8-90ab-cdef-1234567890ab",
"completion_id": "550e8400-e29b-41d4-a716-446655440000",
"prompt_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"prompt_version_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
"project_id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
"thumbs_up": false,
"reason": "Response was inaccurate",
"expected_output": "The correct answer should mention X, Y, and Z",
"metadata": {
"evaluated_by": "automated_system",
"evaluation_score": 0.45
},
"created_by": "user_id",
"created_at": "2025-11-22T10:30:00Z",
"updated_at": "2025-11-22T10:30:00Z"
}
Example with cURL
curl -X POST https://api.zeroeval.com/v1/prompts/support-bot/completions/550e8400-e29b-41d4-a716-446655440000/feedback \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"thumbs_up": false,
"reason": "Response was too vague",
"expected_output": "Should provide specific steps",
"metadata": {
"user_satisfaction": "low"
}
}'
If feedback already exists for the same completion from the same user, it will be updated with the new values. This allows you to correct or refine feedback as needed.
Prompt optimizations from feedback
Once you’ve given a good amount of feedback on the incoming traffic for a given task, you can generate prompt optimizations using that feedback by clicking on the “Optimize Prompt” button in the “Suggestions” tab for the task.
Once you’ve generated a new prompt, you can test it with various models and see how it performs against the feedback you’ve already given.
