> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroeval.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Calibration

> Correct judge evaluations to improve accuracy over time

Judges get better the more you correct them. Each time you mark an evaluation as right or wrong, that correction is stored and used to refine future scoring. This is calibration.

## Calibrating in the dashboard

For each evaluated item in the console, you can mark the judge's assessment as correct or incorrect and optionally provide the expected answer.

<img src="https://mintcdn.com/zeroeval/E0iRjqjhHa4EiARK/images/calibrated-judge-2.png?fit=max&auto=format&n=E0iRjqjhHa4EiARK&q=85&s=c164c43cba26cd22cd96acb729222f58" alt="Calibrating your judge" width="1050" height="316" data-path="images/calibrated-judge-2.png" />

<video src="https://mintcdn.com/zeroeval/rv6tdNcftISY5Oep/videos/calibrated-judge.mp4?fit=max&auto=format&n=rv6tdNcftISY5Oep&q=85&s=2f4b12d4459bef8d53cfa2bae5c5a468" controls muted playsInline loop preload="metadata" data-path="videos/calibrated-judge.mp4" />

## Calibrating programmatically

Submit corrections via the SDK or REST API. This is useful for bulk calibration from automated pipelines, custom review workflows, or external labeling tools.

### Finding the right IDs

Judge evaluations involve two related spans:

| ID                     | Description                                        |
| ---------------------- | -------------------------------------------------- |
| **Source Span ID**     | The original LLM call that was evaluated           |
| **Judge Call Span ID** | The span created when the judge ran its evaluation |

| ID            | Where to Find It                                                  |
| ------------- | ----------------------------------------------------------------- |
| **Task Slug** | In the judge settings, or the URL when editing the judge's prompt |
| **Span ID**   | In the evaluation modal, or via `get_judge_evaluations()`         |
| **Judge ID**  | In the URL when viewing a judge (`/judges/{judge_id}`)            |

<Tip>
  The easiest way to get the correct IDs: open a judge evaluation in the
  dashboard, expand "SDK Integration", and click "Copy" to get pre-filled code.
</Tip>

### Binary judges

Mark a judge evaluation as correct or incorrect:

<CodeGroup>
  ```python Python theme={null}
  import zeroeval as ze

  ze.send_feedback(
      prompt_slug="your-judge-task-slug",
      completion_id="span-id-here",
      thumbs_up=True,
      reason="Judge correctly identified the issue",
      judge_id="automation-id-here",
  )
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.zeroeval.com/v1/prompts/{task_slug}/completions/{span_id}/feedback" \
    -H "Authorization: Bearer $ZEROEVAL_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "thumbs_up": true,
      "reason": "Judge correctly identified the issue",
      "judge_id": "automation-uuid-here"
    }'
  ```
</CodeGroup>

### Scored judges

For judges using scored rubrics, provide the expected score and direction:

<CodeGroup>
  ```python Python theme={null}
  ze.send_feedback(
      prompt_slug="quality-scorer",
      completion_id="span-id-here",
      thumbs_up=False,
      judge_id="automation-id-here",
      expected_score=3.5,
      score_direction="too_high",
      reason="Score should have been lower due to grammar issues",
  )
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.zeroeval.com/v1/prompts/{task_slug}/completions/{span_id}/feedback" \
    -H "Authorization: Bearer $ZEROEVAL_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "thumbs_up": false,
      "judge_id": "automation-uuid-here",
      "expected_score": 3.5,
      "score_direction": "too_high",
      "reason": "Score should have been lower"
    }'
  ```
</CodeGroup>

### Per-criterion feedback

For scored judges with multiple criteria, correct individual criterion scores:

<CodeGroup>
  ```python Python theme={null}
  ze.send_feedback(
      prompt_slug="quality-scorer",
      completion_id="span-id-here",
      thumbs_up=False,
      judge_id="automation-id-here",
      reason="Criterion-level score adjustments",
      criteria_feedback={
          "CTA_text": {
              "expected_score": 4.0,
              "reason": "CTA is clear and prominent"
          },
          "CX-004": {
              "expected_score": 1.0,
              "reason": "Required phone number is missing"
          }
      }
  )
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.zeroeval.com/v1/prompts/{task_slug}/completions/{span_id}/feedback" \
    -H "Authorization: Bearer $ZEROEVAL_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "thumbs_up": false,
      "judge_id": "automation-uuid-here",
      "criteria_feedback": {
        "CTA_text": {"expected_score": 4.0, "reason": "CTA is clear and visible"},
        "CX-004": {"expected_score": 1.0, "reason": "Phone number is missing"}
      }
    }'
  ```
</CodeGroup>

To discover valid criterion keys before sending per-criterion feedback:

```python theme={null}
criteria = ze.get_judge_criteria(
    project_id="your-project-id",
    judge_id="automation-id-here",
)

for c in criteria["criteria"]:
    print(c["key"], c.get("description"))
```

### Parameters

| Parameter           | Type    | Required | Description                                      |
| ------------------- | ------- | -------- | ------------------------------------------------ |
| `prompt_slug`       | `str`   | Yes      | Task slug associated with the judge              |
| `completion_id`     | `str`   | Yes      | Span ID being evaluated                          |
| `thumbs_up`         | `bool`  | Yes      | `True` if judge was correct, `False` if wrong    |
| `reason`            | `str`   | No       | Explanation of the correction                    |
| `judge_id`          | `str`   | Yes      | Judge automation ID                              |
| `expected_score`    | `float` | No       | Expected score (scored judges only)              |
| `score_direction`   | `str`   | No       | `"too_high"` or `"too_low"` (scored judges only) |
| `criteria_feedback` | `dict`  | No       | Per-criterion corrections (scored judges only)   |

## Bulk calibration

Iterate through evaluations and submit corrections programmatically:

```python theme={null}
evaluations = ze.get_judge_evaluations(
    project_id="your-project-id",
    judge_id="your-judge-id",
    limit=100,
)

for eval in evaluations["evaluations"]:
    is_correct = your_review_logic(eval)

    ze.send_feedback(
        prompt_slug="your-judge-task-slug",
        completion_id=eval["span_id"],
        thumbs_up=is_correct,
        reason="Automated review",
        judge_id="your-judge-id",
    )
```