Evaluation

scriva.eval runs an Orchestrator recipe (or preset) against a directory of ground-truth annotations and reports how well it did. It is the same tool whether you are tuning a prompt, picking between two Worker engines, or gating a pull request.

import scriva

report = scriva.eval(
    recipe=scriva.presets.invoice.recipe(),
    ground_truth="./annotations/invoices/",
)

print(report.f1, report.calibration_error)
report.to_markdown("eval.md")
report.to_html("eval.html")

Ground-truth format

Annotations live next to the source files:

annotations/invoices/
├── acme-2026-04.pdf
├── acme-2026-04.json        # the ground truth
├── acme-2026-05.pdf
├── acme-2026-05.json
└── manifest.yaml             # optional, see below

A ground-truth JSON is the same shape as result.to_dict() — same field names, same nesting — so you can use the output of a known-good recipe as the starting point:

{
  "schema": "scriva.schemas.Invoice",
  "fields": {
    "invoice_number": "AC-2026-04",
    "invoice_date": "2026-04-12",
    "total_amount": "1240.00",
    "currency": "USD",
    "line_items": [
      {"description": "Widget A", "quantity": 5, "unit_price": "20.00", "total": "100.00"}
    ]
  }
}

For region-level evaluation (was the bbox right, not just the text):

{
  "schema": "scriva.schemas.Invoice",
  "fields": { ... },
  "regions": [
    {"id": "p0-total", "bbox": [820, 1180, 140, 32], "text": "1240.00"}
  ]
}

When the source is multi-page, paginate the JSON the same way the recipe does — pages: [{...}, {...}].

A manifest.yaml in the directory lets you describe per-source overrides:

defaults:
  schema: scriva.schemas.Invoice
items:
  - source: acme-2026-04.pdf
    ground_truth: acme-2026-04.json
    weight: 1.0
  - source: acme-2026-05.pdf
    ground_truth: acme-2026-05.json
    weight: 2.0                  # double-weight critical samples

When there’s no manifest, scriva.eval pairs *.json with the matching source by stem.

Metrics

class EvalReport:
    n: int                                       # total samples
    precision: float
    recall: float
    f1: float
    exact_match: float                           # whole-instance equality
    field_metrics: dict[str, FieldMetric]        # per-field breakdown
    confusion: dict[str, dict[str, int]]         # per-field, predicted -> actual
    calibration_error: float                     # expected calibration error
    mean_latency_s: float
    total_cost_usd: float
    by_source: dict[str, SourceReport]           # per-input drill-down

FieldMetric per field:

class FieldMetric:
    precision: float
    recall: float
    f1: float
    exact_match: float
    near_match: float                # field-typed similarity (Levenshtein for str, abs diff for Decimal, etc.)
    n_present_actual: int
    n_present_predicted: int
    confidence_quantiles: tuple[float, float, float]   # 25/50/75

near_match uses field-typed comparators — see Custom comparators.

Custom comparators

The default field comparators:

Pydantic type	Comparator
`str`	normalized exact + Levenshtein (>0.9 == near match)
`Decimal`, `float`	abs diff < `1e-2`, relative diff < 0.5%
`int`	exact
`date`, `datetime`	exact, normalised to ISO
`Enum`	exact
`BaseModel`	recursive
`list[T]`	bipartite-matched by element comparator

Override per field via the comparators= kwarg:

def loose_str(actual, predicted) -> float:
    return 1.0 if actual.lower().strip() == predicted.lower().strip() else 0.0

report = scriva.eval(
    recipe=...,
    ground_truth="./annotations/",
    comparators={
        "vendor.name": loose_str,
        "total_amount": lambda a, p: 1.0 if abs(a - p) < Decimal("0.05") else 0.0,
    },
)

Calibration

report.calibration_error                 # scalar (ECE)
report.calibration_curve                 # list[(confidence_bin, accuracy)]
report.to_html("eval.html")              # includes a reliability diagram

A well-calibrated Worker reports confidence=0.9 on regions that turn out to be correct 90% of the time. Wide divergence between confidence and accuracy means the threshold you use in low_confidence(...) is meaningless — either retrain (when you control the scorer) or switch .score(method="self_check") to .score(method="rendering"), which is calibrated end-to-end.

Comparing two recipes

left  = scriva.presets.invoice.recipe(model="gpt-4o")
right = scriva.presets.invoice.recipe(model="claude-opus-4-7")

compare = scriva.eval.compare(
    left=left,
    right=right,
    ground_truth="./annotations/",
)
compare.delta_f1                         # right.f1 - left.f1
compare.regressions                      # sources where right is worse than left
compare.improvements                     # vice versa
compare.to_markdown("compare.md")

The Markdown output is suitable to paste into a PR description — fields flipped from wrong → right and right → wrong are tabulated separately.

CI gates

The CLI counterpart is scriva eval (see CLI › scriva eval):

scriva eval --preset invoice --ground-truth ./annotations/ \
  --min-f1 0.90 --min-precision 0.95

Exits non-zero when any threshold fails — wire it into a GitHub Actions job as the gate for changes to your prompt registry, Worker configuration, or fine-tuned head.

Programmatic gate:

report = scriva.eval(
    recipe=scriva.presets.invoice.recipe(),
    ground_truth="./annotations/invoices/",
)
assert report.f1 >= 0.92, report.to_markdown()

Regression fixtures

For deterministic regression tests in unit-test suites:

def test_invoice_regression(snapshot):
    recipe = scriva.presets.invoice.recipe(cache=".test_cache")
    result = recipe("fixtures/acme.pdf")
    assert result.to_dict() == snapshot

Pair with a cached Worker (Worker.openai(...).cache(Cache.layered(...))) so the test does not hit the network. The first run records the cache and the snapshot; every subsequent run is hermetic.

Evaluation

Evaluation

Ground-truth format

Metrics

Custom comparators

Calibration

Comparing two recipes

CI gates

Regression fixtures

What to read next