Cookbook: Receipts

Cookbook: receipts

Receipts are the canonical “mobile photo OCR” target — long, thin, rarely flat, often crumpled, sometimes shot in bad light. The recipe sets the right preprocess steps so you do not have to do it twice.

The recipe

from scriva import Orchestrator, Worker
from scriva.prompts import Prompt

worker = (
    Worker.openai("gpt-4o")
    .prompt(Prompt.receipt)
    .cache(".scriva_cache")
    .score(method="rendering")
)

recipe = (
    Orchestrator()
    .orient()
    .deskew()
    .denoise()
    .normalise(target="contrast")
    .split.whole_page()
    .recognize(worker)
    .reconstruct.kv()
    .export.json("receipt.json")
)

result = recipe("receipt.jpg")

.orient().deskew().denoise().normalise() is the canonical phone-photo preprocess sequence; .split.whole_page() is right for receipts because the recognised structure comes from the VLM, not from a grid.

What you get back

The standard receipt record:

class Receipt(BaseModel):
    merchant_name: str
    merchant_address: Address | None
    transaction_date: datetime
    currency: str
    subtotal: Decimal | None
    tax_amount: Decimal | None
    tip_amount: Decimal | None
    total_amount: Decimal
    payment_method: Literal["cash", "card", "mobile", "other"] | None
    last4: str | None
    line_items: list[LineItem]

result.to_dict()["fields"] reads exactly this shape.

Customising

Phone-photo preprocess

The recipe above has the right defaults. When an input fails, swap specific steps:

recipe = recipe.replace("deskew", deskew(method="hough", max_angle_deg=20))
recipe = recipe.replace("denoise", denoise(strength="aggressive"))

See Preprocessors for the full catalogue.

Glare and shadows

For chronic glare (e.g. fluorescent lighting), add a shadow-aware normaliser before deskew:

recipe = recipe.insert_before("deskew", shadow_remove(method="retinex"))

shadow_remove is a pure OpenCV step — no model dependency. Use it sparingly; for already-flat scans it can wash out faint text.

Long receipts (multi-photo stitching)

When the user has photographed a long receipt in two or three overlapping shots, stitch first, then run the recipe:

from scriva.preprocess import stitch

doc = stitch.images(["receipt-top.jpg", "receipt-bot.jpg"], overlap_hint="vertical")
result = recipe(doc)

stitch.images uses ORB feature matching to align the photos. The returned Document behaves exactly like a single-image input.

Specialised Workers per field

When you want stricter parsing on amounts and dates than a general VLM gives you, split the recognise step:

recipe = recipe.replace(
    "recognize",
    Worker.router({
        "number": Worker.number(),
        "date":   Worker.date(),
        "address": Worker.address(),
        "text":   worker,
    }),
)

The router uses region.kind set by .classify(...); for whole-page receipts the classifier sees the VLM’s structured output and labels fields accordingly.

Production notes

Confidence calibration matters more here than elsewhere

Phone photos vary so much that a single global threshold rarely works. Run Evaluation on a sample of real submissions and use the per-field calibration to pick thresholds per field:

report = scriva.eval(recipe, "./labelled/")
report.calibration_curve         # per-field auto-accept cutoffs

Tip vs total ambiguity

US restaurant receipts often print the pre-tip total and a hand-written tip. If tip_amount and total_amount disagree by the tip, prefer the line-item math:

from decimal import Decimal

receipt = result.to_dict()["fields"]
if receipt.tip_amount and receipt.total_amount and receipt.subtotal:
    expected = receipt.subtotal + (receipt.tax_amount or Decimal(0)) + receipt.tip_amount
    if abs(expected - receipt.total_amount) > Decimal("0.05"):
        receipt.total_amount = expected

For systematic handling, encode this as a @step(phase="reconstruct") (see Invoices › Tax sanity-check for the pattern).

Batch from a phone-camera upload directory

import scriva

scriva.watch(
    folder="./uploads",
    to="./extracted",
    recipe=recipe,
    pattern="*.jpg",
    move_on_success=True,
    move_on_error="./failed",
)

This is the right shape for an expense-report ingest endpoint that writes its uploads to a local folder.

Mobile cost notes

gpt-4o against full-resolution phone photos is the dominant cost. Two cheap wins:

from scriva.preprocess import resize

recipe = recipe.insert_after("orient", resize(max_dim=1600))   # 1600px long edge
recipe = recipe.replace("recognize", Worker.openai("gpt-4o-mini").cache(".scriva_cache"))

Cuts cost ~5×; accuracy drops <2% on the public receipts benchmark. See Reliability › Cost caps for budgets.