Cookbook: Handwritten forms

Cookbook: handwritten notes

Handwriting is where the Worker-side choice matters most. Most VLMs do well on print mixed with cursive; classical OCR engines do badly. Worker.handwriting() picks accordingly.

The recipe

from scriva import Orchestrator, Worker

worker = (
    Worker.handwriting()
    .using(Worker.anthropic("claude-opus-4-7"))   # strongest on IAM-Handwriting
    .cache(".scriva_cache")
    .score(method="rendering")
)

recipe = (
    Orchestrator()
    .orient()
    .deskew()
    .split.boxes(detector="block-segment")        # paragraph / list / heading blocks
    .classify()
    .recognize(worker)
    .reconstruct.document()
    .export.json("notes.json")
)

result = recipe("whiteboard.jpg")
notes = result.to_dict()["fields"]
print(notes["title"])
for block in notes["blocks"]:
    print(block["text"], "—", block["confidence"])

Worker.handwriting() is a specialised kind factory — pre-tuned prompt, scorer, and post-processing. .using(Worker.anthropic(...)) swaps the underlying engine without changing the kind-specific machinery.

What you get back

class Notes(BaseModel):
    title: str | None
    captured_on: date | None              # from a date written on the page, if visible
    blocks: list[Block]
    diagrams: list[Diagram]                # rough sketches detected as non-text

class Block(BaseModel):
    text: str
    block_type: Literal["paragraph", "list_item", "heading", "label", "table_row"]
    page_index: int
    bbox: BBox
    confidence: float | None
    contains_strikethrough: bool
    contains_underline: bool

Block.contains_strikethrough is a heuristic — the Worker is asked to flag struck-out text. Useful for meeting minutes where decisions get crossed out mid-stream.

Customising

Print vs cursive routing

Mixed inputs (a printed form filled in by hand) benefit from running the two kinds separately. Classify by kind, then dispatch:

recipe = recipe.replace(
    "recognize",
    Worker.router({
        "handwriting": Worker.handwriting().using(Worker.anthropic("claude-opus-4-7")),
        "text":        Worker.openai("gpt-4o"),
    }),
)

region.kind is set by .classify(...); the router dispatches per-region so handwriting Workers see only the handwritten cells and print Workers see only the printed ones. Where a region is unsure, add the cheap fallback:

recipe = recipe.replace(
    "recognize",
    Worker.router({
        "handwriting": Worker.handwriting(),
        "text":        Worker.openai("gpt-4o"),
    }, default=Worker.cascade(
        Worker.handwriting(),
        Worker.openai("gpt-4o"),
        accept=lambda r: (r.confidence or 0) >= 0.7,
    )),
)

Per-line confidence

result = recipe("notes.jpg")
for block in result.to_dict()["fields"]["blocks"]:
    if (block["confidence"] or 0) < 0.7:
        flag_for_review(block)

Handwriting confidence is harder to calibrate than printed text — run Evaluation › Calibration on a sample of your own corpus before picking a threshold.

Whiteboard photos

Whiteboards need their own preprocessing:

from scriva.preprocess import whiteboard_clean

recipe = recipe.insert_after(
    "orient",
    whiteboard_clean(saturation_drop=0.8, contrast_boost=1.4),
)

whiteboard_clean is roughly the algorithm Office Lens uses — pulls ink off the background and normalises the lighting.

Diagrams as out-of-band attachments

Sketches and diagrams are not text. The reconstruct step records them so the UI can show them alongside the transcribed text:

for diag in notes["diagrams"]:
    save_crop(source="notes.jpg", bbox=diag["bbox"], path=f"out/diag-{diag['id']}.png")

Diagram records the box plus a one-line VLM-generated caption; the crop is the only real artifact.

Production notes

Handwriting is the right place to keep a sample store

The Worker is fallible enough that an oracle re-pass on the most uncertain blocks is worthwhile — and every result is a candidate training sample. Use Worker.escalate(...) + .export.samples(...):

from scriva import samples

store = samples.layered(
    fs=samples.fs(".scriva_handwriting_samples"),
    index=samples.pgvector("postgresql://localhost/scriva", dim=1024),
)

worker = Worker.escalate(
    primary=Worker.handwriting().using(Worker.openai("gpt-4o"))
                                .score(method="rendering")
                                .few_shot(store),
    oracle=Worker.handwriting().using(Worker.bedrock("qwen.qwen3-vl-235b-a22b")),
    when=lambda r: (r.confidence or 0) < 0.7,
)

recipe = recipe.replace("recognize", worker)
recipe = recipe.export.samples(store)

Over time the store grows; train a small head on it and re-OCR uncertain blocks against the local head — see Cookbook › ocr-agent §4.

Cost matters less; latency matters more

Notes are usually a foreground user workflow (“upload meeting photo → see transcript”). Optimise for first-paint latency:

async for page_result in recipe.stream("notes.jpg"):
    update_ui(page_result.fields.blocks)             # incremental render

Streaming returns blocks as they finish. See Orchestrator › Running.

Multi-language handwriting

Worker.handwriting() auto-detects locale via the underlying engine. To pin (e.g. when you know the class is Japanese handwriting):

worker = Worker.handwriting().with_language("ja")
recipe = recipe.replace("recognize", worker)

Or set a recipe-wide default:

recipe = recipe.options(default_language="ja")