Skip to content

Cookbook: Rebuilding ocr-agent

ocr-agent is the production system scriva was extracted from: a browser OCR tool for Japanese tabular forms, agentic schema discovery, P&ID diagrams, and a Bedrock-driven annotation loop. This page is the ocr-agent → scriva translation, end-to-end. Every snippet is copy-pasteable into a fresh project.

The shape of the page mirrors ocr-agent’s own surface:

  1. Normal OCR — grid split → cell OCR → dictionary → rule-splitter → confidence → Excel.
  2. Agentic OCR — discover fields, OCR rows, extract values into a key-value sheet.
  3. P&ID OCR — find symbols by template, OCR each box, emit a structured catalogue.
  4. Annotation loop — primary + oracle Workers, sample store, classifier training.
  5. Wrapping it in a FastAPI app — SSE progress, cancellation, human-in-the-loop pause/resume.

If you only read one section, read §1 — every other workflow is a rearrangement of the same Worker + Orchestrator surface.

These examples assume one application package on top of scriva:

ocr_app/
├── recipes/
│ ├── forms.py # §1
│ ├── agentic.py # §2
│ ├── pid.py # §3
│ └── annotation.py # §4
├── steps/ # custom @step / @worker helpers
│ ├── blank_merges.py
│ └── refusal_strip.py
├── prompts/
│ └── jp.py # locale-specific Prompt registrations
└── api/ # §5 — FastAPI app, routers, templates

Everything in recipes/, steps/, and prompts/ is pure scriva configuration. Only api/ knows about HTTP, sessions, or job persistence — that boundary is the same boundary architecture.md draws.

Bring the extras you actually use:

Terminal window
pip install "scriva[openai,bedrock,excel,pgvector,pdf]"

…and configure the credentials each adapter needs through environment variables (OPENAI_API_KEY, AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, DATABASE_URL).

This is the workflow /api/ocr/upload runs in ocr-agent. Split a grid, classify blank + merged cells, OCR each non-blank crop, correct against a dictionary, split internal ruled lines, score confidence, write Excel.

ocr_app/recipes/forms.py
from pathlib import Path
from scriva import Orchestrator, Worker
from scriva.cache import Cache
from scriva.prompts import Prompt
from scriva.reconstruct import dictionary, rule_splitter
# ocr-agent-specific: a blank "merged" region is almost always a
# detection mistake — split it back into individual blank cells.
# See notes/excluded-from-docs.md §4 for why this stays in the app layer.
from ocr_app.steps.blank_merges import dissolve_blank_merges
def tabular_forms_recipe(
*,
dict_yaml: Path | None = None,
blank_density: float = 0.003,
excel_out: Path = Path("out.xlsx"),
json_out: Path | None = None,
crop_bbox: tuple[int, int, int, int] | None = None,
) -> Orchestrator:
"""The ocr-agent 'normal OCR' recipe.
`blank_density` corresponds to ocr-agent's strict/standard/loose
presets (0.001 / 0.003 / 0.008). `crop_bbox` is what the
Cropper.js selection becomes after the user confirms the crop
screen.
"""
# Worker: GPT-4o, Japanese prompt, layered cache, rendering score.
worker = (
Worker.openai("gpt-4o")
.prompt(Prompt.ocr(locale="ja"))
.cache(Cache.layered(".scriva_cache"))
.score(method="rendering") # cairosvg + Azure Vision embedder
.parallel(max=8)
)
recipe = Orchestrator()
# Preprocess: optional fixed crop, then orient + deskew.
if crop_bbox is not None:
recipe = recipe.crop(box=crop_bbox)
recipe = recipe.orient().deskew()
# Split: morphological grid with a Hough fallback (mirrors grid_detect.py).
recipe = recipe.split.grid(method="morphological", fallback="hough")
# Classify: rule-based first, learned head for the in-between cases
# (mirrors `_detect_blank_cells_hybrid`).
recipe = recipe.classify.hybrid(
rule=classify.rule_based(blank_density=blank_density),
learned=classify.embedding.load("models/cells.joblib"), # falls back if missing
blank=True,
merged=True,
)
# Domain rule: dissolve all-blank merges back into singletons.
recipe = recipe.then(dissolve_blank_merges)
# Recognize: route by kind so numeric cells use the typed Worker.
recipe = recipe.recognize.by_kind({
"text": worker,
"number": Worker.number().using(worker),
"date": Worker.date().using(worker),
"checkbox": Worker.checkbox(),
"blank": Worker.skip(),
})
# Reconstruct: whitespace → drop noise → dictionary → split ruled lines.
# Order matters — see orchestrator.md.
recipe = (
recipe
.reconstruct.whitespace()
.reconstruct.blank_suppress()
)
if dict_yaml is not None:
recipe = recipe.reconstruct.dictionary.from_yaml(dict_yaml)
recipe = recipe.reconstruct.rule_splitter()
recipe = recipe.reconstruct.grid()
# Export: Excel (confidence colouring + legend) + optional JSON.
recipe = recipe.export.xlsx(
excel_out,
font="MS Gothic",
confidence_thresholds=(0.6, 0.8),
legend_sheet=True,
)
if json_out is not None:
recipe = recipe.export.json(json_out)
return recipe.options(page_concurrency=1)
recipe = tabular_forms_recipe(
dict_yaml=Path("dict.yaml"),
blank_density=0.003,
excel_out=Path("outputs/jobs/abc123.xlsx"),
)
result = recipe("scan.png")
print(f"{len(result.regions_with_text())} cells, "
f"mean confidence {result.mean_confidence:.2%}")

That run wrote outputs/jobs/abc123.xlsx. For the confidence-coloured download that ocr-agent’s UI offers as a second button, swap the export step for a per-cell callback:

def colour(rec):
if rec.confidence is None or rec.confidence >= 0.8: return None
if rec.confidence >= 0.6: return "yellow"
return "red"
recipe = recipe.replace(
"export.xlsx",
export.xlsx(
Path("outputs/jobs/abc123-coloured.xlsx"),
confidence_fill=colour,
legend_sheet=True,
font="MS Gothic",
),
)
result = recipe("scan.png")

The “dissolve all-blank merges” rule runs against the layout, not the recognitions, so it slots in after classify and before recognize. A Step subclass is the right shape:

ocr_app/steps/blank_merges.py
from scriva import Step, Context, MergeInfo
class DissolveBlankMerges(Step):
phase = "classify"
name = "dissolve-blank-merges"
async def run(self, ctx: Context) -> Context:
regions = ctx.layout.regions
new_regions = []
for r in regions:
if r.merge_group_id and r.role == "blank":
# Strip the merge — leave the region as a standalone blank cell.
r = r.model_copy(update={
"merge_group_id": None,
"merge": MergeInfo(),
})
new_regions.append(r)
ctx.layout.regions = new_regions
return ctx
dissolve_blank_merges = DissolveBlankMerges()

The other thing ocr-agent’s postprocess_vlm_text does that the built-in reconstruct.blank_suppress() does not cover is stripping VLM refusal boilerplate (「申し訳ありません … 読み取れません」). That is a reconstruct-phase step:

ocr_app/steps/refusal_strip.py
import re
from scriva import step
_REFUSAL = re.compile(
r"(申し訳ありません|I'm sorry|I apologize|I cannot).*?"
r"(?:読み取|確認|判読|認識|識別|read|unable).*?(?:できません|ません|cannot)[。..]?\s*",
re.IGNORECASE,
)
@step(phase="reconstruct", name="strip-vlm-refusal")
async def strip_refusal(ctx):
for rid, r in ctx.recognitions.items():
if r.text:
ctx.recognitions[rid] = r.with_text(_REFUSAL.sub("", r.text).strip())
return ctx

Drop it into the chain right after reconstruct.blank_suppress():

recipe = recipe.insert_after("reconstruct.blank_suppress", strip_refusal)

Confidence-driven re-OCR (a second recipe)

Section titled “Confidence-driven re-OCR (a second recipe)”

ocr-agent’s /api/ocr/{job}/reocr re-runs only the low-confidence cells through a different model, using the previous text as a hint. The canonical recipe lives in orchestrator.md › Confidence-driven re-OCR; here is the ocr-agent adaptation:

from scriva import Orchestrator, Worker, RecognitionHint
from scriva.cache import Cache
from scriva.prompts import Prompt
def reocr_recipe(
result: "scriva.DocumentResult",
*,
threshold: float = 0.6,
model: str = "gpt-4o",
) -> Orchestrator:
engine = Worker.openai if model.startswith(("gpt-", "o1", "o3", "o4")) else Worker.anthropic
worker = (
engine(model)
.prompt(Prompt.ocr_with_hint(locale="ja"))
.cache(Cache.layered(".scriva_cache"))
.score(method="rendering")
)
return (
Orchestrator()
.split.from_layout(result, where=lambda r: (r.confidence or 0) < threshold)
.recognize(worker.with_hints(RecognitionHint.from_result(result)))
.reconstruct.grid()
.export.xlsx(Path("outputs/jobs/abc123-refined.xlsx"), legend_sheet=True)
)
refined = reocr_recipe(result, threshold=0.6, model="gpt-4o")("scan.png")
combined = result.merge(refined, strategy="highest_confidence")
combined.to_excel("outputs/jobs/abc123-final.xlsx")

split.from_layout(result, where=...) re-uses the previous run’s regions instead of re-detecting; RecognitionHint.from_result(result) seeds the new Worker with the previous text as a per-region hint.

ocr-agent’s agentic mode: the Worker discovers what fields exist, OCRs the form row-by-row, then extracts each field’s value with the row context in scope. The output is a key-value sheet (項目名 / 抽出 値 / 参照行 / 備考), not a grid.

The whole thing collapses to the scriva.domains.agentic.extract factory:

ocr_app/recipes/agentic.py
from pathlib import Path
from scriva import domains, Worker
from scriva.cache import Cache
def agentic_recipe(*, excel_out: Path):
return domains.agentic.extract(
schema=None, # None = discover (3 passes by default)
rounds=3,
worker=Worker.openai("gpt-4o").cache(Cache.layered(".scriva_cache")),
json_out=excel_out.with_suffix(".json"),
xlsx_out=excel_out,
)

When you do know the schema up front — say you have a pydantic.BaseModel for the form — pass it as schema= and skip the discovery phase:

from pydantic import BaseModel
class InspectionForm(BaseModel):
patient_name: str
date_of_birth: str
diagnosis_code: str
dose_mg: float | None
recipe = recipes.agentic.extract(
schema=InspectionForm,
worker=Worker.openai("gpt-4o"),
json_out=Path("outputs/jobs/abc123.json"),
)

ocr-agent’s discovery prompt is bilingual and includes the already-discovered list each round. To match that exactly, register your own prompt and hand it to the discovery Worker:

from scriva.prompts import Prompt
Prompt.register(
"discover-ja",
"""この帳票画像を分析し、抽出すべきデータ項目(フィールド名)を列挙してください。
{% if known %}
【既に発見済みの項目】
{% for item in known %}- {{ item }}
{% endfor %}
上記以外にまだ発見されていない項目があれば、それらも含めて網羅的に列挙してください。
{% endif %}
各行は「- 項目名」の形式で出力。項目名のみ、値は不要です。""",
locale="ja",
)
recipe = recipes.agentic.extract(
rounds=3,
discover_worker=Worker.openai("gpt-4o").prompt(Prompt.from_registered("discover-ja")),
worker=Worker.openai("gpt-4o"),
json_out=Path("outputs/jobs/abc123.json"),
)

DocumentResult.to_excel() writes a flat key-value table when the layout has no grid — exactly the shape ocr-agent’s _export_agentic_excel produces. The recipe wires this in via .export.xlsx(...); for ad-hoc serialisation:

result = recipe("scan.png")
result.to_excel(
"outputs/jobs/abc123.xlsx",
font="MS Gothic",
columns=["項目名", "抽出値", "参照行", "備考"], # column header overrides
)

Two routes match ocr-agent’s behaviour. Pick by how much of the workflow the user drives interactively.

3a. User drags patterns; scriva finds matches

Section titled “3a. User drags patterns; scriva finds matches”

ocr-agent’s UI lets the user drag-select a symbol, then runs cv2.matchTemplate to find similar instances. A custom splitter wraps that:

ocr_app/recipes/pid.py
from pathlib import Path
import cv2
import numpy as np
from pydantic import BaseModel
from scriva import Orchestrator, Worker, Layout, Region, BBox, Capability, splitter
from scriva.prompts import Prompt
class PidSymbol(BaseModel):
types: str # valve, pump, instrument, tank, …
text: str # equipment IDs, numbers, labels
@splitter(name="pid-template-match", capabilities=frozenset({Capability.POLYGON}))
async def pid_template_match(page, *, templates: list[Path], threshold=0.8,
nms_overlap=0.2, min_distance=10):
"""Run cv2.matchTemplate for each pattern, NMS, min-distance filter."""
image = page.image_as_ndarray() # PIL or ndarray; both supported
boxes: list[tuple[int, int, int, int, float, str]] = []
for tpl_path in templates:
tpl = cv2.imread(str(tpl_path))
score = cv2.matchTemplate(image, tpl, cv2.TM_CCOEFF_NORMED)
ys, xs = np.where(score >= threshold)
for x, y in zip(xs, ys):
boxes.append((int(x), int(y), tpl.shape[1], tpl.shape[0],
float(score[y, x]), tpl_path.stem))
# NMS + min-distance: see template_matching.py for the reference impl.
boxes = _nms(boxes, overlap=nms_overlap)
boxes = _min_distance(boxes, distance=min_distance)
regions = [
Region(bbox=BBox(x, y, w, h), role="data",
attrs={"pattern": pat, "match_score": s})
for x, y, w, h, s, pat in boxes
]
return Layout.from_regions(regions, page=page)
def pid_recipe(templates: list[Path], *, json_out: Path,
xlsx_out: Path | None = None) -> Orchestrator:
worker = (
Worker.openai("gpt-4o")
.prompt(Prompt.structured(schema=PidSymbol))
.parallel(max=3)
)
recipe = (
Orchestrator()
# 10% crop padding ≈ ocr-agent's _crop_box_image
.split(pid_template_match(templates=templates, threshold=0.8,
nms_overlap=0.2, min_distance=10))
.split.refine.pad(percent=10)
.recognize(worker)
.reconstruct.whitespace()
.export.json(json_out)
)
if xlsx_out is not None:
# embed_crops=True + crop_height_px=60 matches ocr-agent's
# pid_export.py (one row per box with a 60px thumbnail).
recipe = recipe.export.xlsx(
xlsx_out, embed_crops=True, crop_height_px=60, crop_column="A",
)
return recipe

(_nms and _min_distance are pure NumPy; lift them verbatim from ocr-agent’s pipeline/template_matching.py.)

recipe = pid_recipe(
templates=[Path("uploads/pid_presets/valve.png"),
Path("uploads/pid_presets/pump.png")],
json_out=Path("outputs/pid/session-001.json"),
xlsx_out=Path("outputs/pid/session-001.xlsx"),
)
result = recipe("drawing.png")

3b. Whole-page Worker (no template matching)

Section titled “3b. Whole-page Worker (no template matching)”

The opinionated scriva.domains.pid.diagram(...) factory drops template matching entirely and asks the VLM to find symbols directly. Right for when you do not have templates yet or the drawing style varies:

from scriva import domains
recipe = domains.pid.diagram(
worker=Worker.openai("gpt-4o"),
json_out=Path("outputs/pid/session-002.json"),
)
result = recipe("drawing.png")

Both shapes are legitimate; ocr-agent’s UI happens to use 3a because of the drag-to-select interaction.

ocr-agent’s annotation backend (annotation_runner.py) does three distinct things — blank annotation, merge/border annotation, and OCR annotation — but all three follow the same pattern: primary classifier or Worker runs first, results sorted by |score - 0.5|, Bedrock Qwen-VL re-runs the uncertain ones, every result becomes a training sample.

Worker.escalate(...) + .export.samples(...) + classify.embedding.train(...) cover it. The pre-built recipe:

ocr_app/recipes/annotation.py
from scriva import Orchestrator, Worker, samples
from scriva.cache import Cache
def annotation_recipe(*, dsn: str, json_out: str) -> Orchestrator:
store = samples.layered(
fs=samples.fs("training_data/samples"),
index=samples.pgvector(dsn, dim=1024),
)
primary = (
Worker.openai("gpt-4o")
.cache(Cache.layered(".scriva_cache"))
.score(method="rendering")
)
oracle = Worker.bedrock("qwen.qwen3-vl-235b-a22b")
worker = Worker.escalate(
primary=primary,
oracle=oracle,
when=lambda r: (r.confidence or 0) < 0.5, # top-uncertain go to the oracle
)
return (
Orchestrator()
.split.grid()
.classify.rule_based()
.recognize(worker)
.reconstruct.whitespace()
.export.json(json_out)
.export.samples(store, include_oracle=True) # writes primary + oracle on every sample
)
recipe = annotation_recipe(
dsn="postgresql+asyncpg://localhost/ocr_agent",
json_out="outputs/annotations/2026-05-21.json",
)
recipe("scan.png") # writes samples to the pgvector store as it runs

The chain composes:

split.grid
──► classify.rule_based
──► recognize(Worker.escalate(primary, oracle, when=…))
──► reconstruct.whitespace
──► (confidence is already on each Recognition via primary.score(...))
──► export.json + export.samples(store, include_oracle=True)

— so Worker.escalate records both the primary recognition and the oracle’s correction on every sample it touches, and .export.samples(...) persists both. That replaces AnnotationRunner._register_sample (annotation_runner.py:88) and AnnotationRunner._register_pair_sample (annotation_runner.py:263) both.

Once samples accumulate, fit a head and drop it into your forms recipe:

from scriva import classify, samples
from scriva.embedders import OpenAIEmbedder
from scriva.classify import LightGBMHead
store = samples.layered(
fs=samples.fs("training_data/samples"),
index=samples.pgvector(dsn, dim=1024),
)
clf = classify.embedding.train(
samples=store,
where=lambda s: s.label is not None, # only labelled samples
embedder=OpenAIEmbedder(model="text-embedding-3-small"),
head=LightGBMHead(),
target=lambda s: "blank" if s.attrs.get("is_blank") else "data",
)
clf.save("models/cells.joblib")

…and back in §1, the call you already had — classify.embedding.load("models/cells.joblib") — picks it up next run. That is the entire ocr-agent active-learning loop on three named primitives.

ocr-agent’s merge annotation operates on pairs of adjacent cells, not single cells. The mapping is the same Worker.escalate wrapper, but with a pair splitter instead of a grid splitter and a different prompt:

from scriva.prompts import Prompt
from pydantic import BaseModel
Prompt.register(
"border-pair",
"""この画像は帳票の隣接する2つのセルを合わせた領域です。
2つのセルの間に境界線(罫線)が存在するかどうかを判定してください。
以下のJSON形式で返答してください:
{ "has_border": true|false, "direction": "horizontal"|"vertical" }""",
locale="ja",
)
class BorderLabel(BaseModel):
has_border: bool
direction: str
# Adapter that wraps a trained classifier as a Worker so .escalate can sort by confidence.
classify_worker = classify.as_worker(classify.embedding.load("models/borders.joblib"))
border_recipe = (
Orchestrator()
.split(pair_splitter()) # custom (lift _extract_pair_region)
.recognize(Worker.escalate(
primary=classify_worker,
oracle=Worker.bedrock("qwen.qwen3-vl-235b-a22b")
.prompt(Prompt.structured_few_shot(
schema=BorderLabel,
base="border-pair",
)),
when=lambda r: abs((r.confidence or 0.5) - 0.5) < 0.2,
))
.export.samples(store, include_oracle=True)
)

The pair_splitter is a 30-line custom @splitter that yields one Region per adjacent cell pair (the body is _extract_pair_region from cell_detect.py:110). classify.as_worker(...) is the adapter that turns a trained head into a Worker whose confidence is the head’s predict_proba — its existence is what lets Worker.escalate sort pairs by |confidence - 0.5|.

scriva’s architecture page is firm about this: the web layer is the application’s job. But the integration is small — events.to_sse, recipe.cancel(), and recipe.split_at_review() are the three primitives — so here is the canonical wiring that replaces ocr-agent’s routers/ocr.py.

ocr_app/api/jobs.py
import asyncio
import uuid
from pathlib import Path
from fastapi import APIRouter, BackgroundTasks, UploadFile
from sse_starlette.sse import EventSourceResponse
import scriva
from scriva.events import to_sse
from ocr_app.recipes.forms import tabular_forms_recipe
router = APIRouter(prefix="/api/ocr", tags=["ocr"])
# Application-layer registry (architecture.md disclaims this from scriva).
_recipes: dict[uuid.UUID, scriva.Orchestrator] = {}
_sources: dict[uuid.UUID, Path] = {}
_results: dict[uuid.UUID, asyncio.Task] = {}
@router.post("/upload")
async def upload(file: UploadFile, bg: BackgroundTasks) -> dict:
job_id = uuid.uuid4()
path = await _save_upload(file, job_id)
recipe = tabular_forms_recipe(
dict_yaml=Path("dict.yaml"),
excel_out=Path(f"outputs/jobs/{job_id}.xlsx"),
)
_recipes[job_id] = recipe
_sources[job_id] = path
_results[job_id] = asyncio.create_task(recipe.aio(str(path)))
return {"job_id": str(job_id)}
@router.get("/{job_id}/progress")
async def progress(job_id: uuid.UUID):
recipe = _recipes[job_id]
source = _sources[job_id]
return EventSourceResponse(to_sse(recipe.events(str(source))))
@router.post("/{job_id}/cancel")
async def cancel(job_id: uuid.UUID) -> dict:
recipe = _recipes.get(job_id)
if recipe is None:
return {"cancelled": False}
recipe.cancel() # cooperative; steps exit at their next yield
return {"cancelled": True}

The to_sse helper emits text/event-stream chunks shaped exactly the way ocr-agent’s existing UI already consumes. The event schema (stage, kind, payload) is documented in architecture.md › Observability and is stable across releases.

ocr-agent’s interactive mode runs detection, pauses on awaiting_review, waits for the user to edit the cell map, then resumes with PUT /api/ocr/{job}/cells. The scriva-native version uses .review.hitl(...) and recipe.split_at_review():

from scriva import Orchestrator, Worker
from scriva.cache import Cache
from scriva.prompts import Prompt
worker = (
Worker.openai("gpt-4o")
.prompt(Prompt.ocr(locale="ja"))
.cache(Cache.layered(".scriva_cache"))
.score(method="rendering")
)
recipe = (
Orchestrator()
.orient().deskew()
.split.grid(method="morphological", fallback="hough")
.classify.hybrid(
rule=classify.rule_based(),
learned=classify.embedding.load("models/cells.joblib"),
blank=True, merged=True,
)
.then(dissolve_blank_merges)
.review.hitl(sidecar=f"outputs/jobs/{job_id}/layout.json") # pauses here
.recognize(worker)
.reconstruct.whitespace()
.reconstruct.blank_suppress()
.then(strip_refusal)
.reconstruct.dictionary.from_yaml("dict.yaml")
.reconstruct.rule_splitter()
.reconstruct.grid()
.export.xlsx(f"outputs/jobs/{job_id}.xlsx", legend_sheet=True)
)
# Split the recipe at the review step so the web handler can return
# between phases.
phase1, phase2 = recipe.split_at_review()
# Phase 1 — runs detect/classify, writes the layout sidecar, returns.
phase1(str(path))
# Respond to the browser; status = "awaiting_review"
# Phase 2 — when the user PUTs the edited layout back, run from review on.
phase2(str(path))

The sidecar’s schema is the same as the regions field of result.to_dict(), so the UI can edit it in place and round-trip losslessly. See Orchestrator › Review for the queue-based variant.

These pieces of ocr-agent are not in any of the scriva snippets above and are not meant to be. They live in ocr_app/api/:

  • FastAPI app, routers, Jinja2 templates, static assets.
  • HMAC session cookies, basic-auth middleware, role checks.
  • OCRJob SQL model, job lifecycle states, the /jobs page.
  • Cropper.js UI, cell-editor UI, P&ID drag-to-select.
  • Dictionary CRUD UI, training-data CRUD UI, CSV import/export endpoints.
  • P&ID PIDSession + pattern-preset library bookkeeping.

scriva’s surface for them is exactly three calls: recipe.cancel() for stop, recipe.events(doc)to_sse(...) for progress, and recipe.split_at_review() for HITL. Anything beyond that is your application.