Reconstruction

Postprocessors

A postprocess step refines structure or text after — and around — the Worker has run. Two phases of the Orchestrator fall under this banner:

classify — .classify(...) adapters that tag each Region with a role and a kind so the recognise step can dispatch.
reconstruct — .reconstruct.* adapters that take the recognitions and assemble them back into a structured output (tables, documents, key/value pairs) — including text-level fixers like dictionaries and rule-line splitting.

Confidence scoring is no longer a postprocess concern: it moved to the Worker as .score(method=...). See Worker for the full reference.

.split.<method>()  ──►  .classify(...)  ──►  .recognize(worker)  ──►  .reconstruct.*  ──►  .export.*

Classify — assign roles and kinds

.classify(...) assigns region.role ("header", "data", "blank", "merged") and region.kind ("text", "number", "date", "checkbox", "handwriting", …). The recognise step dispatches on these.

Cardinality is 0..1. If omitted, every region keeps the role and kind the splitter assigned (typically role="data", kind="text").

Built-in classify adapters

Method	Marks	How
`.classify(blank=True, merged=True)`	`blank`, `merged`	Adaptive threshold + ink-density + border presence
`.classify.rule_based(...)`	`blank`, `merged`	Rule-based with tunable thresholds
`.classify.ml(model=...)`	`blank`, `merged`	Image-embedding + trained sklearn/LightGBM head
`.classify.hybrid(rule=..., learned=...)`	both	Rule-based first; falls back to ML when low-confidence
`.classify(roles={(0, 0): "header"})`	`role` (static)	Hard-coded overrides — grid coords or region IDs
`.classify(kinds=callable_or_dict)`	`kind`	Static dict, callable, or a custom classify step

`.classify.rule_based(...)`

The cheap default. Pure NumPy — no model dependency.

.classify.rule_based(
    blank_density=0.002,        # ink fraction below this → blank
    merged_overlap=0.6,         # bbox overlap with neighbour above
)

`.classify.ml(...)`

Engine-agnostic: pass any ImageEmbedder and any trained model that exposes predict_proba. Two embedders ship — OpenAIEmbedder and AzureVisionEmbedder — both behind optional extras.

.classify.ml(model="scriva-cell-classifier-v1")
.classify.ml(model="models/cells.joblib")

Training an ML classifier

scriva.classify.ml.train(...) fits a head from a SampleStore and returns a ready-to-use classifier you can drop into the recipe:

from scriva import SampleStore
from scriva.classify import ml, LightGBMHead
from scriva.embedders import OpenAIEmbedder

store = SampleStore.fs(".scriva_samples")

clf = ml.train(
    samples=store,
    where=lambda s: s.attrs.get("annotated") is True,
    embedder=OpenAIEmbedder(model="text-embedding-3-small"),
    head=LightGBMHead(),                              # ClassifierHead protocol
    target=lambda s: s.attrs.get("role", "data"),
)
clf.save("models/cells.joblib")

recipe = (
    Orchestrator()
    .split.grid()
    .classify.ml(model="models/cells.joblib")
    .recognize(worker)
)

The head is a ClassifierHead protocol — anything with fit(X, y) / predict_proba(X) / save(path) / load(path). LightGBMHead, SklearnHead, and TorchMlpHead ship in core; bring your own for exotic shapes.

Training reads samples lazily — large stores are streamed, not materialised.

`.classify.hybrid(...)`

.classify.hybrid(
    rule=rule_based(blank_density=0.002),
    learned=ml.load("models/cells.joblib"),
)

Rule-based first; falls back to the learned classifier when the rule output is low-confidence.

Writing your own classify adapter

Subclass:

from scriva import Step, Context, Capability

class MyClassifier(Step):
    phase = "classify"
    name  = "my-classifier"
    capabilities = frozenset({Capability.BLANK_DETECTION})

    async def run(self, ctx: Context) -> Context:
        for region in ctx.layout.regions:
            region.role = ...
            region.kind = ...
        return ctx

…or decorate a function:

from scriva import step

@step(phase="classify", name="my-classifier")
async def my_classifier(ctx):
    for region in ctx.layout.regions:
        region.role = ...
        region.kind = ...
    return ctx

Reconstruct — regions to structure (and text fixers)

.reconstruct.* assembles the recognitions back into a structured output. Without this phase, the DocumentResult carries the raw regions in reading order.

Cardinality is 0..1 for the structural reconstruct call, but the text-level fixers (dictionary, rule_splitter, whitespace, blank_suppress, …) can be chained as many times as you like — the Orchestrator slots them into the reconstruct phase in the order written, before the structural step.

Structural reconstruct adapters

Method	What it produces
`.reconstruct()`	Default: by layout shape (grid → table, polygons → doc).
`.reconstruct.grid()`	Table with merged cells preserved.
`.reconstruct.document()`	Reading-order document with sections / paragraphs.
`.reconstruct.table(...)`	Table with explicit row/column headers.
`.reconstruct.kv(...)`	Key/value pairs (form-style).

The reconstructed structure is what .export.* writes and what result.to_excel(...) / result.to_dict(...) read.

Text-level reconstruct adapters

These read the recognitions and mutate or replace the text. They are pure with respect to the page: they read page for context (cropping, rendering), but they only mutate recognitions.

Method	What it does
`.reconstruct.dictionary(...)`	Fix known OCR errors using a supervised dictionary + fuzzy matching
`.reconstruct.vocabulary(...)`	Snap low-confidence outputs onto a domain vocabulary
`.reconstruct.rule_splitter(...)`	Detect internal ruled lines (`│`, `─`) in output and split the region
`.reconstruct.language_detector()`	Set `Recognition.language` from text
`.reconstruct.whitespace()`	Collapse whitespace, normalise newlines, strip trailing dashes
`.reconstruct.blank_suppress()`	Demote noise outputs (`"None"`, `"-"`, `"N/A"`) to `text=None`
`.reconstruct.filter(predicate)`	Drop recognitions for which `predicate(recognition) is False`

`.reconstruct.dictionary(...)`

A dictionary is the workhorse of the user-environment accuracy loop: you keep a YAML (or a SampleStore) of project-specific corrections, and every run pulls it in.

Two dictionary types:

Supervised — {wrong: correct} pairs. Four passes: exact, substring (longest-first greedy), full-string fuzzy (difflib, threshold 0.7), token-level fuzzy.
Unsupervised — a vocabulary of known-good terms. Used for validation only; does not rewrite.

.reconstruct.dictionary.from_yaml("corrections.yaml")
.reconstruct.dictionary.from_pairs({"ENCL0SURE": "ENCLOSURE"})
.reconstruct.dictionary.from_csv("vendors.csv", from_col="raw", to_col="canonical")
.reconstruct.dictionary(supervised=..., unsupervised=..., fuzzy_threshold=0.7)

`.reconstruct.dictionary.from_samples(...)` — derive a dictionary from your labelled store

When you maintain a SampleStore of corrections (every time a human edits a recognised cell, the corrected label is saved alongside the original recognition.text), scriva can build a supervised dictionary from it automatically:

from scriva import SampleStore

store = SampleStore.fs(".scriva_samples")

.reconstruct.dictionary.from_samples(
    store,
    where=None,                                       # filter samples
    min_observations=2,                                # only adopt pairs seen >= N times
    fuzzy_threshold=0.7,
    refresh="startup",                                 # "startup" | "every_run" | seconds
)

A pair (wrong → right) is added whenever a sample’s recognition.text differs from its human label. The min_observations floor protects against one-off typos in the labels.

This is the same store that RecognitionHint.from_store(...) reads for few-shot examples — see samples.md › Two roads from a SampleStore. A labelled store improves accuracy on the input side (few-shot exemplars guide the Worker) and on the output side (corrections rewrite known errors). Build the store once; both paths benefit.

`.reconstruct.rule_splitter(...)`

When a recognised cell contains │ or ─, the cell was actually a sub-grid the splitter missed. .rule_splitter():

Parses the text into a sub-grid.
Expands the original Layout to add the sub-cells.
Assigns the sub-strings to the new regions.

Requires the Worker to be layout-aware (Capability.LAYOUT_PRESERVING).

Chaining

The Orchestrator runs reconstruct adapters in the order added. Order matters — clean up first, drop noise, then fix, then restructure, then assemble:

recipe = (
    Orchestrator()
    .split.grid()
    .classify(blank=True, merged=True)
    .recognize(worker)
    .reconstruct.whitespace()
    .reconstruct.blank_suppress()
    .reconstruct.dictionary.from_yaml("corrections.yaml")
    .reconstruct.rule_splitter()
    .reconstruct.grid()                       # structural step last
    .export.xlsx("out.xlsx")
)

Confidence scoring (moved to Worker)

The old postprocess.confidence_score.* family is gone. Confidence scoring now lives on the Worker via .score(method=...) — see Worker for "rendering", "logprob", "self_check", and "voting". The Worker is the right place for it because each crop’s confidence is a function of that crop and that Worker, not of the page or document.

Writing your own reconstruct adapter

Subclass:

from scriva import Step, Context

class StripQuotes(Step):
    phase = "reconstruct"
    name  = "strip-quotes"

    async def run(self, ctx: Context) -> Context:
        for rid, r in ctx.recognitions.items():
            if r.text:
                ctx.recognitions[rid] = r.with_text(r.text.strip('"').strip("'"))
        return ctx

…or decorate a function:

from scriva import step

@step(phase="reconstruct", name="strip-quotes")
async def strip_quotes(ctx):
    for rid, r in ctx.recognitions.items():
        if r.text:
            ctx.recognitions[rid] = r.with_text(r.text.strip('"').strip("'"))
    return ctx

recipe = recipe.then(strip_quotes)

Recognition is immutable; use .with_text(), .with_confidence(), or .replace(**kwargs) to derive new ones. This makes reconstruct chains safe to test and to re-run on the same input.

When not to use a reconstruct adapter

If your refinement requires another VLM call, that is a second Worker, not a reconstruct adapter. Use Worker.cascade(...), Worker.escalate(...), or a custom Worker that wraps two engines. Reconstruct adapters are for transforms on already-recognised text and structure.