Caching

Caching

A cache sits inside the Worker as a decorator. worker.cache(...) returns a new Worker that looks up each crop before calling the engine, records the hit on recognition.cache, emits a cache_hit event, and writes back on miss. Engine adapters never know they are cached, and different Workers in a cascade keep their hits separate.

Caching is opt-in. A Worker built without .cache(...) hits the engine on every region.

The shortest path

Worker.cache(...) accepts a path string, a Cache instance, or None:

from scriva import Worker, Cache

Worker.openai("gpt-4o").cache(".scriva_cache")                  # FileSystemCache
Worker.openai("gpt-4o").cache(Cache.layered(".scriva"))         # exact + semantic
Worker.openai("gpt-4o").cache(Cache.redis("redis://..."))       # shared workers
Worker.openai("gpt-4o")                                         # no cache (default)

When you pass a string or path, scriva creates a FileSystemCache at that location. When you want both tiers, Cache.layered(path) is the canonical default.

Two tiers

Exact cache

Keyed by a deterministic hash of (crop_bytes, worker.name, worker.version, prompt.hash, model). A hit returns the previous Recognition verbatim; bit-identical input always hits.

Cache.fs(".scriva_cache")                  # FileSystemCache shorthand

Cheap, fast, lossless. Always safe.

Semantic cache

Keyed by an embedding of the crop. A hit returns a previous result when the cosine similarity to a cached embedding exceeds threshold (default 0.99 — very high, because OCR is unforgiving and “64” vs “65” cells can look 0.97-similar).

from scriva.embedders import OpenAIEmbedder

Cache.vector(
    embedder=OpenAIEmbedder(model="text-embedding-3-small"),
    threshold=0.99,
    path=".scriva_cache/vec",
)

Pays for itself on documents with repeating elements (column headers, boilerplate, common labels).

Layered cache

Combine the two. Exact is checked first; semantic only on miss:

Cache.layered(
    ".scriva_cache",
    semantic=Cache.vector(embedder=OpenAIEmbedder(...)),
)

Cache.layered(path) with no semantic= is just an exact FileSystemCache — but the form survives later upgrades without changing the call site.

Cache key composition

The cache key is composed from everything that can change the recognition for a given crop:

Component	Why it’s in the key
hash(crop_bytes)	The image is the only input to the engine.
worker.name	Two different Workers should not collide.
worker.version	Bumping the Worker invalidates its slice of the cache.
prompt.hash	Prompt changes change the output — they must change the key.
model / engine cfg	gpt-4o and gpt-4o-mini are different Workers, keyed apart.
language hint	A language hint changes downstream behaviour.

Decorators above .cache(...) in the chain (e.g. .score(...), .retry(...)) do not participate in the key — they wrap the cache, not the engine, so a cache hit short-circuits before they run. Place .cache(...) as early as possible (right after the engine factory) to maximise the hit rate.

Hit provenance

Every hit lands on recognition.cache:

recognition.cache
# CacheProvenance(tier="exact",    similarity=1.0,  key="sha256:…")
# CacheProvenance(tier="semantic", similarity=0.994, key="sha256:…")
# None  -> miss; engine was called

The same provenance appears on the event stream as a recognize/cache_hit event with region_id, tier, and similarity — see architecture.md › Observability.

Where the cache lives in a chain

.cache(...) is a Worker decorator, never an Orchestrator step. Different Workers in a cascade, consensus, or escalate composition keep their hits separate by default:

worker = Worker.cascade(
    Worker.tesseract().cache(".scriva_cache/tess"),
    Worker.openai("gpt-4o-mini").cache(".scriva_cache/4o-mini"),
    Worker.openai("gpt-4o").cache(".scriva_cache/4o"),
)

To share a backend across Workers, pass the same Cache instance:

shared = Cache.layered(".scriva_cache")
a = Worker.openai("gpt-4o").cache(shared)
b = Worker.anthropic("claude-opus-4-7").cache(shared)

The worker.name component of the key keeps their entries from colliding even on the same physical store.

Protocol

class Cache(Engine, Protocol):
    name: str
    async def get(self, key: CacheKey) -> CacheHit | None: ...
    async def put(self, key: CacheKey, recognition: Recognition) -> None: ...

CacheHit carries both the recognition and the provenance (which tier hit, similarity score, original key). The Worker records this on recognition.cache and emits a cache_hit event so a UI can render hit/miss.

Built-in adapters

Factory	Class	Backend	Extra
Cache.fs(path)	FileSystemCache	one JSON per entry	core
Cache.redis(url)	RedisCache	Redis key/value	redis
Cache.vector(embedder=, path=)	SemanticCache	NumPy index + responses	core
Cache.layered(path, semantic=)	LayeredCache	exact-then-semantic	core

Eviction

There is no clever invalidation. The cache key includes the Worker name, version, and prompt hash — bumping any of those is the invalidation. Concretely:

• Prompt change — set a new Prompt or bump its version. Old entries remain on disk but never match.
• Worker upgrade — worker.version is part of the key. New entries miss; old entries are dead weight you can delete at leisure.
• Forced refresh — pass cache_policy="bypass" to a single recipe(doc) call to skip lookups for that run only. Writes still happen, so subsequent runs benefit.

FileSystemCache has no built-in TTL or LRU. Two options:

Cache.fs(".scriva_cache", max_entries=100_000)        # bounded LRU
Cache.fs(".scriva_cache", max_age_days=30)            # rolling TTL

RedisCache honours Redis-level expiry — set ttl_s= on the factory and rely on Redis to evict. SemanticCache rebuilds its NumPy index on load; trimming responses.json and re-running is the manual GC.

Sizing

• Cache.fs(...) stores one small JSON per entry under path/. Cleanup is the caller’s job — most users let it grow forever and accept the inode cost.
• Cache.vector(...) stores embeddings.npy + responses.json. Memory is O(entries × dim × 4 bytes); 100k entries × 1024 dim ≈ 400 MB. Use a pgvector backend for sets larger than that.
• Cache.redis(...) is bounded by Redis. One key per entry, with the recognition JSON-encoded; budget ~1 KB per entry plus overhead.

What not to cache

• Outputs flagged Capability.HANDWRITING. Handwriting Workers tend to be context-sensitive; caching them produces brittle results across documents.
• Anything where the recognised text is itself the cache key. Caches consult the image, not the text.
• Workers wrapped in .score(method="voting") — voting deliberately varies its samples per call; caching defeats it. Cache the children, not the consensus.

Custom backends

Implement the protocol. Two patterns are common:

• Redis exact cache — for shared workers. Five lines on top of redis.asyncio; Cache.redis(...) ships in core.
• pgvector semantic cache — for systems that already run Postgres. Replaces the in-memory NumPy index when the corpus is large.

Both are trivial to write against the protocol.

What to read next

• Worker — the full decorator chain.
• Sample stores — labelled-crop persistence, the sibling protocol to Cache.
• Architecture › Observability — the cache_hit event payload.