Datagrunt 5.3.1: In-Process Fast Path, Multimodal VLM Wire & Non-Destructive Consensus

Datagrunt 5.3.1 is now available! This patch release brings significant enhancements to PDF extraction speed and accuracy, multimodal extraction pipelines, and schema-driven consensus.

Key Highlights in Datagrunt 5.3.1

1. Option A In-Process Zero-Loss Fast Path — Pure-Python zero-loss fast paths replace PDF compute microkernels (pdf_char_bgr_medians, pdf_union_area, pdf_char_reason_runs, pdf_snap_and_merge_segments, pdf_lzw_decode), eliminating PyO3 boundary overhead while preserving bit-for-bit exactness.
2. 3-Witness Multimodal VLM Band Wire (#473) — ModelRequest.images now carries up to 64 PNG band images (up to 32 MiB combined) to vision-capable models, with support for render_pages="vision_only".
3. Out-of-Prompt OCR Witness Grounding — Grounding verifies multimodal LLM extractions against out-of-prompt OCR witness elements, ensuring citation precision without crowding text prompt envelopes.
4. Non-Destructive Consensus & Folding (#482) — Adds the Informational marker for non-disputing descriptive fields, Key(normalizer="alnum_upper") for punctuation- and case-insensitive identity matching, child key matching, unkeyed pairing, and Anthropic schema_mode support.
5. Scan ASCII-Mojibake Lexical Shortfall & 4-Way Orientation Voting — Detects garbled ASCII-Mojibake text over scanned pages (>50% cover) to route pages to vision or flag image-only reasons, and introduces conditional 4-way orientation voting (orientation_fallback) when OSD confidence is low.

1. In-Process Zero-Loss PDF Fast Path

Datagrunt 5.3.1 transitions PDF compute routines — background median sampling, sweep-line box union area calculations, character reason run grouping, table lattice segment snapping, and LZW stream decoding — to optimized, in-process zero-loss fast paths.

By eliminating cross-language FFI calls and PyO3 marshalling overhead, PDF extraction performance is accelerated while maintaining 100% bit-for-bit output fidelity.

from datagrunt import PDFReader

# In-process fast path runs automatically during extraction
reader = PDFReader("document.pdf")
doc = reader.document

for page in doc.pages:
    print(f"Page {page.page_number}: {page.route} route, {len(page.elements)} elements")

2. Multimodal VLM Band Wire & Out-of-Prompt OCR Grounding (#473)

For visually complex documents — such as schematics, financial forms, and complex multi-column layouts — text alone is often insufficient. Datagrunt 5.3.1 expands ModelRequest with a 3-witness multimodal wire:

• Image Attachment (ModelRequest.images): Supports up to 64 PNG image bands (up to 32 MiB total per request).
• Vision-Only Page Mode (render_pages="vision_only"): Configures page rendering specifically for multimodal vision prompts.
• Out-of-Prompt OCR Grounding: Grounding checks verify model extraction against out-of-prompt OCR witness elements on the source page without forcing large OCR text blobs into the prompt envelope.

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Key, PDFReader, rationalize

class DiagramCallout(BaseModel):
    label: Annotated[str, Key]
    description: str

reader = PDFReader("schematic.pdf", render_pages="vision_only")
doc = reader.document

result = rationalize(
    doc,
    schema=DiagramCallout,
    prompt="Extract all component labels and descriptions from the diagram.",
    models=("anthropic:claude-sonnet-4-6", "google:gemini-2.5-flash"),
    min_families=2,
)

print(f"Extracted {len(result.records)} callouts with multimodal consensus")

3. Non-Destructive Consensus & Informational Marker (#482)

Consensus in rationalize() has been enhanced with non-destructive entity merging and flexible marker attributes:

The Informational Marker

Use Informational on descriptive fields (e.g., Annotated[str | None, Informational]). Free-text wording differences in Informational fields will not cause consensus disputes or discard otherwise agreed records.

Normalized Keys (Key(normalizer="alnum_upper"))

Key fields can now specify normalizer="alnum_upper". During consensus grouping and cross-page entity folding, ASCII punctuation and spaces are stripped and characters are upper-cased for key matching, while raw verbatim text is preserved for quote grounding verification.

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, rationalize

class HardwarePart(BaseModel):
    # Matches 'PN-100-A', 'pn 100 a', and 'PN100A' as the same entity
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    # Wording variations in notes won't drop an agreed part record
    notes: Annotated[str | None, Informational] = None

result = rationalize(
    doc,
    schema=HardwarePart,
    models=("openai:gpt-5.4", "anthropic:claude-sonnet-4-6"),
    min_families=2,
)

4. Scan ASCII-Mojibake Shortfall & 4-Way Orientation Voting

Scan ASCII-Mojibake Lexical Shortfall Detection

When processing scanned documents (>50% raster scan cover), OCR text layers sometimes yield garbled ASCII-Mojibake noise rather than valid words. Datagrunt 5.3.1 detects when non-space characters fall below a 30% lexical ratio or carry high intraword symbol noise. Such pages are flagged with a garbled_text shortfall and automatically routed to vision or marked as image_only under ocr="never".

Conditional 4-Way Orientation Voting (orientation_fallback)

When Tesseract’s initial Orientation and Script Detection (OSD) is weak or OCR word confidence is under 70%, setting orientation_fallback=True evaluates all four right-angle rotations (0°, 90°, 180°, 270°) using confidence-weighted word scoring to select the true page orientation.

from datagrunt import PDFReader

# Enable 4-way orientation fallback for misoriented scanned PDFs
reader = PDFReader("scanned_form.pdf", ocr="auto", orientation_fallback=True)
doc = reader.document

Summary of Changes

Datagrunt 5.3.1 is available on PyPI. Upgrade today:

uv pip install --upgrade "datagrunt[pdf,rationalize]"