Datagrunt 5.1: Schema-Driven PDF Extraction with Consensus & Grounding

Datagrunt 5.1.0 introduces schema-driven PDF rationalization (rationalize()). While Datagrunt 5.0 established a fast, single-engine PDFium document parser, 5.1 provides a robust, production-grade layer to extract strongly-typed Pydantic records from unformatted or semi-structured PDF documents.

Extracting structured data from enterprise PDFs with large language models often suffers from hallucinations, missing records, or ungrounded values. rationalize() addresses these challenges directly through cross-family model consensus, verbatim quote grounding, cross-page entity folding, and deterministic Polars/Arrow exports.

Quick Start

Install the [rationalize] extra alongside [pdf]:

uv pip install "datagrunt[pdf,rationalize]"

Define your target Pydantic schema using the lightweight Key annotation, and call rationalize() over any PDFDocument or PDF path:

from decimal import Decimal
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Key, PDFReader, rationalize


class LineItem(BaseModel):
    part_no: Annotated[str, Key]
    qty: int
    unit_price: Decimal
    total: Decimal


class Invoice(BaseModel):
    invoice_id: Annotated[str, Key("Invoice #")]
    vendor_name: str
    invoice_date: str
    items: list[LineItem] = []


# Load PDF document
doc = PDFReader("invoices/purchase_order.pdf").document

# Rationalize against multi-provider models
result = rationalize(
    doc,
    schema=Invoice,
    prompt="Extract invoice header details and all listed line items.",
    models=("openai:gpt-5.4", "anthropic:claude-sonnet-4-6"),
    min_families=2,          # Require agreement across >= 2 distinct model families
    cache_dir=".dg-cache",   # Atomic on-disk response cache
)

# Export consensus records directly to a Polars DataFrame
df = result.to_dataframe(flatten="items")

# Review quarantine or disputed items
if result.needs_review:
    print(f"Items needing manual review: {len(result.needs_review)}")

Core Capabilities of Datagrunt 5.1

1. Zero-Dependency Schema Definition (Key)

The Key marker identifies entity key fields on Pydantic models. Importantly, Key is imported directly from datagrunt without requiring pydantic or httpx to be installed:

from typing import Annotated
from datagrunt import Key

# Pure marker annotation without heavy dependencies
customer_id: Annotated[str, Key("Customer ID / Account Number")]

You can pass optional label aliases to Key(...) to provide provider models with hints on how the entity identifier appears in source documents.

2. Cross-Family Model Consensus (min_families)

Relying on a single AI provider model for data extraction exposes pipelines to silent errors. Datagrunt 5.1 routes prompts across multiple LLM families and verifies their results against each other:

• Supported Endpoints: openai:, anthropic:, gemini:, ollama:, and openai-compatible:<model>@<url>.
• Google Cloud Endpoints: Native authentication via GoogleCloudAuth with GeminiProvider.google_cloud(...), AnthropicProvider.google_cloud(...), and OpenAICompatibleProvider.google_cloud(...).
• Consensus Threshold: By default, min_families=2 requires that a strict majority of models from at least two separate provider families agree on every field before a record reaches "consensus".
• Field Unanimity: Pass must_agree=("vendor_name", "invoice_id") to require 100% unanimity on critical header fields.

3. Verbatim Quote and Key Grounding

Every extracted record must prove its source:

• Pre-Filtering: Elements flagged with hidden_reasons (invisible text, off-page elements, background matches) are stripped before building wire requests, protecting models against hidden prompt injection or noisy layers.
• Wire Escaping: Bidi controls, invisible tags, and ASCII control characters are safely sanitized in wire envelopes.
• Citation Verification: Supporting models cite source_element_ids and provide grounding_quotes. Datagrunt verifies quotes against visible page text (exact substring matches on digital/hybrid pages, token containment $\ge 0.85$ on vision/OCR pages).
• Key Grounding: Root entity Key values are matched against visible cited elements.

Records that fail grounding or consensus are placed in result.needs_review or result.quarantined rather than silently dirtying downstream tables.

4. Cross-Page Entity Folding & Deterministic Exports

PDF tables and entity records often split across page breaks. rationalize() automatically folds records across page boundaries based on root entity keys, stitching partial lists and attributes together.

RationalizedResult provides deterministic exports:

# Polars DataFrame with exploded child lists
df = result.to_dataframe(flatten="items")

# Arrow Table export
arrow_table = result.to_arrow_table()

# Python dictionary list
dicts = result.to_dicts()

5. Atomic Response Cache (cache_dir)

Prompt tuning and pipeline debugging can be expensive. By setting cache_dir=".dg-cache", Datagrunt computes canonical SHA-256 hashes over schema definitions and PDF document contents. Provider responses are cached atomically on disk, eliminating duplicate API calls during local testing.

Summary of Updated & Created Files

• README.md & pyproject.toml — Version bumped to 5.1.0, added [rationalize] extra.
• src/datagrunt/rationalization/ — Package implementation for schema markers, provider adapters, consensus engine, quote grounding, entity folding, and caching.
• datagrunt-site/content/docs/_index.md — Updated stable release notice to v5.1.0 and added [rationalize] installation & quickstart.
• datagrunt-site/content/docs/guides/pdf-parsing.md — Added dedicated section on rationalize() schema-driven extraction.
• datagrunt-site/content/blog/2026/november/datagrunt-5-1-0-schema-driven-pdf-rationalization.md — Published release blog post.

Upgrading

Upgrade Datagrunt and install the new rationalization extra:

uv pip install --upgrade "datagrunt[pdf,rationalize]"