Turning Messy PDFs Into Clean Tables Without the Glue Code: Introducing Datagrunt v5 (5.4.1)

November 20, 2026by Martin Graham

How solving a real customer’s PDF headache led to a simple, reliable Python toolkit for batch pipelines, event-driven services, and AI agents.


Almost every data team hits the same wall eventually. Your dashboards, warehouses, and models run smoothly on CSVs, Parquet files, and database tables, until someone drops a folder of 5,000 PDFs on your desk and asks: “Can we get this into our data warehouse?”

Invoices, vendor contracts, product datasheets, inspection reports, scanned manuals: the most valuable operational data in a company is often trapped inside PDFs.

At first, the fix looks easy. Grab a PDF library, send the text to an AI model, ask for JSON, and load the result into a table. On five clean files, it works great.

Then real life shows up:

  • Half the PDFs are scanned images, or worse, scans with garbled hidden text underneath.
  • A dense 30-page table overflows the AI model’s output limit and silently drops half the rows.
  • A model invents a plausible-looking value, and nobody can tell which page it came from.
  • A few files fail halfway through a big run, forcing you to either restart from zero or babysit fragile retry scripts.

I recently solved this exact problem end-to-end for a customer. By the time the pipeline was accurate, fast, and dependable enough for production, it was clear that this shouldn’t be a one-off custom build. Teams run into this exact same challenge every week.

So I packaged everything we learned into a turnkey, open-source feature set inside Datagrunt v5, with the latest release v5.4.1 available on PyPI today.

# Step 1: Parse PDFs into clean text, tables, Markdown, and bundles (Python 3.11+)
uv pip install "datagrunt[pdf]"

# Step 2: Add AI-powered schema extraction with cross-model checks
uv pip install "datagrunt[pdf,rationalize]"

Why Datagrunt Splits PDF Processing Into Two Steps

Most PDF-to-AI scripts try to do everything in one big leap. Datagrunt v5 takes a calmer, two-step approach:

  1. Extraction (no AI required): Read the PDF into clean, organized pieces: headings, paragraphs, tables, lists, key-value pairs, and page images, all in natural reading order.
  2. Rationalization (AI with guardrails): Tell Datagrunt the exact columns you want using a simple Python class, let AI models fill in the rows, verify every answer against the source document, and export straight to a DataFrame or JSONL file.

Splitting the work in two has a huge practical payoff: you only parse and OCR each PDF once. If your business requirements change next week and you want two extra columns, you re-run Step 2 on the saved outputs without re-reading a single PDF.


Step 1: Turn Any PDF Into Structured Building Blocks

In Datagrunt v5, every PDF goes through a single, permissively-licensed engine (PDFium), with optional Tesseract OCR for scanned pages. You don’t have to juggle three different PDF libraries or figure out which page is digital and which one is a scan.

Out of the box, Datagrunt handles the sneaky edge cases that usually break pipelines:

  • Scans vs. digital pages: With ocr="auto", clean digital pages stay fast and crisp, while scanned or garbled pages automatically route to OCR (and optional 4-way rotation detection if a page was scanned sideways).
  • Hidden text protection: Invisible, off-page, or tiny text is flagged rather than silently mixed into your Markdown or AI prompts.
  • No half-written files: Reading a PDF creates zero temp files, and writing outputs is atomic, so a crash never leaves corrupted data behind.

Here is all it takes to inspect a PDF as a Polars DataFrame, pull out its tables, or save a verified bundle:

from datagrunt import PDFReader, PDFWriter

reader = PDFReader("spec_sheet.pdf", ocr="auto")

# View every heading, paragraph, table, and figure as a Polars DataFrame
df = reader.to_dataframe()

# Pull tables directly into DataFrames
for table in reader.tables:
    print(table.scope_path)      # which section heading this table belongs to
    print(table.to_dataframe())

# Save a reusable, checksummed bundle (JSON, Markdown, images, and page PNGs for scans)
PDFWriter("spec_sheet.pdf", ocr="auto").write_bundle(
    "bundles/spec_sheet",
    render_pages="vision_only",
)

Have an entire folder of PDFs? Run extract_corpus() once:

from datagrunt import extract_corpus

corpus = extract_corpus(
    "incoming_pdfs/",
    "bundles/",
    max_workers=4,
    ocr="auto",
    render_pages="vision_only",
    strict=False,  # bad PDFs are logged in corpus.failures without stopping the rest
)

print(f"Extracted: {len(corpus.entries)}, Failed: {len(corpus.failures)}")

If you drop 50 new PDFs into that folder tomorrow and run the exact same code, Datagrunt checks each file’s fingerprint, skips every unchanged PDF automatically, and processes only the 50 new ones.


Step 2: Turn Extracted PDFs Into Trusted Tables

Once your PDFs are extracted, rationalization turns unstructured pages into the exact schema your team needs.

Instead of crossing your fingers and hoping one AI call gets every field right, Datagrunt v5 builds practical quality checks right into the API:

  • Two independent opinions (min_families=2): Send the same document to two different model families (for example, Google’s Gemini 4 and Anthropic’s Claude). When both agree, your confidence shoots up. When they disagree, Datagrunt flags the item for review instead of silently guessing.
  • Show your work (grounding & provenance): Models must cite the exact element and quote where they found each answer. Datagrunt checks that quote against the real document and attaches page numbers and bounding boxes to your output.
  • Smart identity matching (Key and Informational): Mark a field like part_number with Key(..., normalizer="alnum_upper") so "04-60008-27" and "046000827" merge into one clean record across pages. Mark free-text fields like notes as Informational so minor wording differences between models never cause unnecessary disputes.
  • Automatic table splitting (split_on_truncation=True): If a massive table makes a model run out of output room, Datagrunt automatically splits that section or table in half and retries until every row comes back intact.
  • Choose your view (new in v5.4.1): Export strict consensus rows, a wider grounded-union view, or flat per-model audit rows (by_model=True) when you want to see exactly what each model extracted.

Here is a complete, practical example:

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, rationalize


class PartMention(BaseModel):
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    manufacturer: str | None = None
    notes: Annotated[str | None, Informational] = None


result = rationalize(
    "bundles/",
    schema=PartMention,
    prompt="Extract every referenced part number, category, manufacturer, and notes.",
    models=(
        "gemini:gemini-4-hydrogen",
        "anthropic:claude-opus-5-5",
    ),
    min_families=2,
    cache_dir=".dg-cache",
)

# Clean rows where both model families agreed
clean_df = result.to_dataframe()

# Broader grounded set (keeps verified single-model catches too)
union_df = result.to_dataframe(include_grounded_union=True)

# New in 5.4.1: one row per model for easy side-by-side auditing
audit_df = result.to_dataframe(include_grounded_union=True, by_model=True)

# Save atomic JSONL with quotes, page numbers, and bounding boxes included
result.to_jsonl("out/parts.jsonl")

print(f"Ready for analytics: {len(result.records)}, Needs human review: {len(result.needs_review)}")

Need more than one table from the same PDFs?

Real documents rarely fit into a single table. Often you want one summary row per document (title, supplier, effective date) and a separate table for every line item inside it.

In v5.4+, rationalize_many() lets you declare multiple schemas with PassSpec and run them together in one pass, reading your bundles once and sharing the same connections and cache:

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, PassSpec, rationalize_many
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth


class DocumentSummary(BaseModel):
    document_title: str
    supplier: str | None = None
    effective_date: str | None = None
    notes: Annotated[str | None, Informational] = None


class PartMention(BaseModel):
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    manufacturer: str | None = None
    notes: Annotated[str | None, Informational] = None


auth = GoogleCloudAuth()
models = (
    GeminiProvider.google_cloud("gemini-4-neon", project="my-gcp-project", auth=auth),
    AnthropicProvider.google_cloud(
        "claude-opus-5-5",
        project="my-gcp-project",
        auth=auth,
        schema_mode="prompt_fallback",
    ),
)

results = rationalize_many(
    "bundles/",
    passes=[
        PassSpec(schema=DocumentSummary, prompt="Extract document title, supplier, and effective date.", name="summaries"),
        PassSpec(schema=PartMention, prompt="Extract every part listed in the document.", name="parts"),
    ],
    models=models,
    max_concurrency=64,
    event_delivery="live",
    cache_dir=".dg-cache",
)

results["summaries"].to_jsonl("out/summaries.jsonl")
results["parts"].to_jsonl("out/parts.jsonl")

Fast Without Being Fragile: How Concurrency Works Under the Hood

Speeding up PDF pipelines is tricky because the two stages fail in totally different ways. Datagrunt v5 handles concurrency differently in each stage so you get high throughput without babysitting crashes:

  • In Stage 1 (CPU work): When extracting a folder of PDFs, Datagrunt runs a pool of reusable worker processes and starts the largest files first so one huge 300-page manual doesn’t keep every other worker waiting at the finish line. Optional sandboxing puts a strict 2 GiB memory ceiling on every page so one broken PDF can’t crash your job. Running inside Spark, Beam, Flink, or Celery? Datagrunt detects it automatically and avoids spawning nested process storms.
  • In Stage 2 (AI API calls): Datagrunt can keep up to 256 requests in flight at once over reusable HTTPS connections, while giving each AI provider its own concurrency cap and optional requests-per-minute throttle. If one provider slows down or sends a rate-limit response, the other provider keeps cruising in its own lane. Every request has a hard deadline that cuts stalled connections cleanly, every completed unit is cached on disk so restarts never pay twice, and all results still come back in clean, deterministic order.

You tune a couple of simple numbers (max_workers, max_concurrency, max_concurrency_per_provider), and Datagrunt handles the messy scheduling, retries, timeouts, and caching behind the scenes.


Built for Both Event-Driven and Batch Pipelines

One of my biggest goals when adding this to Datagrunt was making sure the exact same library works cleanly in both ways teams actually run document pipelines:

Event-Driven Pipelines (One PDF Arrives, Clean Data Flows Out)

A customer uploads a PDF to Cloud Storage or S3, triggering a Cloud Run service or serverless function. Your handler calls PDFWriter.write_bundle() and rationalize() on that single file, writes the provenance-rich JSONL straight into BigQuery, Snowflake, Iceberg, or DuckDB, and drops anything ambiguous into a human review queue. Atomic writes and on-disk caching make retries safe and idempotent.

Batch Pipelines (Backfills and Nightly Runs)

Need to process 20,000 historical PDFs overnight? Use extract_corpus() (locally, on a VM, or inside Spark/Airflow) to skip unchanged files and extract the rest in parallel. Then either run rationalize_many() at high concurrency, or use prepare_batch() and ingest_batch() to send jobs to offline Vertex AI and Anthropic Message Batches for 50% batch token pricing with zero online quota pressure:

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, ingest_batch, prepare_batch
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth


class PartMention(BaseModel):
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    manufacturer: str | None = None
    notes: Annotated[str | None, Informational] = None


auth = GoogleCloudAuth()
models = (
    GeminiProvider.google_cloud("gemini-4-neon", project="my-gcp-project", auth=auth),
    AnthropicProvider.google_cloud(
        "claude-opus-5-5",
        project="my-gcp-project",
        auth=auth,
        schema_mode="prompt_fallback",
    ),
)

plan = prepare_batch("bundles/", schema=PartMention, prompt="Extract every part.", models=models)
plan.to_vertex_jsonl("batch/gemini_in.jsonl")
plan.to_anthropic_jsonl("batch/claude_in.jsonl")

# After your cloud batch jobs finish:
result = ingest_batch(plan, ["batch/gemini_out.jsonl", "batch/claude_out.jsonl"])
result.to_jsonl("out/parts.jsonl")

Same schema. Same grounding rules. Same output format, whether you process one file a minute or 50,000 overnight.


A Reliable Deterministic Tool for AI Agents, Too

Beyond batch backfills and event-driven pipelines, there is a third place where Datagrunt v5 shines: powering AI agents.

The most reliable AI agents do not try to do everything through open-ended prompting. They pair reasoning with deterministic tools that execute repeatable work cleanly and return trustworthy results.

Giving an AI agent raw PDFs and asking it to improvise parsing code is slow, expensive, and error-prone. Giving an agent Datagrunt v5 as a tool gives it solid ground to stand on:

  • Predictable inputs every time: Stage 1 turns messy PDFs into clean tables, Markdown, and explicit flags for scanned pages or hidden text, protecting the agent from garbled OCR and hidden prompt injections.
  • Guardrails built into the tool: Schema validation, cross-model consensus, automatic table splitting, caching, and hard timeouts all happen deterministically inside Datagrunt, so the agent stops wasting tokens retrying broken JSON or waiting on stalled requests.
  • Clear handoffs when human judgment is needed: Every extracted row carries quotes, page numbers, and bounding boxes, split cleanly between result.records and result.needs_review. The agent can automate the verified records and hand only the genuinely ambiguous items to a person.

Give Datagrunt v5.4.1 a Try

Datagrunt began as a simple way to stop writing the same CSV boilerplate over and over. With v5.4.1, it brings that same practical, batteries-included experience to PDFs: turning a hard customer problem into a few readable lines of Python.

Try it on your ugliest PDFs and let me know what you build with it.