Datagrunt v5 (5.4.1) Deep Dive: Turnkey PDF Extraction, Schema-Driven Rationalization, and Concurrency

November 20, 2026by Martin Graham

How a messy customer PDF problem turned into a two-stage, high-concurrency, open-source Python stack for batch pipelines, event-driven services, and AI agents.


If you have ever been asked to turn a folder of real-world PDFs (engineering datasheets, vendor contracts, invoices, or mixed digital-and-scanned manuals) into clean analytical tables, you already know how the story usually goes.

You start with a quick prototype: grab a PDF parser, dump the extracted text into an LLM prompt, ask for JSON, and write the output to a database. On the first five clean PDFs, it looks like magic.

Then production hits.

A scanned PDF has a broken, garbled OCR layer underneath it. A 40-page catalog hits the model’s output token limit and truncates half the table. An invisible text block buried off-page quietly pollutes the prompt. One model hallucinates a part number that looks plausible until someone checks the original page, except you have no bounding box or quote pointing back to where it came from. And when you try to scale from 5 files to 50,000, your script either explodes in memory, trips provider rate limits, or re-processes everything from scratch after a transient 429 error.

I recently solved this exact problem end-to-end for a customer. Getting it right meant building deterministic PDF deconstruction, scan and hidden-text handling, multi-model consensus, element-level provenance, truncation recovery, caching, and disciplined concurrency. Once the dust settled, it was obvious that nobody should have to rebuild that scaffolding from scratch every time a business problem starts with a pile of PDFs.

So I made it turnkey and built it straight into Datagrunt v5, with the latest release, v5.4.1, live on PyPI today.

# Deterministic PDF parsing, OCR, bundles, and corpora (Python 3.11+)
uv pip install "datagrunt[pdf]"

# Add schema-driven LLM rationalization, consensus, and grounding
uv pip install "datagrunt[pdf,rationalize]"

The Core Idea: Separate Deterministic Extraction from AI Rationalization

The single biggest architectural decision in Datagrunt v5 is refusing to mash PDF parsing and LLM inference into one opaque step. Instead, Datagrunt splits PDF processing into two clean stages connected by inspectable, checksummed artifacts:

Raw PDFs
   │
   ▼  Stage 1: Deterministic Extraction (zero AI)
   │  PDFium + Pillow + optional Tesseract OCR
   ▼
Typed PDFDocument / Checksummed Bundles / CorpusManifest
   │  (json/, elements.jsonl, markdown/, images/, page_images/, manifest.json)
   │
   ▼  Stage 2: Schema-Driven Rationalization (Pydantic + LLMs)
   │  Coalescing · Multimodal routing · Cross-family consensus · Grounding · Folding
   ▼
Queryable Outputs with Provenance
   (Polars DataFrames · Apache Arrow · Provenance JSONL · Review Queue)

Separating these two stages pays off immediately:

  • Zero wasted tokens: You parse and OCR each PDF once. If you tweak a Pydantic schema or add a new extraction pass later, Stage 2 runs directly over the verified Stage 1 bundles without touching the PDFs again.
  • Deterministic debugging: When an extracted record looks wrong, you can inspect the exact Stage 1 Markdown, JSON elements, and rendered page images before blaming a prompt or a model.
  • Clean pipeline boundaries: Stage 1 is CPU- and memory-bound; Stage 2 is I/O- and quota-bound. Separating them lets you scale each stage independently: locally, in serverless jobs, or inside Spark and Airflow.

Stage 1: Deterministic PDF Extraction (v5.0 to 5.4)

Datagrunt 5.0 replaces the older multi-engine PDF dependencies (PyMuPDF, pdfplumber, pytesseract) with a single permissively-licensed core engine: PDFium (via pypdfium2), paired with Pillow and optional system Tesseract for scanned pages.

Every PDF becomes a typed, reading-order PDFDocument made of Heading, Paragraph, KeyValue, Table, BulletList, Figure, and PageImage elements. Every element carries its page_number, reading_order, bbox (bounding box in points), and scope_path (the heading hierarchy it sits under).

What’s built in out of the box

  • Hidden text is flagged, never silently trusted: Off-page, invisible, tiny-font, transparent, background-colored, or superseded text gets tagged in hidden_reasons. JSON and DataFrames retain it for auditing, while Markdown and Stage 2 LLM prompts strip it by default, closing a common PDF prompt-injection vector.
  • Automatic scan and garbled-text detection: Every page gets a route (digital, hybrid, or vision), a status, and image_only_reasons (too_little_text, garbled_text, extraction_failed). With ocr="auto", Datagrunt routes only the pages that actually need OCR to Tesseract. Optional orientation_fallback=True runs a 4-way (0°/90°/180°/270°) OCR confidence vote on rotated scans.
  • Zero-write reads and atomic writes: PDFReader writes nothing to disk, not even temp files. PDFWriter writes atomically via temp-file-and-rename so crashes never leave half-written outputs.
  • Checksummed bundles & incremental corpora: write_bundle() emits JSON, JSONL, Markdown, deduplicated figures, optional page renders (render_pages="vision_only" renders PNGs only for scanned/image-only pages), and a SHA-256 manifest.json. extract_corpus() runs across whole folders and automatically reuses unchanged bundles on re-runs.

Reading a PDF and writing a verified bundle

from datagrunt import PDFReader, PDFWriter

reader = PDFReader("datasheet.pdf", ocr="auto")

# Explore as a Polars DataFrame, Arrow table, Markdown, or DuckDB SQL
df = reader.to_dataframe()
markdown = reader.to_markdown()

for table in reader.tables:
    print(table.scope_path)
    print(table.to_dataframe())

# Query extracted elements directly with DuckDB SQL
summary = reader.query_data(
    f"SELECT kind, COUNT(*) AS n FROM {reader.db_table} GROUP BY kind"
).pl()

# Persist a SHA-256 verified bundle (rendering PNGs only for scanned/image-only pages)
PDFWriter("datasheet.pdf", ocr="auto").write_bundle(
    "bundles/datasheet",
    render_pages="vision_only",
)

Extracting an entire folder incrementally

from datagrunt import extract_corpus

corpus = extract_corpus(
    "incoming_pdfs/",
    "bundles/",
    max_workers=4,
    ocr="auto",
    render_pages="vision_only",  # fast on digital pages, full PNGs for scans
    strict=False,                # isolate corrupt PDFs into corpus.failures
)

print(f"Ready: {len(corpus.entries)} bundles, Failed: {len(corpus.failures)}")

If you run that exact snippet again tomorrow after adding 20 new PDFs to incoming_pdfs/, Datagrunt checks the source SHA-256 and extraction options, skips every unchanged document (reused=True), and extracts only the 20 new files.


Stage 2: Schema-Driven PDF Rationalization (v5.1 to 5.4.1)

Stage 1 gives you structured document layout. Stage 2 (rationalize() and rationalize_many()) turns that layout into validated, grounded Pydantic records ready for downstream analytics.

Here is everything new across v5.1 through v5.4.1:

  • Typed schema markers (Key and Informational): Annotate entity identity fields with Key, including normalizers like Key("part", normalizer="alnum_upper"), so "04-60008-27" and "046000827" fold into a single entity across pages while still verifying each model’s raw printed string against the source text. Mark descriptive free-text or list fields as Informational so harmless phrasing differences never trigger false disputes.
  • Cross-family consensus (min_families=2): Run models from distinct families (for example, Gemini + Claude, or OpenAI, Ollama, and Vertex AI via GoogleCloudAuth). Agreement across independent model families dramatically cuts hallucinations without writing custom adjudication code.
  • 3-witness quote, key, and scan grounding: Every extracted record must cite non-header source elements and verbatim grounding quotes verified against the visible elements (exact substring on digital text, ≥ 0.85 token containment on OCR, and cross-family peer-quote/field corroboration on pure image_only scans), carrying ElementSource page numbers and bounding boxes all the way to your output.
  • Multimodal VLM routing (multimodal=True): Clean digital sections travel as compact text; vision and image_only pages automatically attach their rendered page PNGs to vision-capable models.
  • Adaptive coalescing & automatic truncation bisection (v5.4.0): Consecutive digital sections coalesce up to max_unit_chars to minimize API calls. When a dense table or long list hits finish="truncated", split_on_truncation=True (on by default) automatically bisects the unit, and even splits oversized Table and BulletList elements at their midpoint (up to 16 rounds), until every row fits cleanly.
  • Multi-schema extraction in one pass (rationalize_many() + PassSpec, v5.4.0): Extract multiple schemas (e.g., document metadata + bill-of-materials line items) over one parallel bundle load, one shared HTTP connection pool, and one rate-limiter map.
  • Tiered exports & per-model auditability (v5.4.1): Choose the precision/recall tradeoff your pipeline needs: strict consensus (result.records), grounded union (include_grounded_union=True, keeping verified single-family catches alongside _supporting_models and _model_outputs), flat per-model audit rows (by_model=True, brand new in v5.4.1), human review queues (result.needs_review), atomic provenance-by-default to_jsonl(), and JSON-serializable result.telemetry.to_dict().

Simple single-schema rationalization

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, rationalize


class PartMention(BaseModel):
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    manufacturer: str | None = None
    notes: Annotated[str | None, Informational] = None


result = rationalize(
    "bundles/",  # accepts a PDF path, PDFDocument, bundle dir, or CorpusManifest
    schema=PartMention,
    prompt="Extract every referenced part number, category, manufacturer, and notes.",
    models=(
        "gemini:gemini-4-hydrogen",
        "anthropic:claude-opus-5-5",
    ),
    min_families=2,
    cache_dir=".dg-cache",
)

# 1. High-precision consensus records as a Polars DataFrame
strict_df = result.to_dataframe()

# 2. Grounded union (consensus + element-verified single-family catches + _model_outputs)
union_df = result.to_dataframe(include_grounded_union=True)

# 3. New in 5.4.1: flat per-model audit rows for precision/recall evaluation
audit_df = result.to_dataframe(include_grounded_union=True, by_model=True)

# 4. Atomic JSONL with full page + bounding-box provenance
result.to_jsonl("out/part_mentions.jsonl")

# 5. Disputed, ungrounded, or cross-page-conflict items for human review
print(f"Clean: {len(result.records)}, Needs review: {len(result.needs_review)}")

Multi-schema extraction in one pass (rationalize_many + PassSpec)

When you need more than one table out of the same corpus (say, a document-level profile and every line item), running separate rationalize() calls reloads bundles and spins up separate HTTP pools. In v5.4+, rationalize_many() runs all passes together over one load and one shared transport:

from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, PassSpec, rationalize_many
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth


class DocumentProfile(BaseModel):
    document_title: str
    issuer: str | None = None
    effective_date: str | None = None
    notes: Annotated[str | None, Informational] = None


class PartMention(BaseModel):
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    manufacturer: str | None = None
    notes: Annotated[str | None, Informational] = None


auth = GoogleCloudAuth()
models = (
    GeminiProvider.google_cloud("gemini-4-neon", project="my-gcp-project", auth=auth),
    AnthropicProvider.google_cloud(
        "claude-opus-5-5",
        project="my-gcp-project",
        auth=auth,
        schema_mode="prompt_fallback",  # auto-latches prompt mode if native grammar hits 400s
    ),
)

results = rationalize_many(
    "bundles/",
    passes=[
        PassSpec(schema=DocumentProfile, prompt="Extract document title, issuer, and effective date.", name="profiles"),
        PassSpec(schema=PartMention, prompt="Extract every part in the BOM.", name="parts", max_output_tokens=16_384),
    ],
    models=models,
    max_concurrency=64,
    max_concurrency_per_provider=32,
    event_delivery="live",   # stream TelemetryEvents as they arrive, no head-of-line blocking
    cache_dir=".dg-cache",
)

results["profiles"].to_jsonl("out/profiles.jsonl")
results["parts"].to_jsonl("out/parts.jsonl")

Under the Hood: How Datagrunt v5 Handles Concurrency

PDF pipelines hit two completely different concurrency walls:

  1. Stage 1 is CPU- and native-memory-bound (C++ PDFium parsing, rasterization, Tesseract OCR).
  2. Stage 2 is network-I/O- and quota-bound (hundreds of concurrent HTTPS requests across LLM providers with wildly different latency and rate limits).

Trying to solve both with a single naive multiprocessing.Pool or raw asyncio loop is where production pipelines usually break. Datagrunt v5 engineers each layer specifically for its workload.

Stage 1 Concurrency: Process Safety, LPT Scheduling, and Zero Boilerplate

  • Zero-overhead default fast path (sandbox=False, workers=1): For single-PDF or small workloads, PDFReader and PDFWriter run in-process with zero cold Python subprocess spawns, serializing native PDFium calls under a re-entrant lock.
  • Warm process pools + Graham LPT scheduling (extract_corpus): When extracting a corpus with max_workers > 1 (up to 64 workers), Datagrunt maintains a pool of warm isolated child processes and schedules input files largest-file-first (Graham’s Longest Processing Time algorithm) when strict=False. Starting the 300-page service manuals first prevents a single giant file at the end of the queue from dominating total wall-clock time.
  • Memory-capped isolated sandboxes (workers > 1 or sandbox=True): Intra-document parallel page extraction and sandboxed renders execute inside isolated python -I workers capped at 2 GiB (RLIMIT_AS). Each worker leads its own POSIX process group so a timed-out or runaway page cleanly kills any child render or Tesseract processes, turning that page into a recorded error page while the rest of the document keeps going. Because workers bootstrap from a fixed allowlist rather than importing your script, you never need an if __name__ == "__main__": guard.
  • Safe inside distributed compute: Inside Apache Spark, Apache Beam, Apache Flink, or Celery, Datagrunt automatically detects the runtime environment and clamps per-document worker pools to 1 so executors never fork-bomb a cluster node.
  • Parallel bundle pre-loading: When Stage 2 opens a corpus, load_corpus_documents() verifies SHA-256 manifests and reads bundles concurrently across up to 16 threads in a single pass, sharing those loaded documents across every pass in rationalize_many().

Stage 2 Concurrency: High-Throughput, Quota-Isolated, Deterministic I/O

  • Bounded thread pool up to 256 in-flight requests: rationalize() and rationalize_many() run a bounded ThreadPoolExecutor (max_concurrency up to CONCURRENCY_LIMIT = 256), sized to min(max_concurrency, cache_misses).
  • Per-rate-key semaphores & token-bucket pacing: Every provider carries a rate_key guarded by its own BoundedSemaphore (max_concurrency_per_provider) and optional rpm_limit token bucket. Direct API endpoints isolate by normalized base URL, while Google Cloud / Vertex AI endpoints isolate per publisher + model resource so Gemini and Claude running in the same GCP project never throttle each other.
  • Min-heap work scheduling (no provider head-of-line blocking): The runner schedules the earliest ready request across any rate key with an open slot and available token. If one provider momentarily backs off on a 429 or 529, the other provider keeps running at full speed.
  • Pooled TLS keep-alive + hard socket-level deadlines: Providers reuse pooled HTTP/1.1 connections (64 pooled connections by default, configurable up to 1,024, with 60-second TLS keep-alive). Every request carries a hard monotonic deadline timer that physically shuts down the underlying socket (even on pooled keep-alive sockets) if a server stalls mid-stream or hangs during a TLS handshake, and retires the failed transport pool cleanly.
  • Deterministic assembly + live telemetry: Even with 256 requests completing out of order across multiple providers and truncation-bisection rounds, Datagrunt assembles every RationalizedResult in deterministic (document, unit, provider) order. Telemetry hooks (on_event) support either event_delivery="ordered" (deterministic miss order) or event_delivery="live" (v5.4.0, emitting events immediately on arrival without head-of-line blocking).
  • Two-tier caching (MemoryCache + sharded FileCache): Before a single network request goes out, Datagrunt probes the in-memory and sharded atomic on-disk caches concurrently by (unit, envelope, model, prompt, schema). Crashes, retries, and incremental reruns never pay twice for the same unit.

Built for Both Event-Driven and Batch Document Pipelines

One reason I wanted this inside Datagrunt, rather than locked inside a monolithic framework, is that real organizations process documents in two very different ways and often need both in the same architecture.

1. Event-Driven Pipelines (File Drop → Instant Structured Records)

Think of a document landing in Cloud Storage or S3, firing a Pub/Sub or Eventarc notification to a Cloud Run service or Lambda function:

  • Call PDFWriter(path, ocr="auto").write_bundle(bundle_dir, render_pages="vision_only") for that single document.
  • Run rationalize() or rationalize_many() with cache_dir pointed at persistent storage (so retries on delivery duplicates are free and idempotent).
  • Emit result.to_jsonl(), with _doc_id, _status, _supporting_models, _grounding_quotes, and _sources bounding boxes included by default, straight into BigQuery, Iceberg, Snowflake, or DuckDB, while routing result.needs_review to a review queue.

Zero temp files on read, atomic writes on output, bounded memory, and hard HTTP timeouts mean a single malformed PDF never takes down your worker container.

2. Batch Pipelines (Backfills, Nightly Runs, and Offline Cloud Batches)

Now think of backfilling 50,000 historical PDFs or running a nightly Spark, Airflow, or Cloud Run Jobs pipeline:

  • Run extract_corpus() with max_workers tuned to your machine, or fan out across Spark/Beam executors where Datagrunt automatically clamps inner pools and skips unchanged bundles by SHA-256.
  • For online high-throughput rationalization, scale rationalize_many(..., max_concurrency=64) across pooled connections.
  • Want 50% batch token discounts and zero online quota contention? Use Datagrunt’s offline cloud batch bridge (prepare_batch and ingest_batch). It runs the exact same preflight, coalescing, envelope rendering, and cache probing without making a single network call, writes ready-to-submit JSONL for Vertex AI or Anthropic Message Batches, and ingests the completed batch outputs back through the identical consensus, grounding, and folding engine:
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, ingest_batch, prepare_batch
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth


class PartMention(BaseModel):
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    manufacturer: str | None = None
    notes: Annotated[str | None, Informational] = None


auth = GoogleCloudAuth()
models = (
    GeminiProvider.google_cloud("gemini-4-neon", project="my-gcp-project", auth=auth),
    AnthropicProvider.google_cloud(
        "claude-opus-5-5",
        project="my-gcp-project",
        auth=auth,
        schema_mode="prompt_fallback",
    ),
)

# 1. Build uncached batch requests (zero network calls)
plan = prepare_batch(
    "bundles/",
    schema=PartMention,
    prompt="Extract every referenced engineering part.",
    models=models,
    cache_dir=".dg-cache",
)

plan.to_vertex_jsonl("batch/gemini_requests.jsonl")
plan.to_anthropic_jsonl("batch/claude_requests.jsonl")

# 2. Submit to Vertex AI / Anthropic Batch APIs, then ingest completed outputs
result = ingest_batch(
    plan,
    ["batch/gemini_outputs.jsonl", "batch/claude_outputs.jsonl"],
)

result.to_jsonl("out/part_mentions.jsonl")

Whether a document arrives one-by-one at 2 PM or in a 50,000-file historical backfill at midnight, your schema, grounding rules, consensus logic, and output contract stay 100% identical.


Built as a Deterministic Tool for AI Agents

In addition to batch and event-driven pipelines, Datagrunt v5 fills a gap that shows up fast when teams put AI agents into production: AI agents work best when they call deterministic tools.

Handing an agent raw PDF bytes, ad-hoc parsing scripts, or unconstrained LLM calls compounds nondeterminism at every step. Wrapping Datagrunt v5 as an agent tool (directly in Python, via CLI, or behind an MCP server) gives the agent a strict, predictable contract:

  • Deterministic document inspection: Stage 1 returns identical typed elements (element_id, bbox, scope_path), SHA-256 verified bundles, and explicit hidden_reasons and image_only_reasons flags. The agent knows immediately whether a page is clean text, a scan, or carrying hidden text that could inject prompts.
  • Fail-fast bounds that protect agent loops: Invalid options, encrypted or corrupt PDFs, 2 GiB worker memory caps (RLIMIT_AS), and socket-level HTTP deadlines fail loudly with typed exceptions (DatagruntPDFError, RationalizationError) instead of hanging an autonomous loop.
  • Structured verification and escalation: Stage 2 enforces Pydantic schemas, cross-family consensus, 3-witness grounding, and deterministic output ordering, plus JSON-serializable telemetry (result.telemetry.to_dict()). The agent can act on result.records with element-level provenance and escalate only result.needs_review to a human or downstream adjudication step.

Try Datagrunt v5.4.1 Today

Datagrunt started as a way to eliminate repetitive CSV wrangling, and v5 brings that same pragmatic, low-ceremony philosophy to unstructured PDFs. Everything I had to build the hard way for a real customer (deterministic PDFium parsing, scan and hidden-text safety, cross-family consensus, quote and bounding-box grounding, truncation bisection, per-model audit trails, and bounded concurrency) is now a uv pip install away.

If you’re building batch or event-driven PDF pipelines, or migrating off fragile custom PDF-to-LLM glue code, give v5.4.1 a spin and let me know what breaks or what you’d like to see next.