Datagrunt v5 (5.4.1) Deep Dive: Turnkey PDF Extraction, Schema-Driven Rationalization, and Concurrency
How a messy customer PDF problem turned into a two-stage, high-concurrency, open-source Python stack for batch pipelines, event-driven services, and AI agents.
If you have ever been asked to turn a folder of real-world PDFs (engineering datasheets, vendor contracts, invoices, or mixed digital-and-scanned manuals) into clean analytical tables, you already know how the story usually goes.
You start with a quick prototype: grab a PDF parser, dump the extracted text into an LLM prompt, ask for JSON, and write the output to a database. On the first five clean PDFs, it looks like magic.
Then production hits.
A scanned PDF has a broken, garbled OCR layer underneath it. A 40-page catalog hits the model’s output token limit and truncates half the table. An invisible text block buried off-page quietly pollutes the prompt. One model hallucinates a part number that looks plausible until someone checks the original page, except you have no bounding box or quote pointing back to where it came from. And when you try to scale from 5 files to 50,000, your script either explodes in memory, trips provider rate limits, or re-processes everything from scratch after a transient 429 error.
I recently solved this exact problem end-to-end for a customer. Getting it right meant building deterministic PDF deconstruction, scan and hidden-text handling, multi-model consensus, element-level provenance, truncation recovery, caching, and disciplined concurrency. Once the dust settled, it was obvious that nobody should have to rebuild that scaffolding from scratch every time a business problem starts with a pile of PDFs.
So I made it turnkey and built it straight into Datagrunt v5, with the latest release, v5.4.1, live on PyPI today.
# Deterministic PDF parsing, OCR, bundles, and corpora (Python 3.11+)
uv pip install "datagrunt[pdf]"
# Add schema-driven LLM rationalization, consensus, and grounding
uv pip install "datagrunt[pdf,rationalize]"The Core Idea: Separate Deterministic Extraction from AI Rationalization
The single biggest architectural decision in Datagrunt v5 is refusing to mash PDF parsing and LLM inference into one opaque step. Instead, Datagrunt splits PDF processing into two clean stages connected by inspectable, checksummed artifacts:
Raw PDFs
│
▼ Stage 1: Deterministic Extraction (zero AI)
│ PDFium + Pillow + optional Tesseract OCR
▼
Typed PDFDocument / Checksummed Bundles / CorpusManifest
│ (json/, elements.jsonl, markdown/, images/, page_images/, manifest.json)
│
▼ Stage 2: Schema-Driven Rationalization (Pydantic + LLMs)
│ Coalescing · Multimodal routing · Cross-family consensus · Grounding · Folding
▼
Queryable Outputs with Provenance
(Polars DataFrames · Apache Arrow · Provenance JSONL · Review Queue)Separating these two stages pays off immediately:
- Zero wasted tokens: You parse and OCR each PDF once. If you tweak a Pydantic schema or add a new extraction pass later, Stage 2 runs directly over the verified Stage 1 bundles without touching the PDFs again.
- Deterministic debugging: When an extracted record looks wrong, you can inspect the exact Stage 1 Markdown, JSON elements, and rendered page images before blaming a prompt or a model.
- Clean pipeline boundaries: Stage 1 is CPU- and memory-bound; Stage 2 is I/O- and quota-bound. Separating them lets you scale each stage independently: locally, in serverless jobs, or inside Spark and Airflow.
Stage 1: Deterministic PDF Extraction (v5.0 to 5.4)
Datagrunt 5.0 replaces the older multi-engine PDF dependencies (PyMuPDF, pdfplumber, pytesseract) with a single permissively-licensed core engine: PDFium (via pypdfium2), paired with Pillow and optional system Tesseract for scanned pages.
Every PDF becomes a typed, reading-order PDFDocument made of Heading, Paragraph, KeyValue, Table, BulletList, Figure, and PageImage elements. Every element carries its page_number, reading_order, bbox (bounding box in points), and scope_path (the heading hierarchy it sits under).
What’s built in out of the box
- Hidden text is flagged, never silently trusted: Off-page, invisible, tiny-font, transparent, background-colored, or superseded text gets tagged in
hidden_reasons. JSON and DataFrames retain it for auditing, while Markdown and Stage 2 LLM prompts strip it by default, closing a common PDF prompt-injection vector. - Automatic scan and garbled-text detection: Every page gets a route (
digital,hybrid, orvision), a status, andimage_only_reasons(too_little_text,garbled_text,extraction_failed). Withocr="auto", Datagrunt routes only the pages that actually need OCR to Tesseract. Optionalorientation_fallback=Trueruns a 4-way (0°/90°/180°/270°) OCR confidence vote on rotated scans. - Zero-write reads and atomic writes:
PDFReaderwrites nothing to disk, not even temp files.PDFWriterwrites atomically via temp-file-and-rename so crashes never leave half-written outputs. - Checksummed bundles & incremental corpora:
write_bundle()emits JSON, JSONL, Markdown, deduplicated figures, optional page renders (render_pages="vision_only"renders PNGs only for scanned/image-only pages), and a SHA-256manifest.json.extract_corpus()runs across whole folders and automatically reuses unchanged bundles on re-runs.
Reading a PDF and writing a verified bundle
from datagrunt import PDFReader, PDFWriter
reader = PDFReader("datasheet.pdf", ocr="auto")
# Explore as a Polars DataFrame, Arrow table, Markdown, or DuckDB SQL
df = reader.to_dataframe()
markdown = reader.to_markdown()
for table in reader.tables:
print(table.scope_path)
print(table.to_dataframe())
# Query extracted elements directly with DuckDB SQL
summary = reader.query_data(
f"SELECT kind, COUNT(*) AS n FROM {reader.db_table} GROUP BY kind"
).pl()
# Persist a SHA-256 verified bundle (rendering PNGs only for scanned/image-only pages)
PDFWriter("datasheet.pdf", ocr="auto").write_bundle(
"bundles/datasheet",
render_pages="vision_only",
)Extracting an entire folder incrementally
from datagrunt import extract_corpus
corpus = extract_corpus(
"incoming_pdfs/",
"bundles/",
max_workers=4,
ocr="auto",
render_pages="vision_only", # fast on digital pages, full PNGs for scans
strict=False, # isolate corrupt PDFs into corpus.failures
)
print(f"Ready: {len(corpus.entries)} bundles, Failed: {len(corpus.failures)}")If you run that exact snippet again tomorrow after adding 20 new PDFs to incoming_pdfs/, Datagrunt checks the source SHA-256 and extraction options, skips every unchanged document (reused=True), and extracts only the 20 new files.
Stage 2: Schema-Driven PDF Rationalization (v5.1 to 5.4.1)
Stage 1 gives you structured document layout. Stage 2 (rationalize() and rationalize_many()) turns that layout into validated, grounded Pydantic records ready for downstream analytics.
Here is everything new across v5.1 through v5.4.1:
- Typed schema markers (
KeyandInformational): Annotate entity identity fields withKey, including normalizers likeKey("part", normalizer="alnum_upper"), so"04-60008-27"and"046000827"fold into a single entity across pages while still verifying each model’s raw printed string against the source text. Mark descriptive free-text or list fields asInformationalso harmless phrasing differences never trigger false disputes. - Cross-family consensus (
min_families=2): Run models from distinct families (for example, Gemini + Claude, or OpenAI, Ollama, and Vertex AI viaGoogleCloudAuth). Agreement across independent model families dramatically cuts hallucinations without writing custom adjudication code. - 3-witness quote, key, and scan grounding: Every extracted record must cite non-header source elements and verbatim grounding quotes verified against the visible elements (exact substring on digital text, ≥ 0.85 token containment on OCR, and cross-family peer-quote/field corroboration on pure
image_onlyscans), carryingElementSourcepage numbers and bounding boxes all the way to your output. - Multimodal VLM routing (
multimodal=True): Clean digital sections travel as compact text;visionandimage_onlypages automatically attach their rendered page PNGs to vision-capable models. - Adaptive coalescing & automatic truncation bisection (v5.4.0): Consecutive digital sections coalesce up to
max_unit_charsto minimize API calls. When a dense table or long list hitsfinish="truncated",split_on_truncation=True(on by default) automatically bisects the unit, and even splits oversizedTableandBulletListelements at their midpoint (up to 16 rounds), until every row fits cleanly. - Multi-schema extraction in one pass (
rationalize_many()+PassSpec, v5.4.0): Extract multiple schemas (e.g., document metadata + bill-of-materials line items) over one parallel bundle load, one shared HTTP connection pool, and one rate-limiter map. - Tiered exports & per-model auditability (v5.4.1): Choose the precision/recall tradeoff your pipeline needs: strict consensus (
result.records), grounded union (include_grounded_union=True, keeping verified single-family catches alongside_supporting_modelsand_model_outputs), flat per-model audit rows (by_model=True, brand new in v5.4.1), human review queues (result.needs_review), atomic provenance-by-defaultto_jsonl(), and JSON-serializableresult.telemetry.to_dict().
Simple single-schema rationalization
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, rationalize
class PartMention(BaseModel):
part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
category: str
manufacturer: str | None = None
notes: Annotated[str | None, Informational] = None
result = rationalize(
"bundles/", # accepts a PDF path, PDFDocument, bundle dir, or CorpusManifest
schema=PartMention,
prompt="Extract every referenced part number, category, manufacturer, and notes.",
models=(
"gemini:gemini-4-hydrogen",
"anthropic:claude-opus-5-5",
),
min_families=2,
cache_dir=".dg-cache",
)
# 1. High-precision consensus records as a Polars DataFrame
strict_df = result.to_dataframe()
# 2. Grounded union (consensus + element-verified single-family catches + _model_outputs)
union_df = result.to_dataframe(include_grounded_union=True)
# 3. New in 5.4.1: flat per-model audit rows for precision/recall evaluation
audit_df = result.to_dataframe(include_grounded_union=True, by_model=True)
# 4. Atomic JSONL with full page + bounding-box provenance
result.to_jsonl("out/part_mentions.jsonl")
# 5. Disputed, ungrounded, or cross-page-conflict items for human review
print(f"Clean: {len(result.records)}, Needs review: {len(result.needs_review)}")Multi-schema extraction in one pass (rationalize_many + PassSpec)
When you need more than one table out of the same corpus (say, a document-level profile and every line item), running separate rationalize() calls reloads bundles and spins up separate HTTP pools. In v5.4+, rationalize_many() runs all passes together over one load and one shared transport:
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, PassSpec, rationalize_many
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth
class DocumentProfile(BaseModel):
document_title: str
issuer: str | None = None
effective_date: str | None = None
notes: Annotated[str | None, Informational] = None
class PartMention(BaseModel):
part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
category: str
manufacturer: str | None = None
notes: Annotated[str | None, Informational] = None
auth = GoogleCloudAuth()
models = (
GeminiProvider.google_cloud("gemini-4-neon", project="my-gcp-project", auth=auth),
AnthropicProvider.google_cloud(
"claude-opus-5-5",
project="my-gcp-project",
auth=auth,
schema_mode="prompt_fallback", # auto-latches prompt mode if native grammar hits 400s
),
)
results = rationalize_many(
"bundles/",
passes=[
PassSpec(schema=DocumentProfile, prompt="Extract document title, issuer, and effective date.", name="profiles"),
PassSpec(schema=PartMention, prompt="Extract every part in the BOM.", name="parts", max_output_tokens=16_384),
],
models=models,
max_concurrency=64,
max_concurrency_per_provider=32,
event_delivery="live", # stream TelemetryEvents as they arrive, no head-of-line blocking
cache_dir=".dg-cache",
)
results["profiles"].to_jsonl("out/profiles.jsonl")
results["parts"].to_jsonl("out/parts.jsonl")Under the Hood: How Datagrunt v5 Handles Concurrency
PDF pipelines hit two completely different concurrency walls:
- Stage 1 is CPU- and native-memory-bound (C++ PDFium parsing, rasterization, Tesseract OCR).
- Stage 2 is network-I/O- and quota-bound (hundreds of concurrent HTTPS requests across LLM providers with wildly different latency and rate limits).
Trying to solve both with a single naive multiprocessing.Pool or raw asyncio loop is where production pipelines usually break. Datagrunt v5 engineers each layer specifically for its workload.
Stage 1 Concurrency: Process Safety, LPT Scheduling, and Zero Boilerplate
- Zero-overhead default fast path (
sandbox=False, workers=1): For single-PDF or small workloads,PDFReaderandPDFWriterrun in-process with zero cold Python subprocess spawns, serializing native PDFium calls under a re-entrant lock. - Warm process pools + Graham LPT scheduling (
extract_corpus): When extracting a corpus withmax_workers > 1(up to 64 workers), Datagrunt maintains a pool of warm isolated child processes and schedules input files largest-file-first (Graham’s Longest Processing Time algorithm) whenstrict=False. Starting the 300-page service manuals first prevents a single giant file at the end of the queue from dominating total wall-clock time. - Memory-capped isolated sandboxes (
workers > 1orsandbox=True): Intra-document parallel page extraction and sandboxed renders execute inside isolatedpython -Iworkers capped at 2 GiB (RLIMIT_AS). Each worker leads its own POSIX process group so a timed-out or runaway page cleanly kills any child render or Tesseract processes, turning that page into a recorded error page while the rest of the document keeps going. Because workers bootstrap from a fixed allowlist rather than importing your script, you never need anif __name__ == "__main__":guard. - Safe inside distributed compute: Inside Apache Spark, Apache Beam, Apache Flink, or Celery, Datagrunt automatically detects the runtime environment and clamps per-document worker pools to
1so executors never fork-bomb a cluster node. - Parallel bundle pre-loading: When Stage 2 opens a corpus,
load_corpus_documents()verifies SHA-256 manifests and reads bundles concurrently across up to 16 threads in a single pass, sharing those loaded documents across every pass inrationalize_many().
Stage 2 Concurrency: High-Throughput, Quota-Isolated, Deterministic I/O
- Bounded thread pool up to 256 in-flight requests:
rationalize()andrationalize_many()run a boundedThreadPoolExecutor(max_concurrencyup toCONCURRENCY_LIMIT = 256), sized tomin(max_concurrency, cache_misses). - Per-rate-key semaphores & token-bucket pacing: Every provider carries a
rate_keyguarded by its ownBoundedSemaphore(max_concurrency_per_provider) and optionalrpm_limittoken bucket. Direct API endpoints isolate by normalized base URL, while Google Cloud / Vertex AI endpoints isolate per publisher + model resource so Gemini and Claude running in the same GCP project never throttle each other. - Min-heap work scheduling (no provider head-of-line blocking): The runner schedules the earliest ready request across any rate key with an open slot and available token. If one provider momentarily backs off on a 429 or 529, the other provider keeps running at full speed.
- Pooled TLS keep-alive + hard socket-level deadlines: Providers reuse pooled HTTP/1.1 connections (64 pooled connections by default, configurable up to 1,024, with 60-second TLS keep-alive). Every request carries a hard monotonic deadline timer that physically shuts down the underlying socket (even on pooled keep-alive sockets) if a server stalls mid-stream or hangs during a TLS handshake, and retires the failed transport pool cleanly.
- Deterministic assembly + live telemetry: Even with 256 requests completing out of order across multiple providers and truncation-bisection rounds, Datagrunt assembles every
RationalizedResultin deterministic(document, unit, provider)order. Telemetry hooks (on_event) support eitherevent_delivery="ordered"(deterministic miss order) orevent_delivery="live"(v5.4.0, emitting events immediately on arrival without head-of-line blocking). - Two-tier caching (
MemoryCache+ shardedFileCache): Before a single network request goes out, Datagrunt probes the in-memory and sharded atomic on-disk caches concurrently by(unit, envelope, model, prompt, schema). Crashes, retries, and incremental reruns never pay twice for the same unit.
Built for Both Event-Driven and Batch Document Pipelines
One reason I wanted this inside Datagrunt, rather than locked inside a monolithic framework, is that real organizations process documents in two very different ways and often need both in the same architecture.
1. Event-Driven Pipelines (File Drop → Instant Structured Records)
Think of a document landing in Cloud Storage or S3, firing a Pub/Sub or Eventarc notification to a Cloud Run service or Lambda function:
- Call
PDFWriter(path, ocr="auto").write_bundle(bundle_dir, render_pages="vision_only")for that single document. - Run
rationalize()orrationalize_many()withcache_dirpointed at persistent storage (so retries on delivery duplicates are free and idempotent). - Emit
result.to_jsonl(), with_doc_id,_status,_supporting_models,_grounding_quotes, and_sourcesbounding boxes included by default, straight into BigQuery, Iceberg, Snowflake, or DuckDB, while routingresult.needs_reviewto a review queue.
Zero temp files on read, atomic writes on output, bounded memory, and hard HTTP timeouts mean a single malformed PDF never takes down your worker container.
2. Batch Pipelines (Backfills, Nightly Runs, and Offline Cloud Batches)
Now think of backfilling 50,000 historical PDFs or running a nightly Spark, Airflow, or Cloud Run Jobs pipeline:
- Run
extract_corpus()withmax_workerstuned to your machine, or fan out across Spark/Beam executors where Datagrunt automatically clamps inner pools and skips unchanged bundles by SHA-256. - For online high-throughput rationalization, scale
rationalize_many(..., max_concurrency=64)across pooled connections. - Want 50% batch token discounts and zero online quota contention? Use Datagrunt’s offline cloud batch bridge (
prepare_batchandingest_batch). It runs the exact same preflight, coalescing, envelope rendering, and cache probing without making a single network call, writes ready-to-submit JSONL for Vertex AI or Anthropic Message Batches, and ingests the completed batch outputs back through the identical consensus, grounding, and folding engine:
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, ingest_batch, prepare_batch
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth
class PartMention(BaseModel):
part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
category: str
manufacturer: str | None = None
notes: Annotated[str | None, Informational] = None
auth = GoogleCloudAuth()
models = (
GeminiProvider.google_cloud("gemini-4-neon", project="my-gcp-project", auth=auth),
AnthropicProvider.google_cloud(
"claude-opus-5-5",
project="my-gcp-project",
auth=auth,
schema_mode="prompt_fallback",
),
)
# 1. Build uncached batch requests (zero network calls)
plan = prepare_batch(
"bundles/",
schema=PartMention,
prompt="Extract every referenced engineering part.",
models=models,
cache_dir=".dg-cache",
)
plan.to_vertex_jsonl("batch/gemini_requests.jsonl")
plan.to_anthropic_jsonl("batch/claude_requests.jsonl")
# 2. Submit to Vertex AI / Anthropic Batch APIs, then ingest completed outputs
result = ingest_batch(
plan,
["batch/gemini_outputs.jsonl", "batch/claude_outputs.jsonl"],
)
result.to_jsonl("out/part_mentions.jsonl")Whether a document arrives one-by-one at 2 PM or in a 50,000-file historical backfill at midnight, your schema, grounding rules, consensus logic, and output contract stay 100% identical.
Built as a Deterministic Tool for AI Agents
In addition to batch and event-driven pipelines, Datagrunt v5 fills a gap that shows up fast when teams put AI agents into production: AI agents work best when they call deterministic tools.
Handing an agent raw PDF bytes, ad-hoc parsing scripts, or unconstrained LLM calls compounds nondeterminism at every step. Wrapping Datagrunt v5 as an agent tool (directly in Python, via CLI, or behind an MCP server) gives the agent a strict, predictable contract:
- Deterministic document inspection: Stage 1 returns identical typed elements (
element_id,bbox,scope_path), SHA-256 verified bundles, and explicithidden_reasonsandimage_only_reasonsflags. The agent knows immediately whether a page is clean text, a scan, or carrying hidden text that could inject prompts. - Fail-fast bounds that protect agent loops: Invalid options, encrypted or corrupt PDFs, 2 GiB worker memory caps (
RLIMIT_AS), and socket-level HTTP deadlines fail loudly with typed exceptions (DatagruntPDFError,RationalizationError) instead of hanging an autonomous loop. - Structured verification and escalation: Stage 2 enforces Pydantic schemas, cross-family consensus, 3-witness grounding, and deterministic output ordering, plus JSON-serializable telemetry (
result.telemetry.to_dict()). The agent can act onresult.recordswith element-level provenance and escalate onlyresult.needs_reviewto a human or downstream adjudication step.
Try Datagrunt v5.4.1 Today
Datagrunt started as a way to eliminate repetitive CSV wrangling, and v5 brings that same pragmatic, low-ceremony philosophy to unstructured PDFs. Everything I had to build the hard way for a real customer (deterministic PDFium parsing, scan and hidden-text safety, cross-family consensus, quote and bounding-box grounding, truncation bisection, per-model audit trails, and bounded concurrency) is now a uv pip install away.
- Install:
uv pip install "datagrunt[pdf,rationalize]"(requires Python 3.11+) - Docs & Migration Guide: datagrunt.io/docs
- GitHub: github.com/pmgraham/datagrunt
- PyPI: pypi.org/project/datagrunt/
If you’re building batch or event-driven PDF pipelines, or migrating off fragile custom PDF-to-LLM glue code, give v5.4.1 a spin and let me know what breaks or what you’d like to see next.