Datagrunt 5.3.12: Multi-Document Corpora, Process Sandboxing & Multimodal Rationalization
Datagrunt 5.3.12 is now available! This release brings comprehensive documentation updates and refined APIs for multi-document corpus extraction (extract_corpus, load_corpus), process sandboxing controls (sandbox=False fast path vs sandbox=True), selective bundle page rendering (render_pages="vision_only"), normalized entity key rationalization (Key(normalizer="alnum_upper")), non-disputing Informational fields, and Rust regex dependency upgrades (PR #494, PR #496).
Key Highlights in Datagrunt 5.3.12
- Multi-Document PDF Corpora (
extract_corpus&load_corpus) — Scale PDF extraction across thousands of documents using parallel worker pools, Graham LPT (largest-file-first) scheduling, SHA-256 incremental bundle caching, and unifiedCorpusManifestmanagement. - Flexible Execution & Process Sandboxing (
sandbox=False/sandbox=True) — Zero-subprocess fast path (sandbox=False,workers=1) for high-throughput digital PDF parsing, alongside process-isolated sandboxing (sandbox=True) executing page extraction in memory-capped (2 GiBRLIMIT_AS)python -Iworkers. - Selective Page Rendering (
render_pages="vision_only") — Avoid rendering digital text pages to disk while automatically generating page PNGs forvisionandimage_onlypages needed by multimodal VLMs. - Normalized Identity Keys & Non-Disputing Fields — Normalize entity identity formats (
Key("part", normalizer="alnum_upper")) to fold variation across pages (e.g.,"04-60008-27"vs"046000827") while preserving raw verbatim quote validation, and declare non-disputing descriptive metadata usingAnnotated[..., Informational]. - Grounded-Union Result Tiers & Cloud Batching — Query strict consensus records (
result.records) or high-recall grounded union records (result.grounded_records,to_dataframe(include_grounded_union=True)), and run asynchronous cloud batches viaprepare_batch()andingest_batch(). - Rust Regex Dependency Bump (
fancy-regex0.20.2) — Updatedfancy-regexin/rustto 0.20.2 for improved pattern matching security and performance in the native Rust core.
1. Multi-Document Corpora & Selective Bundle Rendering
Datagrunt 5.3.12 streamlines batch extraction for large document repositories. Instead of orchestrating individual PDFReader calls, extract_corpus processes directories or lists of PDFs in parallel:
from datagrunt import extract_corpus, load_corpus, rationalize
# 1. Parallel extraction across multi-core workers with vision-only page rendering
corpus = extract_corpus(
inputs="pdfs/2026-reports/",
output_dir="corpus_output/",
max_workers=8,
render_pages="vision_only", # Renders page PNGs only for scanned/vision pages
ocr="auto",
)
print(f"Extracted {corpus.document_count} documents with {len(corpus.failures)} failures.")
# 2. Reload existing corpus manifest incrementally
reloaded_corpus = load_corpus("corpus_output/")Selective Page Rendering (render_pages="vision_only")
When generating document bundles or building a corpus, clean digital text pages do not require pixel-based PNG renders. Passing render_pages="vision_only" to write_bundle() or extract_corpus() skips rendering clean text pages, saving storage and CPU cycles while guaranteeing that scanned or low-confidence OCR pages carry crisp page images for multimodal VLM routing.
2. In-Process Fast Path vs. Process Sandboxing
Datagrunt 5.3.12 clarifies the execution modes for PDFReader:
- In-Process Fast Path (
sandbox=False,workers=1): Default execution mode. Executes PDFium calls directly within your Python process under a re-entrant lock, eliminating cold process spawn latency and IPC serialization overhead. - Process Sandboxing (
sandbox=Trueorworkers > 1): Launches page extraction and rendering tasks inside isolatedpython -Ichild processes capped at 2 GiB virtual memory (RLIMIT_AS). Bad, corrupt, or malicious PDF structures are safely isolated without affecting the host application.
from datagrunt import PDFReader
# In-process fast path (zero subprocess overhead)
reader = PDFReader("clean_digital_report.pdf", sandbox=False)
df = reader.to_dataframe()
# Process-isolated sandboxed extraction for untrusted external uploads
sandboxed_reader = PDFReader("untrusted_upload.pdf", sandbox=True, workers=4)
sandboxed_df = sandboxed_reader.to_dataframe()3. Schema-Driven Multimodal Rationalization
The rationalize() pipeline extracts typed Pydantic models across single PDFs, bundles, or multi-document CorpusManifest objects (up to 10,000 targets per invocation).
from decimal import Decimal
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, load_corpus, rationalize
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth
# Define schema with normalized identity keys and informational fields
class ComponentSpec(BaseModel):
# Alphanumeric upper normalizer matches "04-60008-27" and "046000827" to the same entity
part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
category: str
specs: Annotated[list[str], Informational] = [] # Non-disputing list field
notes: Annotated[str | None, Informational] = None # Non-disputing free-text field
unit_price: Decimal | None = None
corpus = load_corpus("corpus_output/")
auth = GoogleCloudAuth()
result = rationalize(
corpus,
schema=ComponentSpec,
prompt="Extract all component part numbers, categories, specifications, and pricing.",
models=(
GeminiProvider.google_cloud("gemini-2.5-pro", project="my-gcp-project", auth=auth),
AnthropicProvider.google_cloud("claude-opus-4-6", project="my-gcp-project", auth=auth),
),
min_families=2, # Requires consensus across >= 2 model families
cache_dir=".dg-cache", # Sharded atomic response cache
)
# Export strict consensus records
strict_df = result.to_dataframe()
# Export grounded union tier (includes grounded single-family discoveries)
union_df = result.to_dataframe(include_grounded_union=True)4. Grounded-Union Result Tiers & Offline Batches
Datagrunt 5.3.12 structures rationalization results into clear operational tiers:
result.records: Strict clean consensus tier (status in ("consensus", "single_model")).result.grounded_records: Grounded-union tier ("consensus","single_model", and element-grounded"missing_in_some_models"items with zero grounding errors or field disputes).result.needs_review: Flagged items ("field_disagreement","ungrounded_value","cross_page_conflict") complete with source element bounding-box provenance.
For asynchronous high-volume jobs, prepare_batch() generates cloud batch JSONL payloads for Vertex AI and Anthropic Message Batches, which can be re-assembled into a RationalizedResult via ingest_batch().
Verification & Summary
All version bumps (5.3.12) across pyproject.toml, src/datagrunt/__init__.py, and rust/Cargo.lock have been committed and validated.
- Docs Index Updated:
content/docs/_index.md - PDF Parsing Guide Updated:
content/docs/guides/pdf-parsing.md - Release Blog Post Published:
content/blog/2026/november/datagrunt-5-3-12-corpus-sandbox-and-rationalization-docs.md