Datagrunt 5.3.12: Multi-Document Corpora, Process Sandboxing & Multimodal Rationalization

November 16, 2026by Martin Graham

Datagrunt 5.3.12 is now available! This release brings comprehensive documentation updates and refined APIs for multi-document corpus extraction (extract_corpus, load_corpus), process sandboxing controls (sandbox=False fast path vs sandbox=True), selective bundle page rendering (render_pages="vision_only"), normalized entity key rationalization (Key(normalizer="alnum_upper")), non-disputing Informational fields, and Rust regex dependency upgrades (PR #494, PR #496).


Key Highlights in Datagrunt 5.3.12

  1. Multi-Document PDF Corpora (extract_corpus & load_corpus) — Scale PDF extraction across thousands of documents using parallel worker pools, Graham LPT (largest-file-first) scheduling, SHA-256 incremental bundle caching, and unified CorpusManifest management.
  2. Flexible Execution & Process Sandboxing (sandbox=False / sandbox=True) — Zero-subprocess fast path (sandbox=False, workers=1) for high-throughput digital PDF parsing, alongside process-isolated sandboxing (sandbox=True) executing page extraction in memory-capped (2 GiB RLIMIT_AS) python -I workers.
  3. Selective Page Rendering (render_pages="vision_only") — Avoid rendering digital text pages to disk while automatically generating page PNGs for vision and image_only pages needed by multimodal VLMs.
  4. Normalized Identity Keys & Non-Disputing Fields — Normalize entity identity formats (Key("part", normalizer="alnum_upper")) to fold variation across pages (e.g., "04-60008-27" vs "046000827") while preserving raw verbatim quote validation, and declare non-disputing descriptive metadata using Annotated[..., Informational].
  5. Grounded-Union Result Tiers & Cloud Batching — Query strict consensus records (result.records) or high-recall grounded union records (result.grounded_records, to_dataframe(include_grounded_union=True)), and run asynchronous cloud batches via prepare_batch() and ingest_batch().
  6. Rust Regex Dependency Bump (fancy-regex 0.20.2) — Updated fancy-regex in /rust to 0.20.2 for improved pattern matching security and performance in the native Rust core.

1. Multi-Document Corpora & Selective Bundle Rendering

Datagrunt 5.3.12 streamlines batch extraction for large document repositories. Instead of orchestrating individual PDFReader calls, extract_corpus processes directories or lists of PDFs in parallel:

from datagrunt import extract_corpus, load_corpus, rationalize

# 1. Parallel extraction across multi-core workers with vision-only page rendering
corpus = extract_corpus(
    inputs="pdfs/2026-reports/",
    output_dir="corpus_output/",
    max_workers=8,
    render_pages="vision_only",  # Renders page PNGs only for scanned/vision pages
    ocr="auto",
)

print(f"Extracted {corpus.document_count} documents with {len(corpus.failures)} failures.")

# 2. Reload existing corpus manifest incrementally
reloaded_corpus = load_corpus("corpus_output/")

Selective Page Rendering (render_pages="vision_only")

When generating document bundles or building a corpus, clean digital text pages do not require pixel-based PNG renders. Passing render_pages="vision_only" to write_bundle() or extract_corpus() skips rendering clean text pages, saving storage and CPU cycles while guaranteeing that scanned or low-confidence OCR pages carry crisp page images for multimodal VLM routing.


2. In-Process Fast Path vs. Process Sandboxing

Datagrunt 5.3.12 clarifies the execution modes for PDFReader:

  • In-Process Fast Path (sandbox=False, workers=1): Default execution mode. Executes PDFium calls directly within your Python process under a re-entrant lock, eliminating cold process spawn latency and IPC serialization overhead.
  • Process Sandboxing (sandbox=True or workers > 1): Launches page extraction and rendering tasks inside isolated python -I child processes capped at 2 GiB virtual memory (RLIMIT_AS). Bad, corrupt, or malicious PDF structures are safely isolated without affecting the host application.
from datagrunt import PDFReader

# In-process fast path (zero subprocess overhead)
reader = PDFReader("clean_digital_report.pdf", sandbox=False)
df = reader.to_dataframe()

# Process-isolated sandboxed extraction for untrusted external uploads
sandboxed_reader = PDFReader("untrusted_upload.pdf", sandbox=True, workers=4)
sandboxed_df = sandboxed_reader.to_dataframe()

3. Schema-Driven Multimodal Rationalization

The rationalize() pipeline extracts typed Pydantic models across single PDFs, bundles, or multi-document CorpusManifest objects (up to 10,000 targets per invocation).

from decimal import Decimal
from typing import Annotated
from pydantic import BaseModel
from datagrunt import Informational, Key, load_corpus, rationalize
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, GoogleCloudAuth

# Define schema with normalized identity keys and informational fields
class ComponentSpec(BaseModel):
    # Alphanumeric upper normalizer matches "04-60008-27" and "046000827" to the same entity
    part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
    category: str
    specs: Annotated[list[str], Informational] = []     # Non-disputing list field
    notes: Annotated[str | None, Informational] = None  # Non-disputing free-text field
    unit_price: Decimal | None = None

corpus = load_corpus("corpus_output/")
auth = GoogleCloudAuth()

result = rationalize(
    corpus,
    schema=ComponentSpec,
    prompt="Extract all component part numbers, categories, specifications, and pricing.",
    models=(
        GeminiProvider.google_cloud("gemini-2.5-pro", project="my-gcp-project", auth=auth),
        AnthropicProvider.google_cloud("claude-opus-4-6", project="my-gcp-project", auth=auth),
    ),
    min_families=2,         # Requires consensus across >= 2 model families
    cache_dir=".dg-cache",  # Sharded atomic response cache
)

# Export strict consensus records
strict_df = result.to_dataframe()

# Export grounded union tier (includes grounded single-family discoveries)
union_df = result.to_dataframe(include_grounded_union=True)

4. Grounded-Union Result Tiers & Offline Batches

Datagrunt 5.3.12 structures rationalization results into clear operational tiers:

  • result.records: Strict clean consensus tier (status in ("consensus", "single_model")).
  • result.grounded_records: Grounded-union tier ("consensus", "single_model", and element-grounded "missing_in_some_models" items with zero grounding errors or field disputes).
  • result.needs_review: Flagged items ("field_disagreement", "ungrounded_value", "cross_page_conflict") complete with source element bounding-box provenance.

For asynchronous high-volume jobs, prepare_batch() generates cloud batch JSONL payloads for Vertex AI and Anthropic Message Batches, which can be re-assembled into a RationalizedResult via ingest_batch().


Verification & Summary

All version bumps (5.3.12) across pyproject.toml, src/datagrunt/__init__.py, and rust/Cargo.lock have been committed and validated.