Datagrunt 5.2: Two-Stage Batch Corpus Extraction, Section Coalescing & Gemini Thinking Levels

Datagrunt 5.2.0 is now officially available! Building upon the schema-driven PDF rationalization engine introduced in 5.1, version 5.2 delivers high-throughput enterprise batch document processing capabilities, greedy section coalescing for prompt optimization, and advanced reasoning controls for Google Gemini models.

What’s New in Datagrunt 5.2.0

1. Two-Stage Batch Corpus Extraction (extract_corpus & load_corpus) — Batch-extract directories or lists of PDF files into verified artifact bundles and index them in corpus_manifest.json.
2. Pure-Stdlib Manifest & Bundle Loaders — load_corpus(), load_bundle(), CorpusManifest, CorpusEntry, and CorpusFailure are decoupled into stdlib/core modules. Reading and indexing pre-extracted document bundles requires zero optional dependencies (pypdfium2 or Pillow are not required).
3. Greedy Section Coalescing (coalesce_sections=True) — Consolidates consecutive text-bearing elements across document section boundaries, maximizing LLM context window efficiency while maintaining exact quote grounding.
4. Gemini Thinking Level Configuration (thinking_level) — Fine-tune reasoning depth ("LOW", "MEDIUM", "HIGH") on Gemini 2.0 / 3 models via GeminiProvider or model strings.

1. Two-Stage Batch Corpus Extraction

Large-scale data engineering pipelines often process thousands of PDF documents in batches. Parsing heavy binary PDF layouts directly inside downstream analytical or LLM worker tasks can consume excessive memory and slow down pipelines.

Datagrunt 5.2 introduces a formal Two-Stage Architecture:

• Stage 1 (Heavy Parsing & Bundling): Extract raw PDFs in parallel using isolated multi-worker processes, saving structured JSON/Markdown artifacts into standardized bundle directories.
• Stage 2 (Lightweight Pure-Stdlib Loading & Processing): Downstream worker nodes or serverless tasks load pre-parsed bundles using standard Python libraries, eliminating heavy C++ library bindings (pypdfium2) or image processing binaries (Pillow).

Batch Extraction Example

from datagrunt import extract_corpus, load_corpus

# Stage 1: Batch-extract incoming PDFs into verified corpus bundles
manifest = extract_corpus(
    "data/inbound_invoices/",
    output_dir="data/corpus_bundles/",
    overwrite=False,     # Incremental skip: reuse existing valid bundles (reused=True)
    strict=False,        # Fault isolation: unreadable PDFs are recorded in manifest.failures
    workers=8,           # Parallel execution across worker threads (up to 64)
)

print(f"Extracted {len(manifest.entries)} documents successfully.")
print(f"Reused {sum(1 for e in manifest.entries if e.reused)} existing bundles.")

if manifest.failures:
    print(f"Failed to extract {len(manifest.failures)} documents:")
    for failure in manifest.failures:
        print(f"  - {failure.source_path}: [{failure.error_type}] {failure.message}")

# Stage 2: Fast, zero-dependency loading of pre-extracted corpus bundles
corpus = load_corpus("data/corpus_bundles/")

for doc_id, document in corpus.items():
    print(f"Doc ID: {doc_id} | Pages: {document.page_count} | Headings: {len(document.headings)}")

Key Corpus Features

• Incremental Reuse: When overwrite=False, Datagrunt computes SHA-256 hashes of input files. Unchanged PDFs are skipped, flagging reused=True in the corpus entry.
• Fault Isolation: With strict=False, bad or corrupt PDFs register a CorpusFailure entry without terminating the batch runner.
• Zero Heavy Dependencies for Readers: load_corpus() and load_bundle() execute purely using standard library json, hashlib, and os modules.

2. Section Coalescing in rationalize()

When extracting structured schemas from lengthy PDF documents with rationalize(), strict section boundaries previously split prompts into smaller extraction units.

Datagrunt 5.2 introduces Greedy Section Coalescing (coalesce_sections=True), which packs consecutive text-bearing elements across heading and section boundaries into larger unified extraction units:

from datagrunt import PDFReader, rationalize
from pydantic import BaseModel, Field

class FinancialReport(BaseModel):
    revenue: float = Field(description="Total reported revenue")
    net_income: float = Field(description="Net income after tax")

doc = PDFReader("financial_report.pdf").document

# Rationalize with greedy section coalescing enabled by default
result = rationalize(
    doc,
    schema=FinancialReport,
    prompt="Extract key financial metrics from the document.",
    models=("openai:gpt-5.4", "anthropic:claude-sonnet-4-6"),
    coalesce_sections=True,  # Default: packs text across section boundaries
)

Why Section Coalescing Matters

• Reduced API Calls: Merging sections reduces the total number of wire requests sent to AI models.
• Preserved Context: Relevant context spanning adjacent sections is presented together, improving model comprehension.
• Verbatim Grounding Integrity: Even with coalesced sections, Datagrunt maintains line-level element tracking and quote verification for every extracted value.

3. Gemini Thinking Level Configuration

Google’s Gemini 2.0 and 3 reasoning models support explicit thinking depth controls. Datagrunt 5.2 adds first-class support for thinking_level across both direct Gemini API (generativelanguage.googleapis.com) and Google Cloud Vertex AI endpoints.

Using thinking_level in Model Strings or Provider Instances

from datagrunt import PDFReader, rationalize
from datagrunt.rationalization.providers.gemini import GeminiProvider

doc = PDFReader("complex_contract.pdf").document

# Option 1: Via model query parameter string
result = rationalize(
    doc,
    schema=MySchema,
    prompt="Analyze contract terms.",
    models=("gemini:gemini-2.5-flash?thinking_level=HIGH",),
)

# Option 2: Programmatically instantiating GeminiProvider
provider = GeminiProvider(
    "gemini-2.5-flash",
    api_key="YOUR_GEMINI_KEY",
    thinking_level="HIGH",  # Options: "LOW", "MEDIUM", "HIGH"
)

Setting thinking_level attaches {"thinkingConfig": {"thinkingLevel": "HIGH"}} to generationConfig on the wire, directing Gemini models to perform deeper reasoning before outputting structured JSON responses.

Summary of Updated & Created Files

• src/datagrunt/pdf/corpus.py — Added extract_corpus(), load_corpus(), CorpusManifest, CorpusEntry, CorpusFailure.
• src/datagrunt/pdf/_bundle_load.py — Decoupled stdlib/core bundle reader functions (load_bundle()) from heavy rendering dependencies.
• src/datagrunt/rationalization/units.py & api.py — Added greedy _Coalescer and coalesce_sections option to rationalize().
• src/datagrunt/rationalization/providers/gemini.py — Added thinking_level configuration to GeminiProvider.
• datagrunt-site/content/docs/_index.md — Updated stable release notice to v5.2.0 and highlighted pure-stdlib corpus loading.
• datagrunt-site/content/docs/guides/pdf-parsing.md — Added documentation guide for Two-Stage Batch Corpus Extraction and Section Coalescing.
• datagrunt-site/content/blog/2026/november/datagrunt-5-2-0-two-stage-batch-corpus-extraction.md — Published release blog post.

Upgrading to Datagrunt 5.2.0

Upgrade Datagrunt via uv:

uv pip install --upgrade "datagrunt[pdf,rationalize]"