Datagrunt 5.2.2: Pure-Rust PDF Compute Kernels, Worker Pool Reuse & Offline Cloud Batch Mode

Datagrunt 5.2.2 is now officially available! This feature-packed release accelerates Stage 1 PDF parsing and quality assessment with pure-Rust compute kernels, introduces process worker pool reuse and image extraction skipping for extract_corpus(), and adds full Offline Cloud Batch Mode (prepare_batch & ingest_batch) for 50% LLM token cost savings.

Key Highlights in Datagrunt 5.2.2

1. Pure-Rust PDF Compute Kernels (_native) — Core PDF layout, segment cover trees, backdrop quality sampling, and LZW stream decoding algorithms now execute in native Rust with PyO3 GIL detachment (py.detach).
2. Stage 1 Worker Pool Reuse — extract_corpus() reuses a unified PageWorkerPool across the entire document batch, eliminating process spawn overhead per PDF file.
3. Selective Bundle Image Skipping (include_images=False, render_pages=False) — extract_corpus() and write_bundle() add fine-grained flags to skip embedded figure extraction and page image rendering for maximum speed and storage efficiency.
4. Offline Cloud Batch Mode (prepare_batch & ingest_batch) — Prepare batch plans offline without network calls, export request files to Vertex AI (.to_vertex_jsonl) or Anthropic Message Batches (.to_anthropic_jsonl), and ingest completed cloud batch results into Datagrunt’s consensus and grounding pipeline.

1. Pure-Rust PDF Compute Kernels

Stage 1 PDF extraction and layout parsing involve compute-intensive geometry calculations and backdrop image sampling. In Datagrunt 5.2.2, five key compute functions have been ported directly into datagrunt-core::pdf in pure Rust with PyO3 bindings:

• pdf_char_bgr_medians — Performs backdrop pixel sampling and BGR lower median calculation across text bounding boxes for character visibility evaluation.
• pdf_union_area — Calculates exact bounding box union area using a sweep-line algorithm paired with a dynamic segment cover tree.
• pdf_char_reason_runs — Groups characters into contiguous run-length sequences categorized by off-page and invisible text reasons.
• pdf_snap_and_merge_segments — Snaps and merges vector graphic line segments along page axes to discover table lattices.
• pdf_lzw_decode — Decodes PDF LZW compressed streams in Rust with chunked output buffers and maximum byte caps.

High-Concurrency GIL Detachment

All native PDF compute kernels execute inside py.detach(|| ...) blocks. By releasing Python’s Global Interpreter Lock (GIL) during heavy mathematical and bitmap computations, multi-threaded worker pools in extract_corpus() achieve near-linear multi-core CPU scaling.

from datagrunt.pdf._compute import backend

# High-performance native Rust PDF compute dispatch
active_backend = backend()  # datagrunt._native when Rust is active

2. Worker Pool Reuse & Bundle Image Skipping in extract_corpus()

Process Worker Pool Reuse

When extracting large batches of PDF files with extract_corpus(), prior versions spawned fresh worker processes for every single PDF. Datagrunt 5.2.2 wraps batch processing in a single, persistent PageWorkerPool, reusing worker processes across all files in the batch.

Selective Image Skipping

For pipelines that only require structured text, Markdown, or tabular data, rendering page images or extracting high-resolution embedded figures adds unnecessary I/O and storage overhead.

Datagrunt 5.2.2 adds include_images=False and render_pages=False parameters to extract_corpus() and write_bundle():

from datagrunt import extract_corpus

# Extract text and layout without storing figure images or rendering page PNGs
manifest = extract_corpus(
    "incoming_pdfs/",
    output_dir="corpus_bundles/",
    workers=16,
    include_images=False,  # Skip saving embedded figure PNGs
    render_pages=False,    # Skip rendering full-page backdrop PNGs
)

print(f"Extracted {len(manifest.entries)} documents with reduced disk footprint.")

3. Offline Cloud Batch Mode (prepare_batch & ingest_batch)

Running real-time online requests against LLM provider APIs (rationalize()) can incur high token costs and encounter rate-limit (RPM/TPM) bottlenecks. Datagrunt 5.2.2 introduces Offline Cloud Batch Mode via prepare_batch() and ingest_batch().

Step 1: Prepare the Batch Plan

prepare_batch() executes zero-I/O preflight checks, target resolution, unit packing, prompt envelope rendering, and FileCache probing. It returns a BatchPlan containing all uncached BatchItem requests without calling any network APIs.

from datagrunt import prepare_batch
from pydantic import BaseModel, Field

class ContractSchema(BaseModel):
    vendor: str = Field(description="Vendor name")
    effective_date: str = Field(description="Contract effective date")
    total_value: float = Field(description="Total contract monetary value")

# Prepare uncached requests offline
plan = prepare_batch(
    "corpus_bundles/",
    schema=ContractSchema,
    prompt="Extract key contract terms.",
    models=["gemini:gemini-2.5-flash", "anthropic:claude-3-5-sonnet-20241022"],
    min_families=2,
    cache_dir=".cache/rationalize",
)

print(f"Uncached items: {len(plan.items)}")
print(f"Cached hits: {len(plan.cached_hits)}")

Step 2: Export Cloud JSONL Files

Export the prepared batch requests directly into standard batch formats for Vertex AI (Gemini) or Anthropic Message Batches:

# Export to Google Cloud Vertex AI batch prediction format
plan.to_vertex_jsonl("batch_requests/vertex_gemini.jsonl")

# Export to Anthropic Message Batches format
plan.to_anthropic_jsonl("batch_requests/anthropic_claude.jsonl")

Submit these JSONL files to cloud batch prediction APIs using cloud CLIs (gcloud or Anthropic API). Cloud batch endpoints provide a 50% discount on token costs and operate asynchronously without consuming online API quota limits.

Step 3: Ingest Completed Responses

Once the cloud batch jobs complete, download the result JSONL files and pass them to ingest_batch():

from datagrunt import ingest_batch

# Ingest results and run consensus, grounding, and folding
result = ingest_batch(
    plan,
    responses=[
        "batch_results/vertex_gemini_output.jsonl",
        "batch_results/anthropic_claude_output.jsonl",
    ],
)

# Export identical RationalizedResult as rationalize()
df = result.to_dataframe(flatten="items")
print(f"Successfully processed {len(result.documents)} documents into DataFrames.")

ingest_batch() runs the exact same cross-family consensus (min_families), verbatim quote grounding, cross-page entity folding, and persistent file caching as standard rationalize() calls.

Upgrading to Datagrunt 5.2.2

Upgrade to Datagrunt 5.2.2 today via uv or pip:

uv pip install --upgrade "datagrunt[pdf,rationalize]"

Or using standard pip:

pip install --upgrade "datagrunt[pdf,rationalize]"