Datagrunt 5.4.0: Multi-Pass rationalize_many, Truncation Bisection & High-Concurrency Engine
Datagrunt 5.4.0 is now officially available! This major feature release delivers the Cloud-Agnostic PDF Rationalization Roadmap (PR #498), introducing multi-schema extraction (rationalize_many()), automatic unit bisection on LLM output truncation, first-class provenance JSONL exports, 256 in-flight concurrency scaling, live telemetry streaming, and sub-millisecond hot-path speedups across consensus and grounding engines.
Highlights in Datagrunt 5.4.0
- Multi-Schema
rationalize_many()&PassSpec— Extract multiple Pydantic schemas in a single pass over a loaded PDF corpus using shared provider HTTP pools, unified rate limiters, and isolated per-pass caching without redundant bundle verification. - Automatic Truncation Bisection (
split_on_truncation=True) — Automatically bisects multi-element extraction units, multi-row tables, and bullet lists at their midpoint (up to 16 rounds) whenever an LLM response hits token limit truncation (finish="truncated"). - First-Class Provenance-by-Default JSONL Export (
to_jsonl()) — Export atomic UTF-8 JSONL with full attribution fields (_doc_id,_source_name,_status,_grounding_quotes,_disputed_fields,_sources) and pure JSON-serializable telemetry dicts (result.telemetry.to_dict()). - 256 In-Flight Concurrency & Live Telemetry (
event_delivery="live") — Scale concurrency up toCONCURRENCY_LIMIT = 256with 64-socket TLS keep-alive connection pools per provider and live event streaming without head-of-line blocking. - Textless Scan Grounding via Cross-Family Agreement — Pure textless scan items backed by $\ge \max(2, min_families)$ distinct model families with substantive field agreement waive peer-quote string overlap while maintaining strict key verification.
- Sub-Millisecond Hot-Path Speedups — Up to 71.5x faster unkeyed table consensus, 7.2x faster table grounding, 2.9x faster reply JSON parsing, and 2.6x–10x faster corpus loading and preflight reuse.
1. Multi-Schema Extraction: rationalize_many() & PassSpec
When extracting complex PDF documents (such as engineering packages, invoices, or medical records), you often need to extract multiple distinct schemas (e.g. document classification headers alongside granular bill-of-materials line items). Previously, running separate rationalize() calls incurred repeated disk reads, redundant bundle verification, and uncoordinated HTTP connection pools.
Datagrunt 5.4.0 introduces rationalize_many() and PassSpec:
from pydantic import BaseModel
from typing import Annotated
from datagrunt import Key, PassSpec, load_corpus, rationalize_many
# Schema 1: High-level document classification
class DocProfile(BaseModel):
drawing_number: Annotated[str, Key("drawing")]
title: str
revision: str
# Schema 2: Granular bill-of-materials parts list
class PartMention(BaseModel):
part_id: Annotated[str, Key("part", normalizer="alnum_upper")]
qty: int
description: str
# Load corpus documents once
corpus = load_corpus("engineering_drawings_corpus/")
# Execute multiple schemas across the corpus with shared runner infrastructure
results = rationalize_many(
corpus,
passes=[
PassSpec(
schema=DocProfile,
prompt="Extract drawing title and revision header.",
name="profile",
),
PassSpec(
schema=PartMention,
prompt="Extract all BOM part line items.",
name="parts",
max_output_tokens=16_384,
),
],
models=["openai:gpt-5.4", "anthropic:claude-sonnet-4-6"],
min_families=2,
max_concurrency=64,
event_delivery="live", # Stream telemetry events in arrival order
cache_dir=".cache/rationalize",
)
# Access individual pass RationalizedResult objects
results["profile"].to_jsonl("output/doc_profiles.jsonl")
results["parts"].to_jsonl("output/part_mentions.jsonl")Technical Benefits of Multi-Pass Execution:
- 1x Corpus Loading (
load_corpus_documents): Manifest bundles and PDF documents are loaded and verified once across all extraction passes usinglstat-stamp validation. - Shared Connection Pools & Rate Limiters: One provider HTTP connection pool and unified per-model token/RPM rate limiters are shared across all passes.
- Isolated Per-Pass Caching: Each
PassSpecmaintains an isolatedCacheMetanamespace so cache keys never collide between passes.
2. Automatic Truncation Bisection (split_on_truncation=True)
When extracting large tables or long lists from PDF documents, provider output token limits (e.g., 4,096 or 8,192 tokens) can cause LLM responses to be cut off mid-output (finish="truncated"), losing trailing table rows or list items.
In Datagrunt 5.4.0, split_on_truncation=True is enabled by default in both rationalize() and rationalize_many(). When a provider response encounters output truncation, Datagrunt’s unit scheduler (bisect_unit()) automatically bisects multi-element extraction units and multi-row Table or BulletList elements at their exact midpoint:
from datagrunt import PDFReader, rationalize
doc = PDFReader("massive_parts_catalog.pdf").document
# If an extraction unit hits provider token limits, it bisects automatically up to 16 rounds
result = rationalize(
doc,
schema=PartMention,
prompt="Extract every single part number in this catalog.",
models=["gemini:gemini-2.5-flash", "anthropic:claude-3-5-sonnet-20241022"],
split_on_truncation=True, # Enabled by default
)
print(f"Extracted {len(result.records)} records cleanly across {result.telemetry.total_units} units.")How Truncation Bisection Works:
- Midpoint Splitting: Splittable elements (
Table,BulletList) and multi-element units are divided into two equal halves. - Deterministic Part IDs: Bisected units generate collision-free sub-unit IDs with
(part_index, part_count)suffixes (e.g.unit_001_p1_of_2,unit_001_p2_of_2). - Iterative Resolution: Bisection executes up to
MAX_TRUNCATION_ROUNDS = 16rounds before consensus, quote grounding, and entity folding, preserving parent token counts and cache telemetry.
3. First-Class Provenance-by-Default Exports & Telemetry
Datagrunt 5.4.0 makes result provenance first-class across JSONL exports and runtime telemetry.
Atomic JSONL Export (to_jsonl())
The new RationalizedResult.to_jsonl() method exports atomic UTF-8 JSONL containing strict records, grounded union records, disputed items, and complete attribution metadata:
result = rationalize(doc, schema=PartMention, prompt="Extract parts.")
# Write full provenance JSONL (includes _doc_id, _source_name, _status, _sources, _grounding_quotes)
count = result.to_jsonl(
"out/parts_provenance.jsonl",
include_provenance=True,
include_disputed=True,
)
print(f"Wrote {count} audit records to JSONL.")Each JSONL record includes first-class metadata attributes:
_doc_id&_source_name: Document identifier and original file name._status: Consensus status ("consensus","single_model","missing_in_some_models", or"disputed")._is_grounded&_is_grounded_union: Grounding state flags._supporting_models&_grounding_quotes: Model citations and verified text quotes._sources: Page numbers, element IDs, and exact bounding boxes (bbox) for auditability.
Structured Telemetry (result.telemetry.to_dict())
Export complete run stats, unit counters, and per-model latency/token usage as pure Python dictionaries ready for stdlib json.dumps():
import json
telemetry_data = result.telemetry.to_dict()
print(json.dumps(telemetry_data, indent=2))
# Output includes:
# - total_units, cache_hits, cache_misses, skipped_units
# - model_usage: prompt_tokens, completion_tokens, latency_ms per model4. High-Concurrency Engine Scaling & Live Telemetry
To support high-volume document extraction pipelines, Datagrunt 5.4.0 significantly boosts concurrent execution throughput and streaming delivery:
CONCURRENCY_LIMIT = 256: In-flight request concurrency caps have been increased from 64 to 256 acrossRunLimitsand provider wrappers.- Pooled 64-Socket TLS Keep-Alive Transports: Expanded HTTP connection pools from 8 to 64 sockets per provider with 60-second TLS keep-alive (
KEEPALIVE_EXPIRY_S = 60.0), eliminating connection teardowns during extended model thinking pauses. - Live Telemetry Events (
event_delivery="live"): Passingevent_delivery="live"streams completion callbacks instantly as units finish, preventing fast units from waiting on slow stragglers. - Vertex AI Quota Isolation: Added per-
(project, location, publisher, model)rate keys and lock-free bearer token caching (_CachedBearer), ensuring high-volume Claude workloads on Vertex AI never starve Gemini models.
5. Hot-Path Concurrency Multipliers & Latency Speedups
Datagrunt 5.4.0 delivers massive performance improvements across hot-path execution routines while preserving 100% bit-identical extraction invariants:
| Execution Engine / Layer | Optimization Strategy | Measured Speedup |
|---|---|---|
| Unkeyed Consensus | FIFO score == 1.0 pre-pass before fuzzy candidate postings |
227.55 ms -> 3.18 ms (71.5x faster) on 200-row tables |
| BOM Table Grounding | Pre-computed lazy per-element word_set & joined_plain indices |
254.67 ms -> 35.13 ms (7.2x faster) on 2,000-cell tables |
| Reply JSON Parsing | Single-walk parsed_response with UTF-8 byte length & ASCII fast-paths |
199.99 ms -> 69.04 ms (2.9x faster) on 200-row structured replies |
| CAD Quality & Gate Classifier | x0-sorted _GlyphBand split, _gate_selection bisection, ASCII fast-path |
2.5x / 2.4x faster on dense CAD sheets |
| Preflight & Corpus Loading | 16-thread doc packing, cross-pass cache prewarm, manifest-first reuse | 2.6x–10x faster preflight & incremental corpus loading |
6. Textless Scan Grounding via Cross-Family Agreement
On textless scanned document pages (image_only scans with ocr="never"), visible elements contain no extracted text strings for verbatim quote matching. In previous releases, this required vision models to rely solely on peer quote superset matching.
Datagrunt 5.4.0 introduces Cross-Family Field Agreement for textless scans. When pure textless_anchor consensus items are backed by $\ge \max(2, min_families)$ distinct known model families with substantive structured field agreement, Datagrunt waives peer-quote string overlap while strictly enforcing key verification (key_not_quoted) and low-entropy safeguards to prevent false positives.
Upgrading to Datagrunt 5.4.0
To upgrade your environment, run:
uv pip install --upgrade "datagrunt[pdf,rationalize]"Or with standard pip:
pip install --upgrade "datagrunt[pdf,rationalize]"For full details on the new APIs, check out the updated PDF Parsing & Rationalization Guide and API Reference.