Datagrunt 5.3.10: Wave 2B Pooled HTTP Keep-Alive, Single-Pass Cache, Min-Heap Runner & Streaming Batch

Datagrunt 5.3.10 is now available! This release delivers Wave 2B performance optimizations across the rationalization structured extraction subsystem (src/datagrunt/rationalization/), accelerating provider communication, request key hashing, scheduler dispatching, offline batch processing, and envelope rendering (PR #492).

Key Highlights in Datagrunt 5.3.10

1. Pooled HTTP Keep-Alive & Transport Fault Retirement (_BoundConnection & Client Retirement) — Retains pre-TLS plain socket.socket duplicates across pooled HTTP/1.1 connections so reused connections can still be cut at exact deadlines (keepalive_connections > 0), while automatically retiring failed transport clients on network/TLS errors to prevent httpcore 1.0.9 pool stalls.
2. Single-Pass Fast Request Key & Unified Preflight (_fast_request_key & _WorkAccumulator) — Pre-serializes canonical provider suffixes (_provider_key_suffix) and envelope prefix states (_envelope_key_prefix), using copyable SHA-256 state (hashlib.sha256().copy()) in _fast_request_key to compute bit-identical request keys without canonical JSON dict/string overhead. _WorkAccumulator unifies document expansion, unit packing, envelope generation, key hashing, and cache probing into a single pass.
3. Min-Heap Rate-Key Scheduler (heapq Min-Heap) — Replaces linear queue scanning in _RequestScheduler with a heapq min-heap across active rate key queues, reducing request dispatch overhead from $O(K)$ to $O(\log K)$ under high concurrency.
4. Streaming Batch Preparation & Ingestion (to_vertex_jsonl, to_anthropic_jsonl, _read_bounded_jsonl) — Streams Vertex AI (generateContent) and Anthropic Message Batches JSONL requests line-by-line (to_vertex_jsonl, to_anthropic_jsonl) and ingests batch responses via bounded binary streams (_read_bounded_jsonl), providing $O(1)$ memory usage with strict 16 MiB line and 1,000,000 line bounds.
5. Shared Frame & Band Cache Envelope Isolation (build_envelopes_isolated) — Prepares shared document frames, prompt text, context JSON, and _BandCache once per document during non-strict envelope generation, isolating per-unit PDF rendering errors without redundant re-parsing.
6. Ephemeral Provider Lifecycle Cleanup (_close_ephemeral) — Automatically tracks and executes close() hooks for string-resolved ephemeral model providers on preflight failure or normal termination without masking active exceptions.

1. Pooled HTTP Keep-Alive & Transport Fault Retirement

The Challenge of Connection Reuse & Deadline Cuts

When sending high volumes of online extraction requests to LLM endpoints, establishing fresh TCP and TLS handshakes on every request introduces significant network overhead. However, retaining pooled HTTP connections introduces two challenges:

1. Reused pooled connections must remain strictly bound to request deadline timers so hung servers cannot block worker threads past their unit deadlines.
2. Transport failures (such as interrupted TLS handshakes or proxy CONNECT tunnel errors) can leave corrupted connection handles active inside httpcore 1.0.9 connection pools, eventually exhausting max_connections and stalling the runner.

Socket Duplication and Client Retirement

Datagrunt 5.3.10 introduces pooled HTTP/1.1 keep-alive connection management (http.py):

• Pre-TLS Socket Duplication (_BoundConnection): When keepalive_connections > 0, HttpClient duplicates the underlying plain socket.socket before TLS negotiation. During pooled connection reuse, the timer thread uses this duplicate handle to cut the socket at the exact request deadline.
• Immediate Client Retirement: Upon any transport error (network drop, deadline cut, or TLS handshake failure), the failing client instance is immediately retired (_retire_client). Subsequent requests receive a fresh client instance while the retired client closes cleanly once its remaining active requests complete.

from datagrunt.rationalization.providers.http import HttpClient

# Initialize an HTTP client with pooled keep-alive connections
client = HttpClient(max_connections=64, keepalive_connections=16)

# Requests reuse pooled HTTP/1.1 connections while preserving deadline enforcement

2. Single-Pass Fast Request Key Generation & Unified Preflight

The Preflight Hashing Bottleneck

Before executing requests, Datagrunt computes a canonical SHA-256 request key (request_key) for every (unit, provider) pair to check the memory and file caches. Previously, this required serializing a complete canonical JSON dictionary holding envelope key material, model identifiers, provider schemas, generation parameters, and schema versions for every candidate request.

Copyable SHA-256 Digest State & Single-Pass Accumulator

Datagrunt 5.3.10 accelerates cache key generation and preflight checks:

• Canonical Byte Suffix Pre-Serialization (_provider_key_suffix): Non-envelope fields (format, generation_params, model_id, provider_kind, provider_schema, schema_version) are pre-serialized into canonical UTF-8 bytes once per provider snapshot.
• Copyable SHA-256 Hashing (_fast_request_key): Envelope key material is converted to an opening SHA-256 prefix state (_envelope_key_prefix). _fast_request_key() duplicates this state via hashlib.sha256().copy() and updates it with the pre-built provider suffix, yielding a bit-identical hex digest without dictionary or string serialization passes.
• Unified Target Work Accumulator (_WorkAccumulator): Combines target document expansion, unit packing, envelope building, fast request key generation, and cache probing into a single pass (_build_target_work), populating _PreflightPlan efficiently.

from datagrunt.rationalization.cache import (
    _envelope_key_prefix,
    _provider_key_suffix,
    _fast_request_key,
)
import hashlib

# Fast bit-identical request key generation using copyable SHA-256 state
prefix = _envelope_key_prefix(envelope)
env_hasher = hashlib.sha256(prefix)

# Compute key in O(1) time without dictionary canonicalization
key = _fast_request_key(envelope, provider_suffix, envelope_hasher=env_hasher)

3. Min-Heap Rate-Key Scheduler

Logarithmic Scheduler Dispatch

In multi-model extraction runs spanning multiple providers and rate keys (e.g., separate rate limits for Gemini, OpenAI, and Anthropic), the request runner dispatches queued requests as rate-limit tokens and concurrency slots become available. Previously, the scheduler scanned rate key queues linearly on every dispatch iteration.

Datagrunt 5.3.10 updates _RequestScheduler (runner.py) to maintain a heapq min-heap of active rate keys ordered by earliest due token time. When concurrency slots open up, the scheduler pops the earliest due request in $O(\log K)$ time instead of scanning $O(K)$ rate key queues, eliminating scheduling latency when managing dozens of concurrent rate limits.

4. Streaming Batch Preparation & Ingestion

Memory-Efficient Offline Cloud Batch I/O

Datagrunt’s offline cloud batch mode (prepare_batch and ingest_batch) allows users to prepare extraction requests offline, submit them to provider batch endpoints (Vertex AI / Gemini and Anthropic Message Batches) for a 50% token price discount, and ingest the results back into the rationalization pipeline.

Datagrunt 5.3.10 transitions batch I/O to streaming operations:

• Direct Line-By-Line JSONL Export (to_vertex_jsonl, to_anthropic_jsonl): Batch items stream directly to output file handles line-by-line, avoiding full payload accumulation in memory.
• Bounded Binary Stream Ingestion (_read_bounded_jsonl & _load_jsonl_file): Response JSONL files are read via binary stream buffers with strict per-line caps (BATCH_LINE_MAX_BYTES = 16 MiB) and line count caps (BATCH_MAX_LINES = 1,000,000), stripping CR/LF line endings safely before length validation.

from datagrunt import prepare_batch, ingest_batch
from my_schemas import InvoiceSchema

# 1. Prepare offline batch plan
plan = prepare_batch(
    target="large_corpus/*.pdf",
    schema=InvoiceSchema,
    models=["gemini-2.5-flash", "claude-3-5-sonnet"],
)

# 2. Stream batch requests directly to JSONL files with O(1) memory footprint
plan.to_vertex_jsonl("batch_requests/vertex_gemini.jsonl")
plan.to_anthropic_jsonl("batch_requests/anthropic_messages.jsonl")

# 3. Ingest response JSONL files via bounded binary streams
result = ingest_batch(
    plan,
    responses=[
        "batch_responses/vertex_gemini_results.jsonl",
        "batch_responses/anthropic_messages_results.jsonl",
    ],
)

5. Shared Frame & Band Cache Envelope Isolation

When rationalize() runs under strict=False, rendering errors on individual PDF pages (such as corrupt embedded raster images) are caught and isolated into SkippedUnit records rather than failing the entire extraction run.

Datagrunt 5.3.10 introduces build_envelopes_isolated() (envelope.py), which constructs the document _Frame, context JSON, prompt text, and _BandCache once per document. During unit processing, per-unit envelope generation reuses these shared inputs, eliminating redundant frame creation and input validation while preserving page-level render fault isolation.

Verification & Upgrade

To upgrade to Datagrunt 5.3.10 using uv:

uv pip install --upgrade datagrunt

Verify your installed version in Python:

import datagrunt
print(datagrunt.__version__)  # 5.3.10