Datagrunt 5.3.6: Single-Pass Object Walk, Fast Trusted Constructors, and Single-Load Corpus Renders

Datagrunt 5.3.6 is officially available! This performance-focused release dramatically accelerates PDF extraction, page background rendering, and large-scale batch corpus ingestion (#488).

By unifying PDFium object tree traversals, introducing zero-overhead internal segment constructors, and fusing page stage extraction with background rendering in a single load pass, Datagrunt 5.3.6 significantly reduces wall-clock extraction time and memory overhead across large document sets.

Key Highlights in Datagrunt 5.3.6

1. Single-Pass PDFium Object Tree Walk (pdfium_engine.py) — Replaces redundant page traversals (text extraction, vector stroke segment compilation, image inspection, and render refusal checks) with a single-pass object walk (collect_images_and_refusal / _read_stages_with_guard).
2. Fast Trusted Segment Constructors (_path_segments.py) — Introduces RawSegment._trusted and PageSegments._trusted class constructors for engine-validated path segments, bypassing redundant $O(N)$ type checks and finite-float validation during dense vector extraction.
3. Single-Load Fused Page Renders (_page_worker.py) — Eliminates repeated page handle re-opening. PDF pages are loaded once to extract document stages, assess OCR/vision routes, draw background raster images, and capture full-page pre-render PNGs in a single pass (_assessed_fused_single_load).
4. Bounded In-Memory Prerender Cache (_options.py) — Adds _PrerenderCache, a 128 MB bounded cache with pop-on-write eviction that preserves single-load full-page PNG bytes between extraction and pipeline stages without re-drawing.
5. CI Runner Resiliency (.github/workflows/ci.yml) — Adds automatic fallback to GPG-verified mirrors.edge.kernel.org mirrors to protect CI builds against Azure runner DNS outages on Ubuntu package archives.

1. Single-Pass PDFium Object Walk (#488)

The Overhead of Multi-Pass Traversals

In earlier releases, extracting a PDF page required multiple traversals over the underlying C/PDFium page object tree:

• Traversing top-level and form objects for text spans and character boxes.
• Traversing path objects to assemble vector strokes and fill segments.
• Traversing image objects to gather dimensions and stream metadata.
• Traversing image objects a second time to test for render refusal guards.

For complex PDFs—such as CAD schematics or multi-page financial statements containing thousands of vector paths and inline images—the repeated context switching across Python and the underlying PDFium C library introduced measurable CPU overhead.

Unified Object Tree Walking

Datagrunt 5.3.6 unifies image collection and render refusal checking into collect_images_and_refusal(), integrated directly into _read_stages_with_guard() inside src/datagrunt/pdf/pdfium_engine.py:

# Internal engine architecture: single-pass traversal
# Collects text, path segments, images, and first render refusal in ONE object walk
page_images, first_refusal = collect_images_and_refusal(
    placed_images, page_raw, page_box, to_display
)

This single-pass architecture halves C API traversal calls per page while preserving identical page stage isolation and error handling bounds.

2. Single-Load Fused Page Renders & Prerender Cache (#488)

Eliminating Duplicate Page Loads

When processing batch document corpora with extract_corpus() or running vision/OCR workflows, pages previously underwent multiple open/close cycles: one for text/segment stage assessment, and another for drawing background raster layers or OCR page rendering.

Datagrunt 5.3.6 introduces _assessed_fused_single_load() in src/datagrunt/pdf/_page_worker.py and _PrerenderCache in src/datagrunt/pdf/_options.py. When _can_fuse_page_load(options) is active:

1. The page is opened once via _eng._load_page(handle, index).
2. Text, vector segments, and images are extracted.
3. If background rendering or OCR pre-rendering is required, 8-bit grey PNG pre-renders and background rasters are drawn immediately on the active pdfium.PdfPage handle.
4. Pre-rendered PNG bytes are stored in _PrerenderCache (bounded at 128 MB by default) and popped on write, releasing memory instantly as downstream pipeline workers consume them.

from datagrunt.pdf.corpus import extract_corpus, load_corpus

# Batch extract a directory of multi-page PDFs using fused single-load rendering
extract_corpus(
    sources="input_documents/",
    output_dir="corpus_output/",
    ocr="auto",
    workers=8,
)

# Load the verified corpus manifest and query extracted bundles
manifest = load_corpus("corpus_output/")
print(f"Extracted {len(manifest.entries)} documents successfully.")

3. Fast Trusted Constructors for Dense Vector Paths (#488)

PDF documents with heavy vector graphics (such as CAD architectural drawings, spatial maps, or charts) contain thousands of RawSegment instances per page.

Previously, RawSegment.__init__ performed scalar type checking, bounds validation (start < end), finite number checks (math.isfinite), and tuple verification for every single path segment. In 5.3.6, internal engine functions use RawSegment._trusted() and PageSegments._trusted(), which construct frozen dataclass instances directly via object.__new__ after the engine has already validated finite coordinates. This removes validation bottlenecks during heavy vector extraction.

Summary of Changes in 5.3.6

Getting Started

To upgrade to Datagrunt 5.3.6 using uv:

uv pip install --upgrade datagrunt

For PDF parsing support:

uv pip install --upgrade "datagrunt[pdf]"