Datagrunt 5.3.3: Multi-Core Corpus Processing, CAD Vector Text, and Fast 16MP Renders

November 10, 2026by Martin Graham

Datagrunt 5.3.3 is now officially released! This feature and performance update brings process-isolated multi-core scaling to extract_corpus(), restores vector-backed text and table extraction for CAD technical drawings and blueprints (Text Render Mode 3), suppresses micro-tiled raster image fragments, and delivers massive rendering speedups for high-resolution 16MP pages (#484).

Key Highlights in Datagrunt 5.3.3

  1. Multi-Core Corpus Extraction (extract_corpus) — Scales batch corpus processing across isolated worker process pools (_CorpusProcPool) with strict IPC timeout protection (_CORPUS_DOC_TIMEOUT_S), worker health monitoring, worker retirement, and seamless warning re-emission back to the parent process.
  2. Smart Extraction Bundle Reuse — Relaxes bundle validation (_same_output_params) to ignore execution parameters like worker count (workers), allowing existing extraction bundles to be reused instantly across different concurrency settings.
  3. CAD Vector-Backed Text & Table Extraction (Render Mode 3 / Tr 3) — Identifies invisible text elements (Text Render Mode 3) overlaid on vector stroke paths (glyph_stroke_boxes) in CAD diagrams, blueprints, and engineering schematics (assembly.py), accurately extracting labels, dimension text, and vector tables without misclassifying them as hidden content.
  4. Prompt Injection & Resource Guards — Protects vector text classification against prompt injection attempts or pathological geometry with strict bounding box scale constraints and evaluation budget limits (_MAX_VECTOR_TEXT_CHECKS).
  5. Mosaic & Tiled Figure Deduplication — Suppresses redundant raster image tiles and strip fragments generated by CAD and satellite software exports while preserving legitimate overlapping subplots and figures. Features an $O(S \log S)$ strip chain retirement algorithm with forward union span checks.
  6. Ultra-Fast 16MP PNG Page Rendering Engine — Introduces a custom, zero-copy 8-bit grayscale PNG encoder (_png.py) operating with filter-type 0 and zlib level 1 compression to eliminate CPU-intensive Pillow row filtering on high-resolution 16MP page and band renders.
  7. PDFium Session Batching & Single-Pass Guards — Shares DocumentHandle instances across bundle renders (bundle.py), consolidates image guard checks and page object walks into a single pass (_guarded_with_objects), reuses FigureSession handles across in-process batch figure reads, culls micro-paths via outer-mapped GetBounds, and caches page font metadata (font_cache).

1. Multi-Core Corpus Extraction & Process Isolation (#484)

High-Throughput Batch Corpus Scaling

Large-scale PDF document ingestion across repositories or data lakes often requires processing thousands of files concurrently. In Datagrunt 5.3.3, extract_corpus() spawns worker processes in worker pools (_CorpusProcPool in src/datagrunt/pdf/corpus.py) to execute extraction in process isolation.

Each worker operates independently with strict IPC message passing and document-level timeout protection (_CORPUS_DOC_TIMEOUT_S). If a worker encounters a corrupted or hung PDF, the parent process detects the timeout or failure, cleanly retires the worker, records a CorpusFailure, and continues processing remaining documents.

from pathlib import Path
from datagrunt.pdf.corpus import extract_corpus
from datagrunt.pdf._options import PipelineOptions

# Extract a directory of PDF documents across 8 worker processes
manifest = extract_corpus(
    source_dir=Path("./raw_documents"),
    target_dir=Path("./processed_bundles"),
    workers=8,
    options=PipelineOptions(strict=False),
)

print(f"Successfully processed {len(manifest.entries)} documents.")
print(f"Encountered {len(manifest.failures)} failures.")

Warning Serialization & Re-emission

When worker processes emit non-fatal UserWarnings (e.g., degraded page fallback, unreadable fonts, or OCR resolution warnings), _decode_corpus_reply serializes these warnings in the worker’s JSON reply and re-emits them in the parent process:

# Non-fatal warnings produced inside child worker processes are re-emitted in the main thread:
# UserWarning: Page 4 image drawing degraded: rendered at lower resolution

Smart Extraction Bundle Reuse

Previously, modifying the workers parameter caused Datagrunt to invalidate cached document bundles because extraction parameters did not match. In 5.3.3, _same_output_params in src/datagrunt/pdf/_bundle_load.py ignores runtime execution parameters like workers:

# Subsequent runs with different worker counts reuse previously built bundles instantly:
manifest_reused = extract_corpus(
    source_dir=Path("./raw_documents"),
    target_dir=Path("./processed_bundles"),
    workers=16,  # Bundle output parameters match; cached bundles are reused
)

2. CAD Vector-Backed Text & Table Extraction (Render Mode 3) (#484)

The Technical Drawing Extraction Challenge

In computer-aided design (CAD) software (AutoCAD, MicroStation, Revit), text labels, dimensions, title blocks, and tables are frequently exported as invisible PDF text objects (Text Render Mode 3 / _INVISIBLE_TEXT_MODE, where glyphs have neither fill nor stroke) laid precisely over vector stroke lines (glyph_stroke_boxes) that draw the visible characters.

In earlier versions, Datagrunt flagged all non-raster invisible text objects as invisible hidden layer text and suppressed them from Markdown and DataFrame outputs.

Vector-Backed Character Classification

Datagrunt 5.3.3 introduces vector stroke character matching in src/datagrunt/pdf/assembly.py:

  • Glyph Band Spatial Indexing (_banded): Organizes vector stroke boxes (glyph_stroke_boxes) into horizontal page bands for $O(1)$ spatial lookup.
  • Glyph Stroke Validation (_char_has_glyph_stroke): Checks whether an invisible character box is backed by visible non-white vector stroke paths matching character-scale dimensions.
  • Character Flags (_char_vector_flags): Flags individual characters as vector-backed, preserving spaces between backed neighbors while removing actual hidden text.
from datagrunt import PDFReader

# Extract text and tables from a CAD schematic or engineering blueprint
reader = PDFReader("blueprint_schematic.pdf")

# Vector-backed Text Render Mode 3 labels and tables are now visible and extracted:
for table in reader.tables:
    print(f"Table Scope: {table.scope_path}")
    print(table.to_dataframe())

markdown = reader.to_markdown()
print(markdown)

Security & Prompt Injection Protection

To prevent malicious or pathological PDFs from embedding thousands of micro-vectors to bypass hidden text classification or inject prompt instructions into LLM pipelines:

  • Character Scale Limits: Vector stroke bounding boxes are constrained to character dimensions (max_side capped between 18.0 pt and 36.0 pt).
  • Global Evaluation Budget (_MAX_VECTOR_TEXT_CHECKS): Caps vector lookup budget per page to guarantee deterministic execution bounds.

3. Mosaic & Tiled Figure Deduplication (#484)

Suppressing Image Slices & Tile Fragments

High-resolution GIS exports, satellite imagery, and CAD drawings often break background images into hundreds of tiny image tiles or slice strips (e.g., 48x48 px raster blocks). Extracting every tile produces noisy figure lists.

Datagrunt 5.3.3 introduces mosaic and tiled figure detection in src/datagrunt/pdf/quality.py and src/datagrunt/pdf/assembly.py:

  • Strip Chain Retirement: Uses an $O(S \log S)$ algorithm with forward union span checks to detect and retire non-qualifying image strip chains.
  • Bucket Flattening: Streamlines tile and strip bucket helpers to maintain clean execution paths under structural depth bounds.
  • Subplot & Overlap Preservation: Ensures real overlapping images, diagrams, and multi-panel figures remain untouched while filtering out tile grid noise.

4. Ultra-Fast 16MP PNG Page Renders & Session Batching (#484)

Zero-Copy Lossless PNG Encoder (_png.py)

Rendering high-resolution 16 megapixel (16MP+) PDF pages previously spent significant CPU time inside Pillow’s standard PNG writer applying row filters (Sub, Up, Average, Paeth) and heavy zlib compression.

Datagrunt 5.3.3 implements a targeted 8-bit grayscale PNG encoder in src/datagrunt/pdf/_png.py:

from datagrunt.pdf._png import grey_png_bytes, write_grey_png

# Direct stream encoding with filter-type 0 and fast zlib compression:
png_bytes = grey_png_bytes(width=4096, height=4096, pixels=raw_grayscale_buffer)

By switching to filter-type 0 (no filtering) and zlib compression level 1, page renders complete up to 4x faster with minimal file size impact.

Single-Pass Guard & Object Walking

In src/datagrunt/pdf/_render_worker.py, _guarded_with_objects pre-walks page objects once and extracts image elements (_image_items_of). This eliminates redundant object traversals and avoids accessing destroyed text pointers during image extraction.

Figure Session Batching

FigureSession in src/datagrunt/pdf/pdfium_engine.py keeps the document handle and image loader context open across batch figure reads in _loaders.py:

# In-process figure reads reuse open handles and object walks across batch reads:
with handle_ctx as handle, FigureSession(handle) as session:
    for read in batch:
        yield session.read(loader, kind)

Micro-Path Culling & Font Caching

  • Micro-path Culling (src/datagrunt/pdf/_path_segments.py): Evaluates path bounding boxes via outer-mapped GetBounds to instantly cull sub-pixel micro-paths before point transformation.
  • Page Font Caching (src/datagrunt/pdf/_pdfium_text.py): Caches font descriptors (font_cache) across text objects on a page, eliminating repeated C-FFI calls to PDFium font metadata functions.

Installation & Upgrade

Upgrade to Datagrunt 5.3.3 using uv or pip:

uv pip install --upgrade "datagrunt[pdf]"

For full details on API signatures and usage options, check out the Datagrunt Documentation.

Datagrunt 5.3.2: PDF Stream Line Assembly & CI Workflow Deduplication