Datagrunt 5.3.4: Word-Level CAD Tr 3 Backing, Searchable Scan Tile Deduplication, and Graham LPT Corpus Scheduling

Datagrunt 5.3.4 is officially available! This performance and reliability update upgrades vector-backed CAD text extraction to word and span scale, introduces advanced tiled image deduplication for searchable scans and satellite exports, brings Graham’s Longest Processing Time (LPT) scheduling to multi-worker corpus extraction (extract_corpus), and boosts developer iteration speed with parallel pytest-xdist execution and PEP 669 low-overhead line coverage (#485, #486).

Key Highlights in Datagrunt 5.3.4

1. Word-Level CAD Vector Text Backing (Render Mode 3 / Tr 3) — Upgrades CAD technical drawing and blueprint text extraction in assembly.py (_char_has_glyph_stroke and _char_vector_flags) from character-level checks to word and span level, ensuring full dimension callouts (e.g. DIM-001), title blocks, and multi-word schematic labels overlaying vector stroke paths are preserved intact.
2. Searchable Scan & Tiled Raster Image Deduplication — Broadens scanline strip stack detection (_strip_stack_positions, _strip_chains_in_group) and connected tile grid aggregation (_tile_grid_positions, _collapse_and_link_components) across searchable scans, satellite imagery, and CAD exports (_is_scan_mosaic_page), suppressing redundant image tile grids while preserving legitimate figures.
3. Graham LPT (Longest Processing Time) Corpus Scheduling — Orders pending PDF documents largest-file-first (_lpt_todo_indices in corpus.py) before dispatching them across worker process pools (_CorpusProcPool) when strict=False, eliminating long-pole worker idle time and tail latency in multi-worker batch processing.
4. Parallel CI & PEP 669 Low-Overhead Line Monitoring — Speeds up CI test suites with pytest-xdist (-n logical --dist=worksteal), splits wall-clock performance tests marked @pytest.mark.no_cover, and leverages Python 3.12+ low-overhead sysmon monitoring (COVERAGE_CORE=sysmon).
5. Rust & uv Build Caching — Adds actions/cache workflow steps for Cargo dependencies and build artifacts (rust/target/), alongside --reinstall-package datagrunt maturin invalidation to guarantee fresh Rust native binary links across CI runs.

1. Word-Level CAD Vector Text Backing (Render Mode 3) (#486)

The Challenge with CAD Drawing Text

In technical engineering drawings, AutoCAD, Revit, and MicroStation exports, text labels, callouts, and dimension markers are frequently rendered as invisible PDF text streams (Text Render Mode 3 / _INVISIBLE_TEXT_MODE) overlaid precisely on visible vector stroke paths (glyph_stroke_boxes) that draw character outlines.

While earlier versions evaluated individual characters against vector stroke paths, single isolated character misses could fragment words or discard multi-word labels like dimension callouts (DIM-042) or title block specs.

Span- and Word-Level Vector Validation

Datagrunt 5.3.4 extends vector stroke validation in src/datagrunt/pdf/assembly.py to evaluate character sequences at the word and span scale:

• Banded Spatial Indexing (_banded): Organizes vector stroke boxes into horizontal page bands for fast $O(1)$ spatial bounding lookups.
• Glyph Stroke Continuity (_char_has_glyph_stroke): Validates that invisible text characters are continuously backed by visible, non-white vector paths matching character dimensions.
• Word & Space Flag Preservation (_char_vector_flags): Retains spaces between adjacent vector-backed characters, preventing merged or broken words across CAD title blocks and dimension tables.

from datagrunt import PDFReader

# Extract text and tables from a CAD engineering schematic or blueprint
reader = PDFReader("pump_assembly_schematic.pdf")

# CAD vector-backed callouts and title blocks are accurately preserved:
for element in reader.elements:
    if element.kind == "heading":
        print(f"Heading [{element.level}]: {element.text}")
    elif element.kind == "paragraph":
        print(f"Callout: {element.text}")

# Markdown export retains full vector-backed text and table structures:
markdown_output = reader.to_markdown()
print(markdown_output[:500])

2. Searchable Scan & Tiled Raster Image Deduplication (#486)

Suppressing Micro-Tiled Background Grids

GIS mapping tools, CAD drawing sheets, and scanned PDF software frequently break large background pages into dozens or hundreds of small, edge-abutting raster image tiles (e.g. $48 \times 48\text{ px}$ tiles) or narrow scanline strips. Emitting each tile as an individual Figure element clutters document outputs and creates unnecessary disk I/O.

In 5.3.4, Datagrunt enhances mosaic deduplication in src/datagrunt/pdf/assembly.py:

• Searchable Scan Detection (_is_scan_mosaic_page): Identifies pages with high raster image cover fraction ($> 50%$) that lack significant painted digital text ($\ge 20$ characters), categorizing them as searchable or image-only scans.
• Connected Component Union (_collapse_and_link_components): Groups edge-abutting tile grids into unified bounding boxes in $O(T \log T)$ time using disjoint-set (union-find) structures across row and column buckets (_link_axis_buckets).
• Scanline Strip Stack Retirement (_strip_runs_by_span): Aggregates advancing zero-gap strip stacks without bucket boundary splits.

from datagrunt import PDFReader

# Process a 3-page scanned architectural PDF containing 90 micro-tiles
reader = PDFReader("scanned_blueprint_tiles.pdf", min_image_dimension=40)

# Micro-raster tiles making up page backgrounds are deduplicated automatically:
print(f"Total extracted figures: {len(reader.figures)}")  # Suppresses tiled grid noise

3. Graham LPT Corpus Scheduling (extract_corpus) (#486)

Long-Pole Stragglers in Batch Extraction

When extracting large repositories or batches of heterogeneous PDF documents with extract_corpus(), document sizes and page counts often vary from $10\text{ KB}$ single-page receipts to $100\text{ MB}$ multi-hundred-page technical manuals.

If large, heavy files are processed late in a batch run, worker process pools can experience significant tail latency, where a single worker process continues working long after all other workers have finished and gone idle.

Longest Processing Time (LPT) Sorting

Datagrunt 5.3.4 implements Graham’s Longest Processing Time (LPT) scheduling algorithm in src/datagrunt/pdf/corpus.py (_lpt_todo_indices):

• File-Size Pre-sorting (_safe_file_size): When strict=False, remaining PDF documents in a batch are sorted by file size descending before being scheduled across worker process pools (_CorpusProcPool).
• Minimal Tail Latency: Heavy documents are assigned to workers first, allowing smaller files to fill in remaining worker capacity and dramatically reducing total batch completion time.

from pathlib import Path
from datagrunt.pdf.corpus import extract_corpus

# Batch extract a directory of heterogeneous PDFs across 8 worker processes
manifest = extract_corpus(
    inputs=Path("./incoming_pdfs"),
    output_dir=Path("./processed_corpus"),
    max_workers=8,
    strict=False, # Enables LPT largest-file-first worker scheduling
    overwrite=False, # Reuses existing valid bundles instantly
)

print(f"Successfully processed {len(manifest.entries)} documents.")
print(f"Failures recorded: {len(manifest.failures)}")

4. Parallel CI & Developer Infrastructure (#485)

Fast Parallel Testing with pytest-xdist

To improve developer iteration speed and CI job execution times, test runs across both Rust native and Python fallback backends now execute in parallel using pytest-xdist:

# Executed in CI workflows across logical CPU cores:
pytest -n logical --dist=worksteal -m "not corpus and not live and not no_cover" --cov-fail-under=95 tests/

PEP 669 Low-Overhead Line Coverage

CI workflows on Python 3.12+ (3.13, 3.14) utilize PEP 669 sys.monitoring line monitoring via COVERAGE_CORE=sysmon, providing near-zero overhead test coverage evaluation while maintaining strict $95%$ coverage enforcement. Wall-clock performance and benchmark tests are decorated with @pytest.mark.no_cover and run serially without coverage instrumentation overhead.

Summary of Changes in Datagrunt 5.3.4

Upgrading to Datagrunt 5.3.4

Upgrade Datagrunt using uv or pip:

uv pip install --upgrade datagrunt

For full PDF parsing support:

uv pip install --upgrade "datagrunt[pdf]"

Check out the updated Datagrunt Documentation for complete guides and API references!