Datagrunt 4.6.22: Groundwork for Vision-Based PDF OCR Primitives

Datagrunt 4.6.22 introduces the core v5 vision route OCR primitives in datagrunt.pdf.ocr. This update lays the foundational pipeline for robust image-only and scanned PDF extraction.

Highlights of 4.6.22

When processing scanned PDF pages or image layers without native text, Datagrunt relies on Tesseract to extract textual content and spatial layout coordinates. Datagrunt 4.6.22 (#414) introduces low-level OCR building blocks that improve stability, accuracy, and process boundary isolation.

Key Capabilities Introduced

1. Page Grayscale Normalization (normalized):

   • Applies Pillow’s C-accelerated ImageOps.autocontrast(cutoff=1) to stretch pixel intensity histograms across the 0–255 grayscale range while ignoring 1% edge noise.
   • Low-contrast scans and faded background documents produce cleaner binary thresholding for downstream OCR.

2. Horizontal Page Banding (bands):

   • Slices rendered pages into 900 px tall horizontal Band objects with 60 px overlapping margins.
   • Standard text lines (up to 14.4 pt at 300 DPI) spanning band boundaries remain whole in at least one band, preventing clipped glyph recognition errors.

3. Page Orientation Detection & Rotation (orientation / rotated):

   • Executes Tesseract orientation detection (--psm 0) to determine page rotation angles (0°, 90°, 180°, or 270°).
   • Rights sideways or upside-down pages prior to word extraction using fast Pillow image transpositions.

4. Tesseract CLI Word Extraction & Coordinate Mapping (ocr_words):

   • Calls the Tesseract CLI (--psm 6, TSV output mode) across page bands using PGM binary streams via stdin (eliminating temporary image files on disk).
   • Converts word bounding boxes back to PDF point coordinates (top-left origin) on the original displayed page, reversing rotation transformations and accounting for DPI scale factors.
   • Merges words detected across adjacent overlapping bands using Intersection-over-Union (IoU ≥ 0.5) spatial matching, keeping single instances with higher confidence ratings.

5. Process Boundary Isolation & Memory Safety:

   • Limits child process thread count (OMP_THREAD_LIMIT=1) and enforces strict stdout bounds (16 MiB max).
   • Captures stderr tails up to 4 KiB in anonymous temporary files for structured error diagnostics.
   • Returns explicit status structures (Orientation, OCRResult) with detailed error reasons instead of raising unhandled process exceptions.

Technical Overview & Internal Primitives

Below is an example showing how the new internal primitives in datagrunt.pdf.ocr process a rendered page:

from datagrunt.pdf.render import render_page
from datagrunt.pdf.ocr import (
    normalized,
    bands,
    orientation,
    rotated,
    ocr_words,
)

# 1. Render page at 300 DPI within a memory-capped worker
rendered_page = render_page(
    "scanned_document.pdf",
    page_index=0,
    dpi=300,
    memory_cap_bytes=512 * 1024 * 1024,
    timeout_s=10.0,
)

# 2. Normalize contrast via 1% autocontrast cutoff
norm_page = normalized(rendered_page)

# 3. Detect page orientation and rotate to righting angle
orient = orientation(norm_page, timeout_s=5.0)
if orient.status == "ok" and orient.degrees != 0:
    righted_page = rotated(norm_page, orient.degrees)
else:
    righted_page = norm_page

# 4. Slice page into 900 px horizontal bands with 60 px overlap
page_bands = bands(righted_page, height=900, overlap=60)

# 5. Extract words with Tesseract and map coordinates back to displayed page
result = ocr_words(
    page_bands,
    dpi=300,
    rotation=orient.degrees,
    page_size_pt=(612.0, 792.0),
    timeout_s=10.0,
    language="eng",
)

if result.status == "ok":
    for word in result.words:
        print(f"Word: {word.text!r} | BBox: {word.bbox} | Confidence: {word.confidence:.1f}%")
else:
    print(f"OCR Status: {result.status} | Reason: {result.reason}")

Upgrading

To install or upgrade to Datagrunt 4.6.22:

uv pip install --upgrade datagrunt