Datagrunt 5.3: Image-Only Page Markers, Bundle Opt-Out Warnings & LLM Skip Integration

Datagrunt 5.3.0 is now officially available! This update introduces explicit image-only page markers, comprehensive bundle opt-out warnings and missing artifact tracking, bundle manifest format 5.3, and automatic rationalization skips for pages whose content lives only in their page image.

Key Highlights in Datagrunt 5.3.0

1. Explicit Image-Only Page Markers (Page.image_only_reasons & PDFDocument.image_only_pages) — Detects pages whose text layer cannot stand (due to sparse text, garbled font encodings, or extraction failures) across all OCR modes, exposing structural reason markers and 1-based page indices.
2. Bundle Opt-Out & Missing Render Warnings — Emits explicit UserWarning alerts when writing bundles with render_pages=False or include_images=False (or when page rendering fails) on pages whose content exists solely in their page image, signaling that the source PDF is their only full copy.
3. Bundle Manifest Format 5.3 & Document JSON v2 — Upgrades manifest validation and serialization to bundle format 5.3 and Document JSON version 2. Datagrunt 5.3 maintains full backward compatibility for loading legacy 5.0 bundles written by versions 5.0–5.2.
4. Automatic Rationalization Skips (SkippedUnit(reason="image_only")) — Automatically excludes image-only pages before LLM prompt envelope assembly in rationalize(), prepare_batch(), and ingest_batch(), preventing wasted token consumption on text-deficient pages.

1. Explicit Image-Only Page Markers

In complex enterprise document processing pipelines, certain PDF pages contain vector graphics, scanned document forms, or corrupted font encodings where extractable text is missing or unreadable.

When running under ocr="never" (or when OCR is unavailable), these pages remain routed as route="digital" and status="ok", but their text layer does not capture the true page content. Datagrunt 5.3.0 introduces explicit tracking via Page.image_only_reasons and PDFDocument.image_only_pages.

Image-Only Reason Codes (ImageOnlyReason)

• "too_little_text" — The page has too few readable characters despite drawing embedded images or vector paths.
• "garbled_text" — A disproportionate share of the page’s visible text characters are invalid or unprintable glyphs.
• "extraction_failed" — An error page where digital extraction failed completely.

API & Serialization Surfaces

Image-only page markers are surfaced consistently across every output interface:

• Python API: page.image_only_reasons returns a tuple of reason strings (e.g., ("too_little_text",)), and document.image_only_pages returns a tuple of 1-based page numbers.
• Element DataFrames & Tables: Exposes a 13th column, image_only_reasons (List(String)), containing the page’s image-only reasons.
• Markdown Headers: Page markers include image_only tags (e.g., <!-- page 2 route=digital status=ok image_only=too_little_text -->).
• Document JSON v2 & Bundle Format 5.3: Document JSON records image_only_reasons, and bundle manifests track image_only_pages.

Downstream Usage Example

Pipelines can query image_only_reasons to route specific marked pages directly to secondary vision models:

from datagrunt import PDFReader

# Load PDF with ocr="never" to inspect digital text layer first
reader = PDFReader("financial_scans.pdf", ocr="never")

print(f"Image-only page numbers: {reader.document.image_only_pages}")

# Selectively process page images for marked pages
for page in reader.pages:
    if page.image_only_reasons:
        print(f"Page {page.page_number} relies on page image: {page.image_only_reasons}")
        if page.page_image is not None:
            raw_png = page.page_image.read_bytes()  # Drawn directly from PDF source

2. Bundle Opt-Out Warnings & Missing Content Tracking

When exporting document bundles with write_bundle(), users can specify flags such as render_pages=False or include_images=False to optimize execution speed and storage footprint.

However, if a page is marked as an image-only page (image_only_reasons non-empty) and its page image is skipped or fails to render, no rendered copy of that page exists inside the bundle.

Opt-Out User Warnings

In Datagrunt 5.3.0, publishing a bundle under these conditions emits a targeted UserWarning naming the affected pages, document stem, bundle destination path, and cause:

from datagrunt import PDFWriter

writer = PDFWriter("scanned_contract.pdf")

# Fast bundle creation with page rendering disabled
writer.write_bundle("out/contract_bundle", render_pages=False)
# Emits UserWarning:
# "scanned_contract.pdf: page(s) [1, 3] content lives only in their page image, "
# "but out/contract_bundle holds no page images (render_pages=False): "
# "the source PDF is their only full copy."

Manifest Missing Entry Detail

Furthermore, manifest.json under missing appends detailed notes explaining why page image artifacts are absent:

{
  "path": "page_images/page_0001.png",
  "page_number": 1,
  "reason": "render_pages=False; its content lives only in its page image (too_little_text): the source PDF is its only full copy"
}

3. Bundle Manifest Format 5.3 & Version Interoperability

Datagrunt 5.3.0 updates the default bundle manifest format to "5.3" and Document JSON format to version 2.

• Format 5.3 Manifests: Includes the top-level image_only_pages list after degraded_pages.
• Backward Compatibility for Format 5.0: Bundles written by Datagrunt 5.0 through 5.2 (manifest format "5.0", document JSON version 1) continue to load without errors. When loading legacy 5.0 bundles, page.image_only_reasons evaluates to None (unrecorded). Re-saving or re-bundling a loaded 5.0 document automatically upgrades it to format 5.3.

from datagrunt.pdf import load_bundle

# Seamlessly loads format 5.0 and format 5.3 bundles
doc = load_bundle("legacy_bundle_v5_1/")

4. Automatic LLM Rationalization Skips

In Datagrunt’s schema-driven extraction pipeline (rationalize(), prepare_batch(), and ingest_batch()), prompt units are pre-built from visible text elements.

In version 5.3.0, any page identified as image-only (image_only_reasons non-empty) is automatically excluded during preflight unit packing before any LLM request is constructed.

SkippedUnit Reporting

These units are registered in result.skipped_units as a SkippedUnit with reason="image_only" (or reason="page_error" for an error page):

from datagrunt import rationalize, PDFReader
from pydantic import BaseModel

class AuditSchema(BaseModel):
    account_number: str
    total_balance: float

reader = PDFReader("scans_and_text.pdf", ocr="never")

result = rationalize(
    reader.document,
    schema=AuditSchema,
    prompt="Extract account number and total balance.",
    models=["openai:gpt-5.4"],
)

# Inspect units skipped prior to prompt assembly
for skipped in result.skipped_units:
    if skipped.reason == "image_only":
        print(f"Page {skipped.page_numbers} skipped: image-only page without extracted text.")

By excluding image-only pages prior to prompt envelope construction, Datagrunt prevents sending garbled or missing text to provider APIs, reducing API costs and preventing hallucinated extractions.

Summary of Changes

Upgrading

To upgrade to Datagrunt 5.3.0 using uv:

uv pip install --upgrade "datagrunt[pdf,rationalize]"