Datagrunt 5.3.11: Textless Scan Peer Grounding, Informational List Union, Grounded-Union Tier & Deployment Healing

November 16, 2026by Martin Graham

Datagrunt 5.3.11 is now available! This release introduces critical accuracy and robustness improvements across the rationalization structured extraction subsystem (src/datagrunt/rationalization/) as well as automated site deployment verification (PR #493, PR #495).


Key Highlights in Datagrunt 5.3.11

  1. Textless Scan Peer-Quote Grounding (_peer_witness_for & _waivable_scan_quote) — Resolves quote grounding false demotions on multimodal scanned PDF pages where OCR text blocks are absent (textless_anchor). Cross-family peer quotes are precomputed (_build_multimodal_peer_map) to corroborate visual quotes ($\ge 0.85$ token containment via VISION_CONTAINMENT), waiving secondary scan context quote demotions (_waivable_scan_quote) when primary record keys are grounded.
  2. Informational Scalar List Union (_merge_informational_scalar_list) — Merges non-disputing Informational scalar_list fields across multi-model consensus responses in first-seen normalized order without triggering false field_disagreement status demotions.
  3. Grounded-Union Tier & Export Queries (include_grounded_union=True) — Adds the grounded-union tier (GROUNDED_UNION_STATUSES) to GroundedRecord, FoldingResult, and RationalizedResult. High-recall pipelines can now query and export clean consensus records, single-model records, and grounded single-family catch records via to_dataframe(include_grounded_union=True) and to_dicts(include_grounded_union=True).
  4. Anthropic Fallback Latch (_latch_prompt_fallback & _should_use_native) — Latches complex JSON schemas or unsupported structured output capability HTTP 400 responses into an LRU cache (_fallback_schemas), automatically routing subsequent requests to prompt-based JSON extraction without wasting API retries.
  5. Live Site Deployment Healing (verify_site_deploy.py) — Verifies post-release deployment on datagrunt.io by probing HTTP 200 responses and version strings, automatically healing dropped Cloudflare Pages GitHub webhooks via bounded empty-commit pushes (retrigger_empty_commit).

1. Textless Scan Peer-Quote Grounding & Secondary Quote Waiving

The Challenge of Scanned PDF Multimodal Extraction

When performing multi-model rationalization across scanned PDF documents or image-only pages, models inspect visual page renderings and extract structured fields along with verbatim evidence quotes. However, because scanned pages may lack bounding OCR text elements (textless_anchor), visual quotes given by one vision model might not match raw OCR text strings line-for-line.

Previously, if a supporting model returned secondary scan context quotes that could not be matched against raw OCR text, the grounding gate marked the item with quote_not_found, demoting clean consensus records to ungrounded_value.

Cross-Family Peer Corroboration & Secondary Quote Waiving

Datagrunt 5.3.11 introduces cross-family peer witness maps (_build_multimodal_peer_map & _peer_witness_for in grounding.py):

  • Peer Witness Precomputation: When a unit includes multimodal image anchors, Datagrunt pre-computes comparable quotes, plain quotes, and token counters across peer models belonging to distinct model families (e.g. Anthropic Claude vs Google Gemini vs OpenAI GPT).
  • Token Containment Verification: If a model’s visual quote achieves $\ge 0.85$ token containment (VISION_CONTAINMENT) against cross-family peer models (_witness_token_corroborated), the quote is verified against peer evidence.
  • Secondary Quote Waiving (_waivable_scan_quote): Unmatched secondary scan context quotes that are corroborated by peer models and do not conflict with strict text elements are safely waived, preventing false demotions to ungrounded_value when primary record keys are fully grounded.
from datagrunt.rationalization import rationalize
from my_schemas import InvoiceSchema

# Rationalize multimodal scanned PDF invoices across Anthropic and Gemini
result = rationalize(
    target="scans/*.pdf",
    schema=InvoiceSchema,
    models=["claude-3-5-sonnet", "gemini-2.5-flash"],
)

# Textless scan quotes corroborated by cross-family peers preserve consensus status
for item in result.items:
    if item.is_grounded:
        print(f"Grounded Record [{item.status}]: {item.record}")

2. Informational Scalar List Union & Disagreement Folding

Uniting Non-Disputing List Fields

In structured schemas, fields marked as Informational scalar_list (such as lists of product tags, category codes, or secondary reference IDs) accumulate supplementary values from different models. Previously, minor differences in list ordering or missing entries between models could trigger field_disagreement status.

Datagrunt 5.3.11 updates consensus.py with _merge_informational_scalar_list():

  • First-Seen Normalized Union: Combines scalar list items across model votes in first-seen normalized order without declaring field disputes.
  • Refined Cluster Winner Selection: Updates _compare_informational() to score populated scalar list clusters correctly, ensuring agreement on populated items yields a clean consensus winner.
from pydantic import BaseModel, Field
from datagrunt.rationalization import rationalize

class ProductExtraction(BaseModel):
    sku: str = Field(..., json_schema_extra={"key": True})
    tags: list[str] = Field(default_factory=list, json_schema_extra={"consensus": "informational"})

# Extraction merges tags from all supporting models in first-seen normalized order
result = rationalize(target="catalog.pdf", schema=ProductExtraction, models=["gpt-4o", "claude-3-5-sonnet"])

3. Grounded-Union Tier & Export Queries

High-Recall Grounded Data Export

While result.records returns clean validated records (consensus or single_model), downstream data engineering pipelines often require high-recall extraction that includes grounded records caught by a single model family (missing_in_some_models) provided they have zero grounding failures.

Datagrunt 5.3.11 introduces the Grounded-Union Tier:

  • Tier Definition (GROUNDED_UNION_STATUSES): Includes statuses ("consensus", "single_model", "missing_in_some_models").
  • Grounded Record Inspection (is_grounded_union): A record belongs to the grounded-union tier if status in GROUNDED_UNION_STATUSES, record is not None, grounding_reasons is empty, disputed_fields is empty, and notes is empty.
  • New Properties (grounded_items & grounded_records): Exposed on both FoldingResult and RationalizedResult.
  • DataFrame & Dict Export Flags (include_grounded_union=True): to_dataframe() and to_dicts() support include_grounded_union=True to export grounded-union records with _status and _doc_id metadata columns.
# Export clean consensus AND grounded single-family catch records to Polars DataFrame
df = result.to_dataframe(include_grounded_union=True)

print(df.select(["_doc_id", "_status", "sku", "price"]))

4. Anthropic Structured Output Fallback Latching

Schema-Complexity & Capability Fallback Latching

Anthropic’s Messages API supports native JSON Schema structured outputs (output_config.format). However, if a schema contains complex nested definitions or unsupported constraints, the API returns an HTTP 400 bad request. Retrying native structured outputs on every unit request causes unnecessary latency and error logging.

Datagrunt 5.3.11 enhances AnthropicProvider (providers/anthropic.py):

  • Fallback Latch (_latch_prompt_fallback): When a 400 error occurs due to schema complexity or structured output limits, _latch_prompt_fallback() records the schema digest in _fallback_schemas (an LRU cache of up to 256 entries) or sets _native_disabled = True.
  • Automatic Fallback Routing (_should_use_native): Subsequent extraction requests for that schema or endpoint automatically switch to prompt-based JSON extraction (prompt_fallback), bypassing doomed native API attempts and ensuring smooth execution.

5. Live Site Deployment Verification & Webhook Healing

Automated Deployment Monitoring (verify_site_deploy.py)

To prevent deployment gaps caused by dropped GitHub push webhooks during Cloudflare Pages deploys, Datagrunt 5.3.11 introduces an automated verification and healing script (scripts/verify_site_deploy.py).

  • HTTP Live Verification: Reads current_version from pyproject.toml, constructs the expected release post URL on datagrunt.io, and performs HTTP GET probes to confirm the post is live and contains the version string.
  • Automatic Webhook Healing (retrigger_empty_commit): If Cloudflare Pages missed the push webhook and the post remains offline after initial polling, the script automatically pushes a single fast-forward empty commit (chore: retrigger Cloudflare Pages deploy for version <ver>) to re-emit the deployment event.
  • CI Workflow Integration: Integrated directly into .github/workflows/auto-publish-site.yml step Verify and heal live site deployment.

Verification & Upgrade

To upgrade to Datagrunt 5.3.11 using uv:

uv pip install --upgrade datagrunt

Verify your installed version in Python:

import datagrunt
print(datagrunt.__version__)  # 5.3.11