Datagrunt 5.3.9: Wave 2A Consensus, Grounding, and Folding Optimizations + 5 Bug Fixes

Datagrunt 5.3.9 is now available! This release delivers Wave 2A performance optimizations across the rationalization structured extraction subsystem (src/datagrunt/rationalization/), accelerating consensus resolution, document folding, and citation grounding, while resolving 5 subtle bug fixes (PR #491).

Key Highlights in Datagrunt 5.3.9

1. Inverted-Index Accelerated Unkeyed Consensus (_UnkeyedFeatures & _match_model_unkeyed) — Replaces quadratic $O(N \times M)$ pairwise comparison loops in unkeyed record consensus with an inverted-index lookup over pre-computed feature vectors and posting keys.
2. Copy-On-Write (CoW) Null Stripping (_drop_unshown_cow) — Eliminates redundant dictionary deep-copies during response validation by lazily duplicating record structures only along mutated path branches.
3. Direct Leaf Wire Serialization (_scalar_to_wire & _dump_leaf) — Replaces costly model_dump(mode="json") calls during document folding with $O(1)$ scalar wire primitive conversions, bypassing Pydantic dump overhead.
4. Pre-Indexed String Grounding (_index_alnum_field & _match_alnum_entry) — Replaces repeated string allocation loops during alnum_upper citation grounding with single-pass indexing of upper-case alphanumeric character positions.
5. 5 Critical Rationalization Bug Fixes — Resolves edge-case issues across unkeyed child model informational field folding, double-validation during source skip rebuilds, multimodal vision scope leaks, peer quote counter isolation, and must_agree quorum checks.

1. Inverted-Index Accelerated Unkeyed Consensus

The Challenge with Quadratic Record Comparison

When multi-model extraction pipelines process documents containing unkeyed model lists (such as itemized line items or execution steps lacking unique primary keys), Datagrunt must pair extracted items across model votes to determine consensus. Previously, matching unkeyed records across candidate lists relied on pairwise comparison loops evaluating field similarity across every pair of items. As candidate list lengths grew, comparison overhead scaled quadratically.

Accelerating Match Resolution with Inverted Feature Indexing

Datagrunt 5.3.9 introduces pre-computed feature extraction (_UnkeyedFeatures) and inverted-index candidate pairing (_match_model_unkeyed):

• Feature Vector Pre-Computation: Normalized field values, word-token sets, and posting keys (eq, tok, __empty__) are computed once per record.
• Inverted Index Lookups: Candidate items are indexed by feature keys, allowing candidate pairs with non-zero similarity to be matched in near $O(N)$ time.
• Greedy Match Pairing: Pre-filtered candidate pairs are sorted by similarity score to form optimal consensus assignments efficiently.

from pydantic import BaseModel
from datagrunt.rationalization.consensus import build_consensus
from datagrunt.rationalization.schema import prepare

class UnkeyedLineItem(BaseModel):
    description: str
    quantity: int
    unit_price: float

class InvoiceSchema(BaseModel):
    invoice_number: str
    items: list[UnkeyedLineItem]

# Prepared schema uses inverted-index feature matching for items
schema = prepare(InvoiceSchema)

2. Copy-On-Write Null Stripping & Direct Leaf Wire Serialization

Copy-On-Write (CoW) Null Stripping

Before validating LLM provider responses against a target schema, Datagrunt removes null values for optional fields that were not shown in the prompt payload (_drop_unshown). In earlier releases, this required creating full deep copies of response dictionaries.

Datagrunt 5.3.9 replaces full dictionary duplication with _drop_unshown_cow():

• Fast-Path Bypass: If spec.has_unshown_droppable is False, the input response dictionary is returned directly without copying.
• Lazy Branch Cloning: Dictionaries and lists are copied only along branches where unshown null values are actually deleted.

Direct Leaf Wire Serialization

During document folding, merged record trees are serialized into wire format before Pydantic validation. Previously, _instance_wire relied on model_dump(mode="json"), which incurred significant Pydantic export overhead.

Datagrunt 5.3.9 introduces _scalar_to_wire() and _dump_leaf():

• Direct $O(1)$ conversion of scalar types (Enum, Decimal, datetime, date, float) to JSON wire primitives.
• Bypasses Pydantic schema serialization passes for leaf field values during document folding.

from datagrunt.rationalization.folding import _scalar_to_wire
from decimal import Decimal
from datetime import date

# Instantaneous leaf serialization to wire primitives
assert _scalar_to_wire(Decimal("123.45")) == "123.45"
assert _scalar_to_wire(date(2026, 11, 13)) == "2026-11-13"

3. Pre-Indexed String Grounding & Lazy Text Lookups

Single-Pass Alphanumeric Indexing

Grounding validates that extracted numbers, strings, and code tokens (such as part numbers or serial codes) exist within cited document text blocks. For alnum_upper normalizers, code tokens with spaces or punctuation require boundary-safe matching.

Datagrunt 5.3.9 optimizes grounding via _index_alnum_field():

• Pre-indexes document element text into (text, joined_upper, source_indices).
• Eliminates repeated string allocations during citation verification loops.
• _UnitTexts defers plain text and numeric set construction until element keys are actively queried (_key_found).

4. Summary of Bug Fixes in Release 5.3.9

Datagrunt 5.3.9 includes 5 targeted fixes for rationalization pipeline edge cases:

1. Informational Field Folding in Unkeyed Child Models: Informational fields (marked with informational=True in field specs) within unkeyed child list models now fold across extraction units without triggering spurious field conflict warnings.
2. Source Skip Rebuild & Field Set Preservation: Rebuilding folded records now checks _same_raw_model() before re-running Pydantic validation. This prevents running validators twice on identical raw models, maintaining model_fields_set flags and preventing double execution of custom mode="before" validators.
3. Multimodal Unsent Vision Heading Scope Leak Fix: When a vision unit is skipped due to image render failure, scope paths from unsent vision headings no longer leak into subsequent digital page unit source metadata.
4. Multimodal Cross-Family & Peer Quote Counter Rules: Corrects multimodal peer quote grounding rules so that quote counters never sum across multiple peer models, same-family models, or unknown model families.
5. must_agree Quorum Checks on Informational & List Fields: Properly enforces quorum consensus checks (_models_differ_on_must, _unkeyed_list_must_disagrees) across must_agree paths targeting unkeyed list items and nested model list children.

Verification & Upgrade

To upgrade to Datagrunt 5.3.9 using uv:

uv pip install --upgrade datagrunt

Verify your installed version in Python:

import datagrunt
print(datagrunt.__version__)  # 5.3.9