Datagrunt 5.4.1: Exposing Per-Model Audit Outputs & Flat By-Model Exports
Datagrunt 5.4.1 is now live! This release introduces structured per-model audit visibility and flat per-model audit exports across the rationalization pipeline (PR #500).
Key Highlights in Datagrunt 5.4.1
- Structured Per-Model Audit Data (
ModelOutput) —GroundedRecordinstances now exposesupporting_modelsand typedmodel_outputs(ModelOutputdataclass), capturing every model’s exact votes, grounding quotes, cited element IDs, and model-specific grounding reasons. - Grounded-Union Provenance & Model Outputs — DataFrame, Arrow, dict, and JSONL exports with
include_grounded_union=True(or explicitinclude_model_outputs=True) automatically append_supporting_modelsand nested_model_outputscolumns. - Flat Per-Model Audit Exports (
by_model=True) — Passby_model=Truetoto_dataframe(),to_dicts(),to_arrow_table(), orto_jsonl()to explode records into individual per-model audit rows (_model_id,_model_family,_vote_index,_status,_doc_id,_is_grounded,_grounding_quotes,_element_ids). - Disputed Item Audit Inspection — Setting
include_disputed=Truealongsideby_model=Truepreserves individual model votes even on ungrounded or disputed items where consensus producedrecord=None.
1. Accessing Per-Model Audit State (ModelOutput)
When extracting entities across multiple LLM/VLM providers using rationalize() or rationalize_many(), Datagrunt computes cross-family consensus and element grounding. In Datagrunt 5.4.1, every folded GroundedRecord carries complete audit metadata per provider model:
from datagrunt import PDFReader, rationalize
from pydantic import BaseModel
class PartNumber(BaseModel):
part_id: str
quantity: int
doc = PDFReader("schematic.pdf").document
result = rationalize(
doc,
schema=PartNumber,
prompt="Extract all part numbers and quantities.",
models=["openai:gpt-5.4", "anthropic:claude-sonnet-4-6", "gemini:gemini-2.5-flash"],
min_families=2,
)
# Inspect per-model audit outputs on folded grounded records
for record in result.grounded_records:
print(f"Record Key: {record.key_value}")
print(f"Supporting Models: {record.supporting_models}")
for mo in record.model_outputs:
print(f" Model: {mo.model_id} ({mo.model_family})")
print(f" Supported: {mo.supported}, Grounded: {mo.is_grounded}")
print(f" Votes: {mo.votes}")
print(f" Quotes: {mo.grounding_quotes}")
print(f" Element IDs: {mo.element_ids}")
if mo.grounding_reasons:
print(f" Grounding Failures: {[r.reason for r in mo.grounding_reasons]}")Technical Design:
- Privacy & Formatting Protection: Text-bearing fields (
votes,grounding_quotes,element_ids,grounding_reasons) are omitted fromModelOutput.__repr__to prevent terminal log pollution. - Defensive Deep Copy (
to_dict()): Callingmo.to_dict()returns a pure, stdlib-JSON-serializable deep copy of the model’s extraction and audit state.
2. Exporting Grounded Union Records with Model Outputs
When analyzing extraction results in Polars or Pandas DataFrames, include_grounded_union=True includes strict consensus items alongside element-grounded single-family catches ("missing_in_some_models" with zero grounding issues).
In Datagrunt 5.4.1, grounded-union exports include _supporting_models and _model_outputs by default:
# Export grounded union to Polars DataFrame with typed _model_outputs
union_df = result.to_dataframe(include_grounded_union=True)
print(union_df.schema)
# Struct columns include _status, _doc_id, _supporting_models, _model_outputs
# Atomic UTF-8 JSONL export with full provenance and model outputs
result.to_jsonl("out/extracted_parts.jsonl", include_grounded_union=True)3. Flat Per-Model Audit Exports (by_model=True)
For down-stream auditing, LLM benchmarking, or model disagreement troubleshooting, you can unroll records into flat per-model rows by setting by_model=True:
# Explode items into flat per-model audit rows
by_model_df = result.to_dataframe(include_grounded_union=True, by_model=True)
# Filter to inspect votes from a specific provider
anthropic_votes = by_model_df.filter(by_model_df["_model_family"] == "anthropic")
for row in anthropic_votes.iter_rows(named=True):
print(f"[{row['_model_id']}] Doc {row['_doc_id']} Part {row['part_id']}: grounded={row['_is_grounded']}")Unrolled Audit Schema (by_model=True):
When by_model=True is passed to to_dataframe(), to_dicts(), to_arrow_table(), or to_jsonl(), each output row represents one (item, model_id, vote) tuple with metadata columns:
_model_id: Canonical model string (e.g."openai:gpt-5.4")._model_family: Model provider family (e.g."openai","anthropic","google")._vote_index: Zero-based vote index within the model’s contribution to the unit._status: Record status ("consensus","single_model","field_disagreement", etc.)._doc_id: Source document identifier._is_grounded: Boolean flag indicating zero grounding failures for this model._grounding_quotes: Grounding quotes cited by this specific model._element_ids: Source element IDs cited by this specific model.
Installation
Upgrade to Datagrunt 5.4.1 today using uv:
uv pip install --upgrade "datagrunt[pdf,rationalize]"For complete usage documentation, check out the PDF Parsing & Rationalization Guide.