Datagrunt 5.3.13: AutoCAD SHX Vernier Backing & Symmetric Scan-Witness Grounding
Datagrunt 5.3.13 is now live! This release brings targeted quality and rationalization upgrades for CAD engineering drawings and multimodal VLM workflows (PR #497).
Key Highlights in Datagrunt 5.3.13
- AutoCAD SHX Render Mode 3 Vernier Backing — Restores invisible text backing (
render_mode = 3) for CAD vector glyph overlays under non-uniform SHX font pitch and Vernier scale crossover (e.g. 12.0pt vs 12.5pt pitch straddling). - Stroke-Continuous Void Verification — Merges stroke x-intervals (
_token_glyph_intervals) and verifies unstroked gaps to distinguish proportional SHX kerning/pitch expansion from genuinely omitted character voids. - CJK & Multi-Word SHX Overhang Recovery — Supports edge character backing for CJK ideographs, bracketed tokens (
"[INCHES]"), and multi-word CAD title block spans ("MFG LOCATION","原来为02XF45005701。"). - Adversarial Hidden Text Protection — Preserves strict security checks so that omitted middle digits (e.g.,
"105"with missing'0'), omitted signs ("-100"), decimal points ("1.5"), negation words ("NOT","CANT"), polarity'A'prefixes, and Cyrillic homoglyphs ("N\u041eTE") remain safely flagged withhidden_reasons=["invisible"]. - Symmetric Cross-Family Scan-Witness Grounding — Ensures peer scan witnesses (
_peer_superset_matches) and bridged numeric identifiers (e.g."04-60008-27","PN-104") evaluate grounding symmetrically across LLM/VLM model providers (OpenAI, Anthropic, Gemini, Ollama).
1. AutoCAD SHX 3 Tr Vernier Backing & Stroke Void Verification
In CAD software exports (such as AutoCAD drawings), text labels are rendered using stroke vectors alongside invisible text overlays (render_mode = 3) for searchability. Because SHX fonts (like Txt or Simplex) use proportional character widths and Vernier scale pitch variations (e.g. 12.0pt vs 12.5pt pitch straddling), individual glyph character boxes can experience slight horizontal position drift or crossovers relative to vector strokes.
In previous releases, these tiny kerning/Vernier shifts could cause Datagrunt’s quality classifier to misidentify characters in part numbers (such as 'F' in "02XF44014601", 'T', or leading 'N') as missing vector strokes, incorrectly marking them with hidden_reasons=["invisible"].
Datagrunt 5.3.13 introduces a stroke interval merging engine (_merge_x_intervals) and void verifier (_gap_has_no_unstroked_void):
from datagrunt import PDFReader
# Read AutoCAD engineering drawing PDF containing SHX vector text overlays
reader = PDFReader("cad_drawing_02XF44014601.pdf")
df = reader.to_dataframe()
# Inspect visible part number elements extracted without false hidden flags
visible_elements = df.filter(df["hidden_reasons"].list.len() == 0)
for row in visible_elements.iter_rows(named=True):
print(f"[{row['element_id']}] {row['kind']}: {row['text']}")Technical Mechanism:
T7FEZJLt+Backing Rules:_box_backs_charnow accepts horizontal stroke widths $0.45 \cdot cw \le vw \le 1.6 \cdot cw$ and height $vh \le 0.5 \cdot ch_h$ for cap-bar and crossbar characters without falsely matching full-width table borders.- Glyph Interval Analysis (
_token_glyph_intervals): Merges glyph-scale stroke intervals across token indices and evaluates local gap continuity ($max_gap \le 1.15 \cdot cw$). - Expanded Edge Digit & Leading
'N'Recovery:_expanded_part_edge_digit_okand_n_lead_continuousallow proportional SHX expansion to continuously back edge digits and uncorroborated leading'N'characters when physical stroke count matches token length.
2. Adversarial Security: Hidden Characters Stay Hidden
While recovering valid SHX kerning drift, Datagrunt 5.3.13 maintains strict verification rules to prevent adversarial text hiding techniques (e.g. hiding digits or signs to manipulate financial figures or engineering specs):
from datagrunt import PDFReader
reader = PDFReader("drawing_with_omitted_digit.pdf")
# Omitted middle digit in "105" (missing stroke for '0') is safely flagged as "1_5"
hidden_rows = reader.to_dataframe().filter(reader.to_dataframe()["hidden_reasons"].list.len() > 0)
for row in hidden_rows.iter_rows(named=True):
print(f"Flagged hidden element: {row['text']} (reasons: {row['hidden_reasons']})")The table below outlines how Datagrunt handles SHX kerning vs. adversarial omissions:
| Scenario | Raw Text | Vector Strokes Present | Classifier Result | Reason |
|---|---|---|---|---|
| SHX Vernier Pitch | "02XF44014601" |
12 continuous stroke intervals | Visible | Stroke continuity verified across Vernier straddle |
| SHX Cap-Bar ‘T’ | "T" |
6.4 pt x-bar stroke | Visible | Cap-bar stroke matches _box_backs_char |
| Omitted Middle Digit | "105" |
2 strokes (missing '0') |
Hidden ("invisible") |
Unstroked void exceeds $1.15 \cdot cw$ threshold |
| Omitted Sign | "-100" |
3 digit strokes (missing '-') |
Hidden ("invisible") |
Leading sign has no backing stroke |
| Negation Word | "NOT" |
2 strokes (missing 'N') |
Hidden ("invisible") |
Leading 'N' lacks continuous stroke count |
| Cyrillic Homoglyph | "N\u041eTE" |
3 ASCII strokes | Hidden ("invisible") |
Non-ASCII homoglyph missing physical stroke |
3. Symmetric Cross-Family Scan-Witness Grounding
In rationalize(), multimodal extraction records are grounded against document source elements using multi-model consensus and peer witness validation (_PeerWitness).
Datagrunt 5.3.13 updates _peer_superset_matches and _witness_token_corroborated to operate symmetrically across model providers (OpenAI, Anthropic, Gemini, Ollama):
from pydantic import BaseModel
from typing import Annotated
from datagrunt import Key, rationalize
from datagrunt.rationalization import AnthropicProvider, GeminiProvider, OpenAIProvider
class PartSpecification(BaseModel):
# Bridged numeric code identifier (e.g., "04-60008-27" or "PN-104")
part_number: Annotated[str, Key("part", normalizer="alnum_upper")]
description: str
# Grounding against multi-provider vision models
result = rationalize(
"scanned_blueprint_bundle/",
schema=PartSpecification,
prompt="Extract all part numbers and descriptions from the blueprint.",
models=[
OpenAIProvider("gpt-4o"),
AnthropicProvider("claude-3-5-sonnet-20241022"),
GeminiProvider("gemini-2.5-flash"),
],
min_families=2,
)
print(f"Grounded records: {len(result.records)}")
for item in result.records:
print(item)Highlights of Symmetric Grounding (#497):
- Bridged Code Identifiers (
_is_code_identifier): Automatically recognizes multi-character alphanumeric and bridged numeric identifiers with hyphens, underscores, slashes, or dots (e.g.04-60008-27,PN-104,V1/2.0). - Symmetric Witness Containment (
_peer_superset_matches): Evaluates peer scan witnesses ($\ge 0.85$ token containment) symmetrically against non-textless elements and strict elements, eliminating asymmetric false rejections when peer vision models ground valid code quotes from scanned page regions.
Summary of Changes
| Component | Status | Description |
|---|---|---|
datagrunt.pdf.quality |
Enhanced | Restored AutoCAD SHX 3 Tr Vernier backing, _merge_x_intervals, _token_glyph_intervals, and _gap_has_no_unstroked_void checks. |
datagrunt.rationalization.grounding |
Enhanced | Added _peer_superset_matches, symmetric _witness_token_corroborated, and expanded _is_code_identifier for bridged numeric codes. |
| PyPI Version | 5.3.13 | Released and available immediately (uv pip install --upgrade datagrunt). |
For further details, explore the PDF Parsing Guide and API Reference.