Datagrunt 5.4.5: Linear-Time CSV Dialect Sniffing on Long Lines
Datagrunt 5.4.5 is now available! This release eliminates regex backtracking and stack limits in the native Rust CSV dialect sniffer on very long lines (PR #514, #381) and hardens the automated release-post generation and validation pipeline (PR #509, #336).
Key Highlights in Datagrunt 5.4.5
- Zero Backtracking in Rust CSV Dialect Sniffing — Replaced all backtracking
fancy-regexpaths indialect.rswith linear-time matching so Rust and the pure-Python reference oracle agree on arbitrarily long lines up toMAX_LINE_CHARS(2,097,152 characters per physical line) without sample truncation. - Direct
O(n)Scanners for Pattern 1 & Doubled Quotes — Quote pattern 1 (tally_pattern_1) and whole-field doubled-quote detection (has_doubled_quote) now execute as direct byte/UTF-8 linear scanners, eliminating\k<delim>backreferences, lookbehind assertions, and per-call regex compilation. - Automata Delegation for Quote Patterns 2–4 — Expanded
"and'into a disjoint, backreference-free alternation (QUOTED_FIELD) sofancy-regexdelegates patterns 2–4 directly toregex-automatawith no backtrack or stack limits. - Expanded Long-Line Differential Parity Coverage — Added Rust unit tests on 1.2M-character adversarial lines,
backtrack_limit(1)verification proving zero backtracking, and 12 long-line differential parity tests (500k–1.2Mcharacters) intests/parity/test_parity_regressions.py. - Hardened Release-Documentation Validation —
scripts/antigravity_workflow.pynow enforces dashed version filenames (datagrunt-{slug}-*.md), dynamic year/month blog paths, strict Markdown link checking, and static AST import verification for Python snippets againstsrc/datagrunt.
Why Long CSV Lines Previously Diverged
Datagrunt’s CSV header probe (probe_csv_header) reads up to 5 sample lines of up to MAX_LINE_CHARS (2 * 1024 * 1024 characters) each. Previously, the Rust sniffer in rust/datagrunt-core/src/dialect.rs evaluated CPython’s four quote-guessing patterns and the whole-field doubled-quote check (gh-98820, gh-109638) using fancy-regex features that triggered its backtracking engine:
- Quote pattern 1 used a
\k<delim>backreference to verify that the closing delimiter matched the opening delimiter. - Quote patterns 2–4 used a
\k<quote>backreference and(?!\k<quote>).negative lookahead insideQUOTE_OTHER. doubled_quote_regexcompiled a per-call regex with a(?<=delim)lookbehind to detect""or''inside whole fields.
Because fancy-regex enforces a default limit of 1,000,000 backtracks and a 1,000,000-entry backtrack stack (whereas CPython’s re has neither limit), a single scan over roughly 250k–1M characters—such as a wide CSV header, a long JSON/text blob column, or an adversarial quote run—could hit a runtime error. While the sniffer already caught that error gracefully instead of panicking across the FFI boundary, aborting the scan early meant Rust could return a different dialect than the Python oracle.
Linear-Time Matching Architecture in 5.4.5
Datagrunt 5.4.5 removes every backtracking code path from dialect.rs:
- Direct
O(n)Pattern 1 Scanner (tally_pattern_1): Scans left-to-right for a delimiter character (is_delim_char, using an ASCII fast path plus a single-characterNON_ASCII_DELIM_CHARcheck overPY_WORD = \p{L}\p{N}_), an optional space, and an opening quote ("or'). It locates the first undoubled closing quote viafind_closing_quoteand checks whether the trailing bytes match the opening delimiter. Becausedelimexcludes both quotes, the first undoubled quote is the unique possible closer, and caching a shareddelim + ' 'probe keeps total scanning strictlyO(n). - Backreference-Free Alternation for Patterns 2–4 (
COMPILED_QUOTE_PATTERNS_2_TO_4): Expanding(?P<quote>["'])into a disjoint two-branch alternation (dqfor"..."andsqfor'...') removes\k<quote>and(?!\k<quote>). With zero lookaround or backreferences,fancy-regexdelegates all three patterns straight toregex-automatain guaranteed linear time—verified in CI by recompiling each pattern with.backtrack_limit(1). - Direct
O(n)Doubled-Quote Scanner (has_doubled_quote): Replacesdoubled_quote_regexwith a direct whole-field scanner reproducing CPython’sfinditernon-overlapping semantics without compiling a regex on everysniff()call.
For Python users, CSVReader, CSVWriter, and CSVDialect automatically benefit from the updated native core with no API changes:
from pathlib import Path
from tempfile import TemporaryDirectory
from datagrunt import CSVReader
from datagrunt.core import CSVDialect
with TemporaryDirectory() as tmpdir:
csv_path = Path(tmpdir) / "wide_payload.csv"
long_note = "x" * 600_000
csv_path.write_text(f'id,note,status\n1,"{long_note}""ok""",active\n')
# Sniffs delimiter, quotechar, and doublequote in linear time across 600k+ char lines
dialect = CSVDialect(csv_path, delimiter=",")
assert dialect.quotechar == '"'
assert dialect.doublequote is True
reader = CSVReader(csv_path, engine="polars")
df = reader.to_dataframe()
assert df.shape == (1, 3)You can still switch to the pure-Python reference implementation at any time for A/B diagnostics via DATAGRUNT_DISABLE_RUST=1 (see CSV Engines & Rust Acceleration):
import os
from datagrunt import CSVReader
os.environ["DATAGRUNT_DISABLE_RUST"] = "1"
reader = CSVReader("data.csv")
df = reader.to_dataframe()Verification & Parity Testing
Both the Rust and Python test suites now pin long-line parity across all four quote patterns and doubled-quote edge cases:
- Rust unit tests (
rust/datagrunt-core/src/dialect.rs):adversarial_lines_stay_within_fancy_regex_runtime_limitsraised from 128k to 1,200,000 characters across all patterns and doubled-quote checks.compiled_patterns_do_not_use_backtracking_engineverifies patterns 2–4 succeed under.backtrack_limit(1).is_delim_char_matches_regex_class_for_ascii_and_unicode,pattern_1_handles_space_fallback_and_multibyte_delimiters,has_doubled_quote_matches_whole_field_non_overlapping_semantics, andsniff_handles_long_lines_past_old_backtrack_limits.
- Differential parity suite (
tests/parity/test_parity_regressions.py):test_long_line_sniff_and_probe_paritytests 12 long-line cases (500k–1.2M characters) and asserts exact equality betweendatagrunt._nativeand the pure-Python oracle for bothprobe_csv_headerandsniff_dialect.
Release Automation Hardening
This release also includes internal release-engineering improvements in scripts/antigravity_workflow.py and tests/test_workflows.py (#336):
- Computes the target Hugo blog directory dynamically from the release year and month and normalizes dotted release-post filenames into the dashed
datagrunt-{slug}-*.mdformat expected by CI deployment checks. - Validates generated Markdown links to permit only
https://URLs, site-relative paths, and anchors, rejecting local file URIs and insecure or scripted schemes. - Runs static AST validation on fenced Python code blocks to ensure syntax validity and verify that
datagruntimports reference real symbols exported bysrc/datagrunt.
Upgrading
Upgrade to Datagrunt 5.4.5 today using uv:
uv pip install --upgrade datagruntOr with PDF and rationalization extras:
uv pip install --upgrade "datagrunt[pdf,rationalize]"For more details on Datagrunt’s Rust CSV core and parity testing, see CSV Engines & Rust Acceleration.