Datagrunt 5.4.5: Linear-Time CSV Dialect Sniffing on Long Lines

November 21, 2026by Martin Graham

Datagrunt 5.4.5 is now available! This release eliminates regex backtracking and stack limits in the native Rust CSV dialect sniffer on very long lines (PR #514, #381) and hardens the automated release-post generation and validation pipeline (PR #509, #336).


Key Highlights in Datagrunt 5.4.5

  1. Zero Backtracking in Rust CSV Dialect Sniffing — Replaced all backtracking fancy-regex paths in dialect.rs with linear-time matching so Rust and the pure-Python reference oracle agree on arbitrarily long lines up to MAX_LINE_CHARS (2,097,152 characters per physical line) without sample truncation.
  2. Direct O(n) Scanners for Pattern 1 & Doubled Quotes — Quote pattern 1 (tally_pattern_1) and whole-field doubled-quote detection (has_doubled_quote) now execute as direct byte/UTF-8 linear scanners, eliminating \k<delim> backreferences, lookbehind assertions, and per-call regex compilation.
  3. Automata Delegation for Quote Patterns 2–4 — Expanded " and ' into a disjoint, backreference-free alternation (QUOTED_FIELD) so fancy-regex delegates patterns 2–4 directly to regex-automata with no backtrack or stack limits.
  4. Expanded Long-Line Differential Parity Coverage — Added Rust unit tests on 1.2M-character adversarial lines, backtrack_limit(1) verification proving zero backtracking, and 12 long-line differential parity tests (500k–1.2M characters) in tests/parity/test_parity_regressions.py.
  5. Hardened Release-Documentation Validation — scripts/antigravity_workflow.py now enforces dashed version filenames (datagrunt-{slug}-*.md), dynamic year/month blog paths, strict Markdown link checking, and static AST import verification for Python snippets against src/datagrunt.

Why Long CSV Lines Previously Diverged

Datagrunt’s CSV header probe (probe_csv_header) reads up to 5 sample lines of up to MAX_LINE_CHARS (2 * 1024 * 1024 characters) each. Previously, the Rust sniffer in rust/datagrunt-core/src/dialect.rs evaluated CPython’s four quote-guessing patterns and the whole-field doubled-quote check (gh-98820, gh-109638) using fancy-regex features that triggered its backtracking engine:

  • Quote pattern 1 used a \k<delim> backreference to verify that the closing delimiter matched the opening delimiter.
  • Quote patterns 2–4 used a \k<quote> backreference and (?!\k<quote>). negative lookahead inside QUOTE_OTHER.
  • doubled_quote_regex compiled a per-call regex with a (?<=delim) lookbehind to detect "" or '' inside whole fields.

Because fancy-regex enforces a default limit of 1,000,000 backtracks and a 1,000,000-entry backtrack stack (whereas CPython’s re has neither limit), a single scan over roughly 250k–1M characters—such as a wide CSV header, a long JSON/text blob column, or an adversarial quote run—could hit a runtime error. While the sniffer already caught that error gracefully instead of panicking across the FFI boundary, aborting the scan early meant Rust could return a different dialect than the Python oracle.


Linear-Time Matching Architecture in 5.4.5

Datagrunt 5.4.5 removes every backtracking code path from dialect.rs:

  1. Direct O(n) Pattern 1 Scanner (tally_pattern_1): Scans left-to-right for a delimiter character (is_delim_char, using an ASCII fast path plus a single-character NON_ASCII_DELIM_CHAR check over PY_WORD = \p{L}\p{N}_), an optional space, and an opening quote (" or '). It locates the first undoubled closing quote via find_closing_quote and checks whether the trailing bytes match the opening delimiter. Because delim excludes both quotes, the first undoubled quote is the unique possible closer, and caching a shared delim + ' ' probe keeps total scanning strictly O(n).
  2. Backreference-Free Alternation for Patterns 2–4 (COMPILED_QUOTE_PATTERNS_2_TO_4): Expanding (?P<quote>["']) into a disjoint two-branch alternation (dq for "..." and sq for '...') removes \k<quote> and (?!\k<quote>). With zero lookaround or backreferences, fancy-regex delegates all three patterns straight to regex-automata in guaranteed linear time—verified in CI by recompiling each pattern with .backtrack_limit(1).
  3. Direct O(n) Doubled-Quote Scanner (has_doubled_quote): Replaces doubled_quote_regex with a direct whole-field scanner reproducing CPython’s finditer non-overlapping semantics without compiling a regex on every sniff() call.

For Python users, CSVReader, CSVWriter, and CSVDialect automatically benefit from the updated native core with no API changes:

from pathlib import Path
from tempfile import TemporaryDirectory
from datagrunt import CSVReader
from datagrunt.core import CSVDialect

with TemporaryDirectory() as tmpdir:
    csv_path = Path(tmpdir) / "wide_payload.csv"
    long_note = "x" * 600_000
    csv_path.write_text(f'id,note,status\n1,"{long_note}""ok""",active\n')

    # Sniffs delimiter, quotechar, and doublequote in linear time across 600k+ char lines
    dialect = CSVDialect(csv_path, delimiter=",")
    assert dialect.quotechar == '"'
    assert dialect.doublequote is True

    reader = CSVReader(csv_path, engine="polars")
    df = reader.to_dataframe()
    assert df.shape == (1, 3)

You can still switch to the pure-Python reference implementation at any time for A/B diagnostics via DATAGRUNT_DISABLE_RUST=1 (see CSV Engines & Rust Acceleration):

import os
from datagrunt import CSVReader

os.environ["DATAGRUNT_DISABLE_RUST"] = "1"
reader = CSVReader("data.csv")
df = reader.to_dataframe()

Verification & Parity Testing

Both the Rust and Python test suites now pin long-line parity across all four quote patterns and doubled-quote edge cases:

  • Rust unit tests (rust/datagrunt-core/src/dialect.rs):
    • adversarial_lines_stay_within_fancy_regex_runtime_limits raised from 128k to 1,200,000 characters across all patterns and doubled-quote checks.
    • compiled_patterns_do_not_use_backtracking_engine verifies patterns 2–4 succeed under .backtrack_limit(1).
    • is_delim_char_matches_regex_class_for_ascii_and_unicode, pattern_1_handles_space_fallback_and_multibyte_delimiters, has_doubled_quote_matches_whole_field_non_overlapping_semantics, and sniff_handles_long_lines_past_old_backtrack_limits.
  • Differential parity suite (tests/parity/test_parity_regressions.py):
    • test_long_line_sniff_and_probe_parity tests 12 long-line cases (500k–1.2M characters) and asserts exact equality between datagrunt._native and the pure-Python oracle for both probe_csv_header and sniff_dialect.

Release Automation Hardening

This release also includes internal release-engineering improvements in scripts/antigravity_workflow.py and tests/test_workflows.py (#336):

  • Computes the target Hugo blog directory dynamically from the release year and month and normalizes dotted release-post filenames into the dashed datagrunt-{slug}-*.md format expected by CI deployment checks.
  • Validates generated Markdown links to permit only https:// URLs, site-relative paths, and anchors, rejecting local file URIs and insecure or scripted schemes.
  • Runs static AST validation on fenced Python code blocks to ensure syntax validity and verify that datagrunt imports reference real symbols exported by src/datagrunt.

Upgrading

Upgrade to Datagrunt 5.4.5 today using uv:

uv pip install --upgrade datagrunt

Or with PDF and rationalization extras:

uv pip install --upgrade "datagrunt[pdf,rationalize]"

For more details on Datagrunt’s Rust CSV core and parity testing, see CSV Engines & Rust Acceleration.