Datagrunt 5.0: A New PDF Engine

Datagrunt 5.0 rebuilds PDF extraction from the ground up. PDFReader and PDFWriter keep their names, but what they read and write is new, so this is a major release. CSV, Excel, and Parquet are unchanged. If you only use those, upgrading needs no code changes beyond the new floor: Datagrunt 5.0 requires Python 3.11 or newer.

Coming from 4.x with PDFs? Start with the migration guide, which maps every 4.x option and output to its 5.0 form. Applications that can’t migrate yet can pin datagrunt[pdf]<5.

One engine, a typed document

Every PDF is now read by one engine, PDFium (via pypdfium2), into a typed document:

• Elements: headings, paragraphs, key-value pairs, tables, lists, and figures, in reading order.
• Per element: its page, its bounding box (in points, top-left origin), and the headings it sits under (scope_path).
• Sections: grouped under their headings.
• Split tables: a table that runs across a page break is linked to its continuation.
• Heading levels: ranked over the whole document by font size and boldness, so a level means the same on every page.
• Page images: every page that can be drawn has one, drawn only when you read it.

from datagrunt import PDFReader

reader = PDFReader("report.pdf")
df = reader.to_dataframe()        # one row per element, in document order
for table in reader.tables:
    print(table.scope_path, table.to_dataframe())
reader.query_data(f"SELECT kind, count(*) AS n FROM {reader.db_table} GROUP BY kind").pl()

PyMuPDF, pdfplumber, and pytesseract are gone. The [pdf] extra installs only pypdfium2 and Pillow, and import datagrunt works without it.

Hidden text is flagged, never dropped

Text a reader can’t see is kept and flagged in hidden_reasons. The reasons cover:

• text that is off the page, invisible, too small, transparent, or the colour of what’s behind it;
• duplicates of other text;
• a scanned page’s own text layer, kept but superseded by OCR.

Markdown leaves hidden elements out by default (include_hidden=True keeps them behind a comment), and JSON and the DataFrame keep everything. Filter on the column before you pass text to an LLM:

import polars as pl

visible = reader.to_dataframe().filter(pl.col("hidden_reasons").list.len() == 0)

Scans and OCR

Each page is routed digital, hybrid, or vision. A vision page is a scan, or a page with too little text or mostly unreadable characters. It is rendered and read by the system Tesseract program, with orientation detection and a per-page time budget.

A page OCR reads at low confidence (a mean word confidence under 70) carries a warning that it may be turned the wrong way or unreadable, so you know not to trust its text as is.

Every page reports its status: ok, ocr_fallback, ocr_unavailable, or error. A page that can’t be read becomes an error page with its reason, and the rest of the document still extracts. Pass strict=True to raise instead.

Reading writes nothing; writing is atomic

• Reading: PDFReader writes nothing to disk, not even temporary files. Figures load only when you ask for them.
• Writing: every PDFWriter output is written atomically, so an interrupted write never leaves a partial file. Every method takes an explicit path.
• New outputs:
  • write_parquet and write_csv;
  • write_bundle, a checksummed directory of JSON, JSON Lines, Markdown (per document and per page), figures, and page images that PDFReader loads back and verifies. Writing the same bundle again reuses it; overwrite=True replaces it.

from datagrunt import PDFWriter

writer = PDFWriter("report.pdf")
writer.write_json("out/report.json")       # lossless
writer.write_parquet("out/report.parquet")
writer.write_bundle("out/report_bundle")

Guarded rendering

Every PDF is parsed in worker processes with a memory cap, on every platform: your own process never loads the PDF engine. A page that passes the cap or its time bound becomes an error page with the reason, and the rest of the document is read by a new worker. Workers run as plain subprocesses, so workers=N needs no if __name__ == "__main__": guard.

Upgrading

uv pip install --upgrade "datagrunt[pdf]"

Install Tesseract separately for OCR (see Optional PDF Support), and read the migration guide before upgrading PDF code.