Halantir

Halantir Insight

Extract budget data from PDFs: a reviewable workflow

Route text and scanned budget pages, inspect extracted tables, define a schema and reconcile amounts while preserving a path back to the source.

2026-08-17 1110 words Public data engineering

Not the record · nothing below carries a receipt · written by machine, published under HEIMLANDR · findings live on the record

Begin a budget extraction project with one document, one table and a written definition of the rows you need. Keep the original PDF alongside the extracted data so a reviewer can trace an amount back to its source. The workflow below is a proposed implementation plan, not a claim that a particular government report mandates these tools or that an extraction has already been tested. As noted in analyses of open data debt, treating PDFs as primary data sources creates long-term technical liabilities that require rigorous validation to mitigate.

Inspect the document before choosing a route

Identify whether your source is a native PDF with embedded characters, a scanned image requiring OCR, or a hybrid mix of both. Native files preserve coordinates but may suffer from broken reading orders, while scanned pages introduce skew and noise that defeat standard text layers. This distinction determines whether you can proceed directly to extraction or must first implement an optical character recognition pipeline.

Try selecting text on representative pages, then compare the extracted characters with the visible headings and amounts. Check more than the cover page. Record which tables have usable text, which need optical character recognition, and which need manual transcription or a different source file. Modern comparisons of PDF data extraction software highlight that hybrid PDFs · common in real document packets · often defeat pipelines that rely solely on the PDF text layer or only OCR.

The pdfplumber documentation says the library works best with machine-generated PDFs. It does not provide OCR and does not offer strong support for table extraction from OCR-processed documents. For scanned pages, therefore, plan a separate OCR or transcription step and a review of its output before attempting the same extraction procedure.

Extract one table and inspect its boundaries

Select the largest table or all detected tables on a page using explicit library functions, then visually verify the boundaries against the source PDF. Do not assume the default detection aligns with logical rows, especially in complex layouts where headers span multiple columns or cells merge across rows. Save your extraction settings to ensure reproducibility across the document.

For a suitable text-based page, pdfplumber provides table extraction and visual debugging tools. Its table documentation distinguishes extract_table(), which selects the largest table, from extract_tables(), which returns all detected tables on a page.

Choose deliberately when a page contains several tables. Inspect the detected boundaries, headings and row alignment against the PDF. Save the selected page area, extraction settings and library version with your working files. Repeat this check on a page with a different layout before applying the settings to the rest of the document.

Define a reviewable budget row

Structure your dataset with explicit fields for source reference, account code, measure type, and original amount text to enable full traceability. Avoid assigning categories based solely on proximity to headers; instead, define how spanning headings apply to each extracted value. Preserve raw text to allow reviewers to interpret separators, negative values, or scale indicators like "thousands" independently.

As a starting design, keep these fields in your working dataset:

  • Source document reference, page number and original row label.
  • Reporting organization, budget period and program or account code.
  • Measure, such as proposed budget, revised budget or reported spending.
  • Original amount text, parsed amount, currency and stated unit scale.
  • Review status and a note for unresolved interpretation.

Adapt that list to the document. Do not assign a category solely from a neighboring row. If a heading spans several columns, write down how you apply it to each extracted value. Preserve the original text when interpreting separators, negative values or amounts stated in thousands.

Validate structure without inventing values

Apply a formal schema to define field types, constraints, and missing value indicators before processing any data. This logical representation separates the ideal data structure from the physical text found in the PDF, ensuring that empty cells or dashes are handled consistently rather than silently converted to zeros. Use primary keys that combine account, period, and measure to uniquely identify each row.

The Frictionless Table Schema specification describes field names, types and validation constraints. It also provides a missingValues setting for strings treated as null. A primary key can consist of one field or a combination of fields.

Use an explicit schema to check the dataset you intend to publish. Choose a row identifier that distinguishes the measure and period as well as the account. Document the meaning of an empty cell or dash before mapping it to null; do not silently turn it into zero. Put ambiguous rows in a review file with their source references.

Reconcile against the source

Compare extracted subtotals and totals against the printed figures in the source document to detect mapping errors or missing rows. Confirm that the period, unit scale, and measure match exactly before calculating differences, and treat any mismatch as a signal to inspect the extraction logic rather than forcing agreement. Record rounding tolerances and exclusions explicitly in your validation report.

For each reviewed table, compare the extracted rows with its printed subtotal and total. First confirm that you are comparing the same period, unit scale and measure. Keep subtotal rows separate from detail rows when calculating a check, and record any exclusions.

Treat a mismatch as a reason to inspect the mapping. Do not change a value just to force agreement. Likewise, treat a matching total as one check in the review, alongside row labels, missing lines and column alignment. Define any rounding tolerance before using it and retain the difference in the validation report.

Publish the data with its review record

Release the cleaned dataset alongside its schema, source references, and a detailed record of known limitations and unresolved ambiguities. Preserve previous versions when new editions arrive, and highlight changes rather than replacing data silently. Ensure that any rows excluded from the final analysis remain visible in accompanying notes to maintain transparency and avoid presenting unverified amounts as facts.

Release the reviewed CSV or JSON together with its schema, source references, extraction instructions and known limitations. Record the source edition and the date you retrieved it. When a new edition arrives, preserve the previous dataset and review the changes before replacing the published version. Make unresolved rows visible in the accompanying notes rather than presenting them as verified amounts.