Extracting data from unstructured documents: design before tooling

According to IDC, over 90% of the data generated by organizations in 2022 was unstructured: information stored in documents, images, emails, video, presentations, and notes. That translated to over 57k exabytes (1 exabyte = 1 million terabytes) of information in a single year.

This data is a goldmine of organizational context, and critical to its operations. As the paper highlights, it is central to supply chain operations, product roadmaps, sales and marketing activity, financial records, and the distilled internal decision making across a company's employees. IDC estimates that only a fraction is ever reviewed to extract information of value. The reasons cited are limited understanding across IT, an inability to quantify ROI, and a shortage of experience working with unstructured data.

Most of that data lives in PDFs. Amongst authoritative sources on .gov and .edu domains, PDFs account for over 80% of published text, and it remains the default format for official government and medical communication. There is good reason for its dominance: it is portable, layout-stable, secure, and renders identically across devices. But notice what those properties have in common. Every one of them serves human, visual consumption. PDF was built for presentation, not for data storage.

The upload instinct, and its limits

Consumption patterns are now changing. LLMs, and the autonomous agents built on them, increasingly do what humans used to: read, understand, and infer from PDF content. So the first instinct in many organizations is simply to upload documents to an LLM and have it reason over them.

That instinct, however, comes with some real limits. Re-running a large document to extract new information is costly, because the input token count is the same on every pass, and wasteful when the data you need is a small snippet in a predictable location. It is hard to trace how an LLM reasons over content, which makes debugging painful, and coordinating multiple agents to split the work only adds expensive hand-off design. Because LLMs are non-deterministic, the same input is not guaranteed to produce the same output twice. And many organizations simply cannot upload the documents at all; in healthcare, finance, and government, that can be a privacy breach, while hosting models in-house to avoid the problem carries its own cost.

The tooling landscape offers no clean answer either. "Heavyweight" cloud APIs mean sending the document to a vendor's server, importing privacy exposure, ongoing cost, and vendor lock-in. "Lightweight" local parsers, like open source Python libraries, are fast but blind to layout; they cannot see the tables, headers, and paragraphs that carry the document's information hierarchy, and they lack the OCR needed for scanned or image-heavy pages. Open-source frameworks such as Marker and Docling run ML models locally and split the difference, but they can be slow and tend to falter as document complexity climbs.

Design for the problem before the tool

The usual impulse is to lead with tooling. It's worth stepping back and designing for the specific problem first. You can arrive at the right approach by looking at three aspects of the problem: the use case, the structure, and the content.

1. Use case
   • Who are the users, what do they do with the file's content, and which parts are most critical to their work?
   • How quickly does a document update need to flow into downstream processes?
   • What level of extraction accuracy is expected, and how critical is it?
   • Is the use case operational (introspecting a specific file) or analytical (reasoning across many documents)?
2. Structure
   • How consistent are the documents? Do they follow templates, or vary widely?
   • What structural anomalies exist, and therefore which elements should simply be extracted without interpretation?
   • Are the documents digitally created with recognizable characters, or scanned images of physical originals?
   • Do you need structured data (tables)? Are they multi-page, multi-column, with sidebars, merged cells, colored cells, captions, or tables nested inside tables?
3. Content
   • Do the documents contain PII or sensitive information, and must that be handled under governance policy before or after extraction?
   • Are there markers or checks that support classification before further processing?
   • How much of the context around an image or table matters to the extracted content, and is that context reliably colocated and annotated?

As you answer these questions, you quickly realize that the setup to process vendor receipts at scale looks nothing like the setup for clinical trial records, which in turn looks nothing like architectural PDFs containing technical blueprints. So a unified approach to building re-usable architecture is set up for failure, or at best, supports a very narrow set of documents. What each of these does have in common, though, is the need for accuracy balanced against cost and speed.

Accuracy is the hardest to solve

Two things signal extraction health: completeness (did we get everything we wanted?) and correctness (is what we got right?).

Completeness is the more mechanical check. Did the table extraction return more than one row? Was the invoice number captured? These are the obvious guardrails, and they catch the obvious failures.

Correctness is harder. It requires domain knowledge (is the extracted data point plausible?) and internal consistency (does it reconcile with other sources, inside or outside the document?). Both of these can best be tackled through careful intermediate representations. These are machine-readable outputs that you can inspect to make surgical improvements in extraction logic, code, or workflow. They also include provenance information in the output, i.e. the specific page, table, row/column, or pixel region that any result can be traced back to and investigated. Implementations that put these checks at the center of their design, rather than bolting them on later, hold up far better as they scale.

Then, cost and speed

Every run should write a consistent log, with runtime statistics for each stage of extraction. Processing documents in parallel speeds things up sharply, especially when writing to an idempotent artifact that stores both run history and current state. Those same logs surface the compute-heavy stages, which can then be re-approached individually based on the actual blocker, and they expose cost per stage by tracking input and output tokens across LLM calls.

Where this leads

Most of the innovation inside data teams still centers on structured data. There is far more to gain from unstructured data, and from PDFs in particular. A well-designed approach automates hours of manual fact-finding, lowers cost, and frees capacity for the strategic, data-driven decisions the data was supposed to enable in the first place.

But the design is the hard part, and it is easy to get wrong the first time by leading with a tool instead of the use case. That is the work we do at Cruxdata: mapping the use case, structure, and content before a line of extraction logic gets written, then building the pipeline that holds up on accuracy, cost, and speed as volume grows.

If your team is sitting on documents it cannot yet turn into data, that is a conversation worth having.