Section 1
Three different tasks under one heading
Classification: what kind of document is this. Cheap, reliable, and usually the fastest win because misfiled documents create work everywhere downstream. Extraction: pull specific fields out. Invoice number, total, date, party names, renewal terms. This is where most of the value sits and where accuracy has to be taken seriously. Comprehension: answer a question about the content, such as whether this contract permits assignment. Genuinely useful for review work, and the least suited to running without a human, because the cost of a confident wrong reading is a legal one. Scope the project to one of the three. Teams that attempt all three at once end up with a system nobody trusts for any of them. Market research follows a similar pattern: [AI-Powered Market Research for Startups](/blog/ai-powered-market-research-for-startups).
Section 2
Accuracy is per field, not per document
Vendors quote document-level accuracy, which is close to meaningless. What matters is how often each field is right, because your business consequences differ wildly by field. A wrong supplier name is an annoyance. A wrong total is a payment error. A wrong date on a renewal clause is a contract you failed to exit. So measure per field, and set a different threshold for each one according to what a mistake costs. Some fields can go straight through. Some need a human every time regardless of confidence. Publishing one accuracy number for the whole document hides exactly the failures you care about most, and it is how systems pass evaluation and then cause an incident.
Section 3
Confidence thresholds and the exception queue
Good document automation is not a system that is always right. It is a system that knows when it is unsure and asks. Set a confidence threshold per field. Above it, the value flows through automatically. Below it, the document lands in a queue with the uncertain fields highlighted, so a human confirms in seconds rather than re-reading the whole thing. Then watch queue size as your primary operational metric. A queue that grows faster than volume means the thresholds are wrong or the document mix has changed. A queue that is empty means the thresholds are too loose and errors are flowing through unexamined, which is the more expensive of the two failures. Related patterns are in [Streamlining Customer Service with AI-Powered Chatbots](/blog/streamlining-customer-service-with-ai-powered-chatbots).
Section 4
Build the test set from real documents
The single highest-return hour in this kind of project is spent assembling a set of fifty real documents from your own history, deliberately including the ugly ones. The faxed copy. The one where the supplier changed template. The multi-page attachment where page three is upside down. Label the correct answers by hand once. That set becomes your evaluation for every candidate tool, every configuration change, and every model update the vendor pushes without telling you. Without it you are judging on impressions, and you will not notice the day accuracy drops, because the outputs will look exactly as confident as they did the week before.
Section 5
Retention, personal data, and what you keep
Trustworthiness, design, evaluation and use are NIST's four concerns, and document processing is where the data ones get sharpest, because documents carry the most sensitive material a small company holds. Identity documents, bank details, health information, salary data and signed contracts routinely pass through these pipelines. Three questions need written answers before you go live: where is the document processed, how long is it retained by the vendor, and is it used to improve anyone's model. Then minimise. If the automation only needs an invoice total, it does not need to store the whole document indefinitely. Set a retention period, restrict who can pull the originals, and log every access. Analysis presentation is covered in [Using AI and Data Analytics to Enhance Storytelling](/blog/using-ai-and-data-analytics-to-enhance-storytelling).
Section 6
Straight-through rate and cost per document
Four numbers make the case or kill it. Straight-through rate: the share of documents processed with no human touch. This is the headline, and it should rise over months as you tune thresholds. Field-level accuracy on your test set, tracked over time so a silent regression is visible. Handling time on the exceptions, which tells you whether the queue is a quick confirmation or a second full read. Cost per document, all in, including the review minutes. Compare that last figure against what manual handling costs today. If it is close, the case for automation is quality and speed rather than money, and you should say so rather than dressing it up as savings.