DevNetwork API+Cloud+AI Hackathon 2026

Extraction accuracy is not execution authority

Two documents can be individually read correctly and still describe a transaction that should never be approved. ProofDesk catches the gap between "we extracted it correctly" and "we should act on it."

Live Demo Read the Research GitHub
The Problem
AI can read documents. That doesn't mean it should act on them.
Current document AI optimizes for extraction accuracy. But in regulated workflows, the hard problem isn't "can the model read this?" — it's "is the evidence sufficient to authorize what happens next?"
Extraction Accuracy
95%+
Fields extracted correctly by modern DWS systems
Action Safety
Unknown
Whether the evidence justifies an irreversible commitment
🔍

Individually correct

Each document is extracted with high confidence. No extraction error.

⚠

Jointly unsafe

Together the facts describe a transaction that violates policy. Insurance expires before the service period ends.

🚫

Agent proceeds anyway

A naive system sees high confidence and auto-approves. The human never reviews the contradiction.

The Solution
ProofDesk: Evidence-Gated Document Automation
Nutrient DWS extracts grounded evidence. ProofDesk determines when that evidence is sufficient to act — and when it must defer to a human.
1
Extract
Nutrient DWS pulls value, confidence, page, bbox for every field
2
Verify
Cross-document checks catch contradictions
3
Gate
6 conditions enforced server-side. Agent cannot negotiate.
4
Human
Reviews exact evidence, not the entire bundle
5
Execute
Hash-chained audit trail, Merkle proofs
6
Learn
Human decisions improve future calibration
15
Extracted Fields
6
Authority Checks
1
Contradiction Found
0
Unsafe Actions
The Demo
A real contradiction, caught live
Click "Run Live" and watch Nutrient DWS extract grounded evidence from four procurement documents. The pipeline finds a contradiction that extraction alone would miss.
The Research
We studied when automation should stop
Behind the authority gate is a research program in calibrated confidence. Not ad-hoc thresholds — published algorithms applied to document automation.
AlgorithmWhat it solvesHow
Conformal-style risk calibration"What threshold gives me bounded false authorization?"Tune/certify/test split, quantile of nonconformity scores
Sheepish transform"Overconfident errors are more dangerous than cautious abstentions"Asymmetric penalty on confidence that's too high
Per-field risk budgets"A wrong signer name is worse than wrong metadata"Signer: 1%, amount: 2%, date: 3%, default: 10%
Online calibration"Human decisions should improve future thresholds"MarginOnlineCalibrator updates from feedback labels
Dual-call verification"Does a second extraction agree?"Hunter (field-guided) vs Mapper (document-guided)
The Convergence Loop

Each time a human resolves an exception, ProofDesk captures the confidence level, the accept/reject decision, and the field involved. This becomes calibration data. The online calibrator updates. Future decisions improve.

Human review falls over time while measured false-authorization risk stays bounded.

Calibration Research

Threshold tuning validated on held-out splits across heterogeneous classification datasets. Not a document-specific benchmark — these are the datasets used to develop the confidence algorithms.

DatasetRecordsType
ColdHearted Fraud19,872Transaction records
ContractNER3,241Contract text
FATURA1,400Invoices
CUAD509Contract clauses
InvoiceBenchmark200Invoices

At 1% false-sign rate: 59.8% auto-sign coverage — 2.7x over baseline logistic regression.

Visualizations
Decision Distribution

Decision distribution: auto-sign / defer / block

Confidence Distribution

Confidence score distribution across extractions

Convergence Projection

Convergence: review rate falls, error stays bounded

Cost Savings

Cost analysis: time saved vs manual review cost

Publications & Technical Reports

SignatureGate: Risk-Budgeted Document Signing

Full paper with abstract, methods, results, limitations. Covers conformal risk control, sheepish metric, per-field budgets, and convergence analysis.

From Confidence to Authority

ArXiv-style preprint on risk-calibrated document automation with source-grounded evidence.

Canonical Thesis

The foundational argument: extraction accuracy is not execution authority. Calibrated confidence as the bridge.

Confidence Metrics Research

Survey of non-hallucinated confidence approaches: UCCI, conformal prediction, ConfidenceBench, FaR.

Feedback-Driven Calibration

How human decisions become calibration data. The convergence loop thesis.

Technical Depth Appendix

Full technical documentation: algorithms, benchmarks, experiments, and methodology.
The Moat
What's hard to replicate

Nutrient source grounding

Every fact carries value, confidence, page provenance, and bounding box. Not just text extraction — evidence with location.

Cross-document verification

Deterministic checks catch contradictions that per-document extraction misses. The conflict is between documents, not within them.

Calibrated authority gate

6 conditions enforced server-side. Thresholds calibrated from research, not hand-tuned. The agent cannot negotiate.

Human-feedback convergence

Human decisions become calibration data. The system learns where this organization can safely automate. Competitors can copy the UI; they can't copy three years of calibrated authority decisions.

Tamper-evident execution

Hash-chained audit trail with Merkle proofs. Every decision replayable. Content-addressed artifacts. Change one byte and the hash changes.

12 document types

Procurement, invoice, contract, insurance, trade, KYC, mortgage, medical — each with specific field schemas, risk budgets, and verification assertions.

Results
What we measured
115
Tests Passing
13
PDF Fixtures Tested
12
Document Types
6
Authority Conditions
15
Extracted Fields
6
Cross-Doc Checks
31
Day Gap Detected
0
Unsafe Actions
Try It
Live interactive demo
Click "Run Live" above and watch the pipeline execute against the real Nutrient DWS API. Every step is visible.

Nutrient DWS: LIVE · 115 tests · Hash-chained audit

"Nutrient DWS performs the core document extraction and source grounding that turns uploaded PDFs into confidence-aware evidence. ProofDesk uses that evidence to determine whether an automated action may proceed or must defer to a human."