| Algorithm | What it solves | How |
|---|---|---|
| Conformal-style risk calibration | "What threshold gives me bounded false authorization?" | Tune/certify/test split, quantile of nonconformity scores |
| Sheepish transform | "Overconfident errors are more dangerous than cautious abstentions" | Asymmetric penalty on confidence that's too high |
| Per-field risk budgets | "A wrong signer name is worse than wrong metadata" | Signer: 1%, amount: 2%, date: 3%, default: 10% |
| Online calibration | "Human decisions should improve future thresholds" | MarginOnlineCalibrator updates from feedback labels |
| Dual-call verification | "Does a second extraction agree?" | Hunter (field-guided) vs Mapper (document-guided) |
Each time a human resolves an exception, ProofDesk captures the confidence level, the accept/reject decision, and the field involved. This becomes calibration data. The online calibrator updates. Future decisions improve.
Human review falls over time while measured false-authorization risk stays bounded.
Threshold tuning validated on held-out splits across heterogeneous classification datasets. Not a document-specific benchmark — these are the datasets used to develop the confidence algorithms.
| Dataset | Records | Type |
|---|---|---|
| ColdHearted Fraud | 19,872 | Transaction records |
| ContractNER | 3,241 | Contract text |
| FATURA | 1,400 | Invoices |
| CUAD | 509 | Contract clauses |
| InvoiceBenchmark | 200 | Invoices |
At 1% false-sign rate: 59.8% auto-sign coverage — 2.7x over baseline logistic regression.
Decision distribution: auto-sign / defer / block
Confidence score distribution across extractions
Convergence: review rate falls, error stays bounded
Cost analysis: time saved vs manual review cost
SignatureGate: Risk-Budgeted Document Signing
Full paper with abstract, methods, results, limitations. Covers conformal risk control, sheepish metric, per-field budgets, and convergence analysis.
From Confidence to Authority
ArXiv-style preprint on risk-calibrated document automation with source-grounded evidence.
Canonical Thesis
The foundational argument: extraction accuracy is not execution authority. Calibrated confidence as the bridge.
Confidence Metrics Research
Survey of non-hallucinated confidence approaches: UCCI, conformal prediction, ConfidenceBench, FaR.
Feedback-Driven Calibration
How human decisions become calibration data. The convergence loop thesis.
Technical Depth Appendix
Full technical documentation: algorithms, benchmarks, experiments, and methodology.