Best Way to Measure AI Medical Coding Accuracy
Most published accuracy numbers are unfalsifiable. Here is the metric set that is not.
The direct answer: no single number measures coding accuracy. You need at least four — top-k recall, micro F1, unsupported recommendation rate, and billing-readiness agreement — and you need to know what corpus they were computed on.
A vendor quoting one number without a corpus description has told you nothing. This post explains each metric, what it hides, and how to compare two vendors who report different ones.
Micro F1
The harmonic mean of precision and recall, computed over all code predictions pooled together rather than averaged per document.
Micro weighting means high-volume codes dominate the score. A system can post strong micro F1 while performing poorly on the rare, high-value codes — which are frequently the ones that matter for reimbursement.
Top-k recall
The share of correct codes that appear anywhere in the system's top k candidates. Top-5 recall is the standard reporting point for human-in-the-loop workflows.
This is the metric that predicts whether a coder can work from the candidate list instead of searching. It says nothing about precision.
Unsupported recommendation rate
The share of proposed codes for which no supporting text exists in the source documentation.
The compliance-critical metric, and the one most rarely published. A system can be accurate on average and still occasionally invent a code, which is a categorically different risk.
Billing-readiness agreement
How often the system's judgment that a claim is ready to submit agrees with expert human review.
This is the closest proxy for operational value, because it measures the decision the workflow actually makes rather than the codes in isolation.
Our published validation figures
- 96.3% — Top-5 recall
- 0.874 — Micro F1, full validation corpus
- 0.901 — Micro F1, validation subset
- 0.0% — Unsupported recommendations
- 92.6% — Billing-readiness agreement
- 78.9% — Pre-submission risk detection
- 5.8 min — Average encounter review time
- $0.18 — Average processing cost per encounter
Why we report two F1 numbers
0.874 is the full corpus. 0.901 is the validation subset. Reporting only the higher figure would be the conventional move and it would be misleading.
The gap between them is informative: it tells you how much of the performance depends on the characteristics of a cleaner subset. Any vendor whose two numbers are identical either has an unusually homogeneous corpus or is reporting one number twice.
How the same system looks under different metrics
Illustrative: one system, four metrics, scaled to a common axis. The story changes depending on which you read.
A vendor choosing which number to lead with can present the same system as 96% accurate or 79% accurate, both truthfully.
Four ways accuracy numbers get gamed
- Corpus selection — Evaluating on clean, single-diagnosis outpatient encounters and reporting the result as general accuracy.
- Metric selection — Leading with top-10 recall, which is nearly always high, and omitting precision entirely.
- Answer key laundering — Grading against codes that were themselves cleaned up after denial rework, which inflates agreement.
- Per-document macro averaging — Averaging per document rather than pooling, which lets simple documents carry the score.
How to run your own evaluation
- Sample consecutively, not selectively — Take every encounter in a window. Curated samples measure curation.
- Freeze the answer key before you look at output — Use the codes as originally billed, with reworks tracked separately.
- Compute all four metrics — Recall, precision or F1, unsupported rate, and readiness agreement. Any one alone is misleading.
- Break out by service line — Blended numbers hide the specialties where the system fails.
- Measure review time on the same sample — Accuracy that costs more minutes is not an improvement.
Frequently asked questions
What accuracy should we require before deploying?
There is no universal threshold, because the workflow determines the requirement. With mandatory human review, high top-5 recall and a zero unsupported rate matter far more than raw F1.
Is 0.874 micro F1 good?
It is strong for multi-label ICD-10 across a heterogeneous corpus and would be unremarkable for a narrow single-specialty task. Context is the whole answer.
Should we trust vendor-published numbers at all?
Trust the methodology, not the number. If the corpus, the answer key construction, and the metric definitions are all disclosed, the number is probably honest. If any of the three is missing, treat it as marketing.
How often should accuracy be re-measured?
Quarterly at minimum, and after any code set update. Coding guidelines change annually and performance drifts with them.