A benchmark score is the easiest thing to get right about a model and the least informative thing about whether you should trust it. Before we sign off on any model touching biological or clinical data, we ask five questions that a holdout accuracy number can’t answer.
1. Does the training data reflect who or what you’ll actually see in production?
A model trained on one population, one instrument, or one lab’s protocol can look excellent on paper and fail quietly the moment it meets data that looks even slightly different. Ask what the training distribution actually was, not just how big it was.
2. What does the model do when it’s uncertain?
Every model is wrong sometimes. The question is whether it knows it. A model that returns a confident-looking answer regardless of how far outside its training distribution the input falls is more dangerous than one that simply says “I don’t know.”
3. Can a domain expert explain a specific prediction, not just the model’s overall behavior?
Global feature importance is not the same as a case-level rationale. If a biologist or clinician can’t look at one specific prediction and understand roughly why the model made it, that’s a gap worth closing before deployment.
4. How will you know when it starts to drift?
Biological and clinical data shift over time — new assay versions, new patient populations, new lab protocols. A model validated once at launch and never rechecked will eventually be running on data it was never actually tested against.
5. What’s the cost of a false negative, specifically?
Not all errors are equal, and aggregate accuracy hides that. A model that’s 95% accurate but concentrates its errors in the cases that matter most is a worse choice than one that’s 90% accurate with errors spread evenly and none of them catastrophic.
The model that mattered wasn’t the most accurate one on paper — it was the one our biologists actually trusted enough to act on.
None of these questions are hard to ask. They’re just easy to skip when a benchmark number is sitting right there, looking like an answer.