The model wasn’t wrong. That was the frustrating part. A diagnostics platform we later started working with had a triage model with solid accuracy numbers on paper. It had also just failed a compliance audit, and the finding wasn’t about accuracy at all — it was that nobody, including the team that built it, could explain why a given case got flagged.
Why “it works” isn’t a compliance answer
In a research context, a model that performs well on a holdout set is often good enough. In a clinical context, performance is necessary but not sufficient. An auditor — and more importantly, a clinician relying on the tool — needs to know why a specific case was flagged, not just that the model is right most of the time in aggregate.
This is where a lot of otherwise-solid models fail. They’re built and validated as black boxes, with explainability treated as a post-hoc reporting problem instead of a design constraint. By the time an audit asks for a case-level rationale, retrofitting one onto an existing model is often harder than building a new one with explainability in mind from the start.
What an explainability report actually needs
- A clinician-readable rationale for each individual prediction, not just a global feature-importance chart.
- Confidence calibration that a non-technical reviewer can interpret without a statistics background.
- A documented validation process an auditor can independently verify, not just a summary metric.
- A clear boundary of what the model is, and isn’t, confident enough to weigh in on.
They understood that a false negative isn’t a metrics problem for us — it’s a patient outcome.
Building explainability in, not bolting it on
We rebuilt the model with explainability reporting as a first-class output, not an afterthought: every flagged case now ships with a clinician-readable rationale alongside the prediction. In parallel, we ran a full security architecture review and remediated every finding from the original audit. The platform passed its follow-up review with zero outstanding findings.
The lesson generalizes past this one engagement: if a model’s output will inform a decision with real consequences, explainability isn’t a reporting exercise you do at the end. It’s an architecture decision you make at the beginning.