Healthcare AI Has a Reliability Problem Nobody Talks About

Accuracy tells us whether a model can predict. Reliability tells us whether we should trust the prediction in the first place. Most conversations about healthcare AI eventually end up in the same place: accuracy. How accurate is the model? What was the AUC? Did it beat the baseline? Those questions matter. But after spending time evaluating healthcare AI systems, I've become increasingly convinced they're not the most important ones. What happens when the model isn't sure? Not in a research paper. Not in a carefully curated validation set. In the real world. Because real healthcare environments are messy. Patients don't arrive as clean rows in a dataset. Sensors fail. Measurements fluctuate. Clinical notes are incomplete. Two hospitals can record the same information in completely different ways. Yet many AI systems still behave as if every prediction deserves the same level of confidence. That feels like a problem. A model can achieve impressive performance metrics and still behave unpredictably when exposed to the kinds of variation that exist in everyday clinical settings. The issue isn't always that the prediction becomes wrong. Sometimes the prediction stays the same while the reasoning behind it changes completely. And when clinicians are expected to trust those recommendations, that distinction matters. More than most benchmark scores can tell us. I think healthcare AI has a reliability problem. The industry talks constantly about accuracy, explainability, and model performance. But we spend surprisingly little time asking whether an AI system behaves consistently when uncertainty enters the picture. That gap becomes much harder to ignore once you start thinking about deployment rather than experimentation. The Problem With Confident AI One of the most interesting things about modern AI systems is how confident they look. The user interface is usually polished. The prediction arrives instantly. A probability score appears with two decimal places. Everything about the experience suggests certainty. In healthcare, that confidence can be reassuring. It can also be misleading. While working on healthcare AI systems, I found myself paying less attention to headline accuracy metrics and more attention to what happened when small changes were introduced into the data. I've noticed that many discussions around clinical AI assume that confidence and reliability are closely related. In practice, they are not always the same thing. A model can be highly confident and still be operating in a region where its behaviour is surprisingly fragile. Think about how clinicians work in real environments. A doctor rarely makes decisions based on a single number. They consider the quality of the information available. They question unusual results. They ask whether a measurement might be affected by timing, equipment, patient movement, or missing context. Uncertainty is not treated as a failure of the process. It is part of the process. AI systems, however, often communicate in a very different way. They typically produce an answer without clearly communicating how sensitive that answer might be to small changes in the underlying data. That becomes particularly important in healthcare, where tiny variations occur constantly. A slight difference in oxygen saturation. A small delay between observations. A variation in how information is recorded. A laboratory value measured at a different point in time. None of these factors would surprise a clinician. But they can sometimes expose weaknesses in how AI systems behave. The result is a gap between how models are evaluated and how they are actually used. In research papers, we often celebrate average performance across thousands of cases. In deployment, clinicians experience one case at a time. And from their perspective, consistency matters just as much as accuracy. A model that is right most of the time but behaves unpredictably in edge cases can quickly lose trust. Once that trust disappears, adoption becomes much harder. Which is why I think reliability deserves far more attention than it currently receives. Before we ask whether a model is accurate, explainable, or state-of-the-art, we should also ask a simpler question: If the input changes slightly, does the model still behave sensibly? The Real World Is Not a Benchmark One of the easiest ways to spot a fragile AI system is to ask a simple question: What happens if the input changes slightly? Not dramatically. Not enough to change the clinical picture. Just enough to reflect the kind of variation that happens every day in healthcare settings. Imagine two patient records arriving in an intensive care unit a few minutes apart. They are almost identical. The patient is the same. The clinical situation is the same. The treatment plan has not changed. The only differences are minor fluctuations in a few measurements — perhaps a small change in oxygen saturation, a slightly different heart rate reading, or a laboratory value captured at a different point in time. Nothing unusual. The sort of variation clinicians encounter is constant. Now imagine the same AI model evaluates those records. The first patient is classified as: Low risk The second is classified as: High risk Most clinicians would immediately want to know why. But let's take the scenario one step further. Suppose the explanation changes as well. For the first prediction, the model identifies respiratory rate as the most influential factor. For the second prediction, it suddenly shifts its attention to age and blood glucose. At this point, the issue is no longer just accuracy. The issue is consistency. The model may still perform extremely well across a large validation dataset. Its average AUC may still look impressive. It may still outperform competing approaches on benchmark tasks. But from the perspective of the clinician sitting in front of the screen, something important has happened. The system has revealed that it may be operating near an unstable decision boundary. That does not necessarily mean the model is wrong. It does mean that confidence should be treated more carefully. And this is where many conversations about healthcare AI start to feel incomplete. We spend a lot of time discussing whether models are accurate. We spend a lot of time discussing whether models are explainable. We spend far less time discussing whether those predictions and explanations remain stable when uncertainty enters the picture. Yet uncertainty is not an edge case in healthcare. It is the operating environment. Every measurement contains some degree of variability. Every clinical workflow introduces noise. Every healthcare organisation has differences in data quality, coding practices, equipment, and patient populations. The question is not whether uncertainty exists. The question is whether our AI systems are designed to recognise it. Because if they are not, we risk creating systems that appear trustworthy during development but become difficult to trust when deployed in the real world. From Prediction Systems to Decision Systems The more I think about this problem, the more I believe that many healthcare AI systems are being designed around the wrong objective. Most models are built to answer a relatively straightforward question: What is the prediction? Will the patient deteriorate? Is this scan likely to contain disease? What is the probability of readmission? Those questions matter. But in clinical environments, they are rarely the only questions that matter. A clinician doesn't just want to know what the model predicts. They also want to know whether that prediction is reliable enough to influence a decision. Those are not the same thing. And that distinction becomes increasingly important as AI moves from research environments into everyday clinical workflows. Perhaps the goal should not simply be: Generate a prediction. Perhaps the goal should be: Determine whether the prediction is reliable enough to act upon. That may sound like a subtle shift, but it fundamentally changes how we think about AI systems. Instead of treating AI as a prediction engine, we begin treating it as part of a broader decision-support process. The model no longer becomes the final voice in the conversation. Instead, it becomes one participant in a workflow that includes clinicians, uncertainty, context, and human judgement. Viewed through that lens, a healthcare AI system should not only answer: What do I predict? It should also answer: How confident am I in this prediction? How sensitive is this prediction to small changes in the data? How consistent is the reasoning behind this recommendation? Should this case be reviewed by a human before anyone acts on it? Interestingly, clinicians already think this way. Experienced healthcare professionals constantly assess confidence in the information available to them. They question unusual findings. They seek second opinions. They repeat tests. They escalate uncertain cases. In many ways, uncertainty management is already built into clinical practice. Yet most AI systems still behave as though uncertainty is something to be hidden rather than communicated. That feels backwards. The more healthcare AI becomes embedded in real-world decision-making, the more important it becomes for systems to recognise when confidence should be reduced rather than projected. Because sometimes the most useful output an AI system can produce is not: "Here is the answer." Sometimes it is: "This case deserves a closer look." A Reliability Check Before the Recommendation So what would a reliability-aware healthcare AI system actually look like? I don't think the answer is to make models less powerful. Nor do I think the answer is to replace sophisticated machine learning systems with simpler models purely for the sake of transparency. Modern AI models are capable of delivering enormous value in healthcare. The challenge is not their predictive capability. The challenge is knowing when that capability should be trusted. One approach I've been exploring is based on a simple idea: Before presenting a recommendation to a clinician, the system should first evaluate whether the recommendation appears sufficiently reliable. In other words, a prediction should pass a reliability check before it reaches the decision-maker. That reliability check can be broken into two practical questions. 1. Does the Prediction Remain Stable? The first question focuses on the prediction itself. If small, clinically plausible changes are introduced into the input data, does the model continue to produce a similar outcome? For example: What happens if a heart rate changes slightly? What happens if an oxygen saturation measurement varies by a small amount? What happens if a laboratory result is recorded at a slightly different point in time? These are not unrealistic scenarios. They happen every day. A reliable system should not dramatically change its behaviour because of minor variations that do not meaningfully alter the underlying clinical picture. If the prediction remains consistent, confidence in the recommendation increases. If the prediction changes significantly, that may indicate the model is operating in a region of uncertainty. And uncertainty should be communicated rather than hidden. 2. Does the Reasoning Remain Stable? The second question is equally important. Even if the prediction remains unchanged, does the explanation remain consistent? This is where many explainability discussions become interesting. A model might continue predicting high risk while constantly changing the factors it considers most important. One moment it highlights respiratory indicators. The next it shifts attention to glucose measurements. Then it focuses on age. If that reasoning changes dramatically between nearly identical inputs, clinicians have good reason to question how much confidence they should place in the explanation. The explanation may still be plausible. But plausibility is not the same thing as reliability. And if explanations are being used to support clinical decisions, reliability matters. A lot. Taken together, these two questions create a much richer picture of model behaviour. Instead of simply asking: "What did the model predict?" We begin asking: "How stable was the prediction?" and "How stable was the reasoning behind it?" That shift may sound small. But I suspect it represents one of the missing pieces in the conversation around trustworthy healthcare AI. Because clinicians do not just need predictions. They need reasons to trust those predictions. Accept, Caution, or Defer One of the things that bothers me about many AI systems is that they are designed as though every prediction were equally important. The model produces an answer. The answer is displayed. The workflow moves on. But real-world decision-making rarely works like that. In healthcare, not all information is treated with the same level of confidence. Clinicians naturally distinguish between: information they trust, information they want to verify, and information that requires further investigation. AI systems should be able to do something similar. If we combine prediction stability and explanation stability, a simple decision-support pattern begins to emerge. Not a prediction framework. A trust framework. Accept The prediction remains stable. The explanation remains stable. Small changes in the input do not meaningfully alter either the recommendation or the reasoning behind it. In this situation, the AI system has earned a higher degree of confidence. The recommendation can be presented as a useful piece of decision support while still leaving final judgment in the hands of the clinician. The system is not claiming certainty. It is demonstrating consistency. And consistency is often what trust is built on. Caution The prediction remains relatively stable. The explanation does not. The model continues producing a similar recommendation, but the factors driving that recommendation appear inconsistent. This creates a different type of risk. The model may still be correct. But if the reasoning shifts dramatically between nearly identical inputs, clinicians should probably treat the explanation more carefully. This is the sort of situation where an AI system could flag: "The prediction appears stable, but the explanatory factors show elevated variability." That signal alone may encourage a clinician to review supporting evidence more closely. Not because the model is wrong. Because the model is uncertain about why it is right. Defer This is the category that I think many AI systems are missing entirely. The prediction becomes unstable. The explanation becomes unstable. Or both. At this point, forcing confidence serves no one. The responsible action is not to produce a stronger recommendation. It is to recognise that the system may not be operating within a trustworthy decision region. Instead of pretending certainty, the AI should escalate. It should effectively say: "I don't have enough confidence in this case. Human review is recommended." That might sound like a limitation. I see it differently. In high-stakes environments, the ability to recognise uncertainty may be one of the most valuable capabilities an AI system can possess. We've spent years teaching machines how to answer questions. Perhaps we should spend more time teaching them when not to. This way of thinking eventually became the foundation for a reliability-oriented approach I've been exploring called Decision-Calibrated Explainable AI (DC-XAI). The name is less important than the principle behind it. Before an AI recommendation influences a clinical decision, the system should have some way of assessing whether its prediction and reasoning are stable enough to deserve trust. Not perfect trust. Not blind trust. Just enough trust to justify moving forward. Why Builders Should Care At first glance, this might sound like a healthcare-specific problem. I don't think it is. Healthcare simply exposes the issue more clearly because the consequences of getting it wrong are easier to see. The underlying challenge exists in almost every high-stakes AI system. Fraud detection systems decide whether transactions should be blocked. Cybersecurity models decide whether an event should be escalated. Autonomous systems decide whether an environment appears safe. Financial models influence lending decisions. Recruitment algorithms influence hiring recommendations. In each case, the model is not simply generating information. It is influencing a decision. And once AI systems start influencing decisions, reliability becomes a product requirement rather than a research metric. That distinction is important. A lot of AI development still revolves around model performance. Teams compete for higher accuracy, better precision, stronger recall, or improved benchmark rankings. Those metrics matter. But users rarely experience AI systems through benchmark metrics. They experience them through behavior. They notice when recommendations feel inconsistent. They notice when explanations seem contradictory. They notice when the system appears confident in situations where confidence is not justified. Trust is rarely lost because a user reads the model's validation report. Trust is lost because the system behaves unexpectedly at the moment it matters. That is why I increasingly think reliability should be treated as a first-class engineering concern. Not something evaluated after deployment. Not something discussed only in governance documents. Something was designed into the system from the beginning. One of the most overlooked organizational parts of AI development today is the user interface around uncertainty. Many systems communicate confidence. Far fewer communicate uncertainty. Those are not the same thing. A confidence score tells users how strongly a model believes something. An uncertainty signal tells users how much caution they should apply when interpreting that belief. The difference can be significant. Imagine two AI systems producing the same prediction. The first simply says: High Risk The second says: High Risk Recommendation Stability: High Explanation Stability: Low Human Review Recommended Which system would you trust more? For me, the answer is obvious. Not because the second system is necessarily more accurate. Because it is being more honest about its limitations. And honesty about limitations is often what separates trustworthy systems from merely impressive ones. As AI becomes embedded in increasingly important decisions, I suspect the organisations that earn the most trust will not be those with the highest benchmark scores. They will be the ones who design systems capable of communicating uncertainty clearly, escalating responsibly, and recognizing when human judgment should remain in control. The Future Is Not Just More Accurate Models If you spend enough time around AI, it's easy to believe that every important problem will eventually be solved by making models larger, faster, or more accurate. History gives us plenty of reasons to think that way. Over the past decade, we've seen remarkable improvements in prediction performance across medical imaging, clinical risk assessment, diagnostics, and countless other healthcare applications. Those advances matter. They will continue to matter. But I don't think the next major challenge in healthcare AI is simply about improving predictions. I think it's about improving judgment. Not human judgment. Machine judgment. More specifically, a model's ability to recognise the limits of its own knowledge. That may sound like an unusual goal. After all, we've spent years training AI systems to become more decisive, more confident, and more capable of producing answers. Yet in healthcare, some of the most important decisions happen when uncertainty is recognised rather than ignored. Experienced clinicians understand this instinctively. They know when a result looks unusual. They know when more evidence is needed. They know when a second opinion is appropriate. And perhaps most importantly, they know when confidence should be reduced rather than increased. The future of healthcare AI should reflect that same maturity. Not by replacing clinical expertise. Not by pretending uncertainty does not exist. But by becoming better at communicating uncertainty honestly and consistently. That means moving beyond questions like: How accurate is the model? and asking questions such as: How stable is this recommendation? How stable is the reasoning behind it? Should a human review this before action is taken? Those questions may not produce headline-grabbing benchmark scores. But they might produce something more valuable. Trust. Because in the end, healthcare is not a leaderboard problem. It is a human problem. Patients do not care whether a model achieved state-of-the-art performance on a benchmark dataset. Clinicians do not care whether a model won a competition three years ago. What matters is whether the system behaves responsibly when faced with the uncertainty that exists in every real healthcare environment. The most trustworthy healthcare AI systems may not be the ones that make the boldest predictions. They may be the ones who recognise when caution is warranted, communicate that clearly, and know when to hand the decision back to a human. We've spent years teaching machines how to answer questions. Perhaps we should spend more time teaching them when not to. Disclosure This article reflects my ongoing work and research interests in trustworthy AI, healthcare intelligence, explainable machine learning, and reliability-aware decision systems. The ideas discussed here are intended as a practical perspective on how AI systems can better communicate uncertainty in high-stakes environments. While some of these concepts inform my broader work on Decision-Calibrated Explainable AI (DC-XAI), the views expressed in this article are my own and are presented as part of an ongoing conversation about building safer and more trustworthy AI systems. Further Reading For readers interested in exploring these topics further: UK Medicines and Healthcare products Regulatory Agency (MHRA) — AI as a Medical Device (AIaMD) Programme. European Union AI Act — Framework for High-Risk AI Systems. Adejumobi AM, Adeniya FA. Decision-Calibrated Explainable AI for Reliability-Aware Clinical Predictions: A Stability-Based Framework . JMIR Preprints. 2026. DOI: 10.2196/preprints.100751. Research on SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-Agnostic Explanations), and uncertainty estimation techniques in machine learning. If there's one idea I'd like readers to take away from this article, it's this: Reliable AI is not simply AI that gets the answer right. Reliable AI is AI that understands when its answer deserves caution. In healthcare and increasingly in many other high-stakes domains, that distinction may matter more than we realize. \