While AI models keep improving, the expectations for those guiding their development and deployment are only getting higher.
By Brian Bocchino, Partner, Ventures Practice – Health and Life Sciences
3 takeaways
- Predictive AI struggles with clinical nuance. Even small changes in patient data caused models to miss critical health risks, revealing a gap between statistical performance and clinical reliability.
- AI leaders must bridge data science and care delivery. Effective leadership today means understanding both how models work and how they’re used at the point of care.
- Hiring should reflect AI’s clinical impact. Organizations benefit from executives who can align AI strategy with patient safety, evolving regulations, and frontline workflows.
A study published in Nature’s Communications Medicine found that many AI models designed to predict in-hospital mortality are missing the mark. Specifically, models trained on historical patient data failed to catch about 66% of deterioration events—the kind that could lead to death if unnoticed.
That stat is hard to ignore. And it raises important questions about the leadership behind these tools. Because while the models keep improving, the expectations for those guiding their development and deployment are only getting higher.
AI models aren’t keeping up with clinical complexity
In the study, researchers ran a series of tests on popular machine learning models that predict patient deterioration. They made small adjustments to patient metrics—things that can and do happen in real-world clinical settings—and asked the models to respond.
Most didn’t.
On average, the models only picked up on about a third of the new deterioration cases. Why? In many cases, they were trained too narrowly. Overfitted to data that doesn’t reflect the messy, nuanced, fast-changing conditions of hospital environments.
It’s a reminder of something many clinicians and data scientists already know: Just because a model performs well in testing doesn’t mean it’s ready for real-world care.
Bias, equity, and the missing context
Zooming out, the issue gets even more layered. Studies have shown that AI tools often underperform for patients from underrepresented populations—those in rural areas, lower-income communities, and racial minorities. Many of these models don’t take social determinants of health (SDoH) into account. When that happens, structural inequities can get baked into the algorithm.
That’s why leaders can’t afford to focus on accuracy alone. Equity and representation matter, too. Not just as ethical priorities—but as indicators of how well a tool will generalize across the full population it’s meant to serve.
What this means for AI leadership in healthcare
For executives leading AI initiatives—whether in hospitals, digital health, or life sciences—this kind of gap between technical output and clinical relevance should be a signal. It might be time to re-evaluate how AI leadership is structured and supported.
Today’s AI leaders are doing more than building models. They’re shaping decisions that affect patient safety, clinical trust, regulatory risk, and long-term outcomes.
The strongest leaders tend to:
- Move comfortably between data science and clinical priorities
- Spot issues like model bias and explainability challenges early
- Build cross-functional teams that include clinicians, engineers, compliance experts, and operators
- Design strategies that work in real-world care environments, not just development labs
These aren’t just technical skills—they’re leadership skills rooted in context, judgment, and collaboration.
Hiring for the AI moment we’re in
If you’re responsible for hiring AI executives, this is an opportunity to think bigger than resumes and titles. The healthcare AI landscape is moving fast—and the leaders who thrive in it usually bring a mix of traits:
- Interdisciplinary fluency
- They speak the language of engineering, clinical care, product, and policy—and know how to get those teams aligned.
- They speak the language of engineering, clinical care, product, and policy—and know how to get those teams aligned.
- Contextual thinking
- They don’t just look at performance metrics. They ask: How will this actually be used? What happens if it fails?
- They don’t just look at performance metrics. They ask: How will this actually be used? What happens if it fails?
- Governance awareness
- They understand the regulatory landscape—FDA guidance, CMS expectations, the coming wave of AI regulations—and can build structures that keep innovation compliant and responsible.
- They understand the regulatory landscape—FDA guidance, CMS expectations, the coming wave of AI regulations—and can build structures that keep innovation compliant and responsible.
- Collaborative leadership
- They build partnerships across departments, break down silos, and understand that no one team can own AI in isolation.
These kinds of candidates are out there—but they’re in high demand. Getting clear on what matters most for your organization can help you find the right fit, faster.
Regulatory and trust considerations are rising
AI in healthcare isn’t just a technical challenge anymore—it’s a regulatory one, too. With new frameworks emerging from the FDA, CMS, the EU AI Act, and others, health systems are under growing pressure to show that their tools are transparent, explainable, and fair.
In a recent conversation with healthcare executives Brian and Ben, a few big themes came up. One was the need for compliance platforms designed specifically for AI in clinical settings. Another was the importance of creating repeatable, responsible release practices—especially for tools that could impact patient care. And perhaps most importantly, they called out the complexity of interdependence. Building responsible AI takes tight coordination across product, clinical, regulatory, and leadership teams.
That means leaders should be thinking about:
- How model validation is tracked and shared
- Who’s involved in AI governance—and how it’s structured
- How they’re building clinician trust through transparency and usability
These aren’t small tasks. But they’re essential if AI is going to meaningfully—and safely—support clinical care.
What to ask next
If you’re leading, hiring, or advising around AI in healthcare or life sciences, here are a few questions worth reflecting on:
- Do we have the right leadership in place to oversee AI tools that directly impact care and outcomes?
- Are our hiring processes designed to find people who understand both the technology and the clinical reality?
- How are we managing risk, oversight, and alignment across teams as AI becomes more embedded in care delivery?
These aren’t just tech decisions—they’re leadership decisions. And getting them right will shape not just the future of your organization, but the safety and quality of care for the people you serve.
Brian Bocchino is Partner, Ventures Practice – Health and Life Sciences at Riviera Partners. Connect on LinkedIn.
Curious how clinical buyers are actually thinking about AI integration and compliance? Don’t miss our follow-up: How leadership will determine whether AI in healthcare survives the compliance and integration gauntlet.
Need help finding the right healthcare technology leaders?
At Riviera Partners, we work with healthcare and life sciences organizations to place executives who bring deep technical expertise and a clear understanding of clinical, regulatory, and operational complexity. Let’s talk.
