AI in Healthcare: Experts Urge Human Oversight to Prevent Deskilling and Ensure Patient Safety
London,UK – As artificial intelligence rapidly integrates into healthcare,concerns are mounting over it’s potential impact on both patient trust and clinician skill. Experts are emphasizing the critical need for “human-in-the-loop” systems – maintaining human oversight – to ensure responsible and effective AI implementation, particularly in high-risk areas like clinical decision support.
Growing public anxieties focus on the use of large language models (LLMs) as unlicensed therapy chatbots3, with reports surfacing of AI-mediated delusions, suicidal ideation, and even instances of bromide poisoning.This raises questions about building patient trust, especially when individuals may turn to chatbots instead of their doctors for medical advice2.
Recent research highlights the potential for AI to inadvertently decrease human expertise. A study found that endoscopists who utilized AI assistance for polyp detection experienced a decline in their detection rates after just three months without the tool4, suggesting continuous AI exposure could lead to physician deskilling.
Though, experts believe these risks can be mitigated. Drawing parallels to aviation, where pilots regularly retrain on manual controls to avoid over-reliance on autopilot, they suggest periodic retraining for clinicians could counter algorithmic dependence.
The key, researchers emphasize, lies in understanding how AI is integrated into clinical workflows. Rigorous trials and real-world testing are needed to identify potential pitfalls and develop a clear framework of accountability for AI-related errors.
Ultimately, the future of medical AI hinges on a relational approach, treating clinicians as active partners rather than passive users, and focusing on systems that leverage the strengths of both humans and machines to overcome their individual biases and limitations.
2 Nature, https://www.nature.com/articles/s41591-025-04033-7#ref-CR2
3 STAT, http://bit.ly/4305RLU (3 September 2025)
4 Budzyński, K. et al. Lancet Gastroenterol. hepatol.10, 896-903 (2025).
