AI in healthcare: Safety, trust and validation are critical

Artificial intelligence is increasingly transforming healthcare, with current use cases largely centred on improving clinical workflows such as automated scribing, documentation, and administrative support. It is also being used to help clinicians manage the growing volume of medical knowledge by rapidly synthesising research, medical literature, and patient data into usable insights.

Dr Stephen Barnett, co-founder of MedLuma, highlights that while these tools are becoming more common, their integration into clinical environments must be handled with caution.

A key concern is reliability, particularly the issue of AI “hallucinations”, where systems generate confident but incorrect medical information. There are also findings suggesting that some AI models may struggle to accurately identify emergency situations, raising important questions about patient safety and trust.

According to Dr Barnett, these risks underline the need for rigorous validation before such systems can be relied upon in real-world clinical settings.

To address these challenges, Dr Barnett advocates for the development of “medical grade” AI that is specifically engineered and tested for healthcare use, rather than adapted from general purpose models.

He emphasises that human oversight remains essential, ensuring outputs are traceable to credible sources and can be verified by clinicians. Alongside stronger safety standards, he also calls for improved education so healthcare professionals can better assess and deploy AI tools responsibly.