Artificial intelligence is transforming healthcare, but is it ready to replace human touch? As technology advances, a new study investigates the effectiveness of AI, specifically ChatGPT, in answering patient questions compared to experienced surgeons, revealing surprising insights about communication in medicine.
Bots like ChatGPT are designed to understand and generate human language. ChatGPT can interpret context, tone, and cultural nuances, allowing it to communicate more effectively. One popular use of Large Language Models (LLMs) like ChatGPT is that they provide quick answers to questions about various topics, including healthcare. Researchers in Germany, focusing on pancreatic cancer surgery, put ChatGPT to the test to see how well it could answer patient questions compared to two experienced surgeons. They wanted to assess not just how accurate its answers were, but also how clearly and empathetically it communicated.
To do this, the research team created a list of 10 common questions patients might ask. These questions were gathered with input from patients at Heidelberg University Hospital. Both ChatGPT and the surgeons responded to these questions separately. A panel made up of 24 patients and 25 surgeons then reviewed the answers using an online survey, focusing on how accurate, clear, and empathetic the responses were.
The findings showed some interesting results. Patients rated ChatGPT’s answers as equally accurate and clear as those from the surgeons. In terms of empathy, ChatGPT actually performed slightly better than one of the surgeons. However, patients preferred the first surgeon’s responses half of the time, while they chose ChatGPT’s answers about 30% of the time. Interestingly, the surgeons were more successful at identifying which responses were generated by AI, while patients found it harder to tell the difference.
These results suggest that ChatGPT could be a helpful tool for making complex medical information easier to understand for patients. Similar successes have been noted in studies about colon cancer and heart disease prevention. However, there are important limitations to be aware of. ChatGPT wasn’t specifically built for medical use, and it can make mistakes or produce confusing information—referred to as “hallucinations.” These errors often occur due to the way LLMs process information.
Unlike humans, AI doesn’t truly “understand” content but instead generates answers by analyzing words as fragments. Because of this, it can sometimes misinterpret data or struggle with visual information like graphs. This underlines the importance of carefully reviewing any medical information that comes from AI.
Currently, AI cannot replace the critical human interaction necessary in healthcare, especially when it comes to addressing the emotional aspects of illness. It is also crucial to clearly label when information is generated by AI to avoid misunderstandings for patients. Legal responsibility for mistakes made by AI is still a significant concern that needs to be tackled before these technologies can be widely adopted in clinical settings, similar to ongoing discussions about accountability in self-driving cars.
