Bad medicine? When asked about prescriptions, ChatGPT mostly got it wrong
The popular AI tool’s answers to most drug-related questions were incomplete or wrong in a new study, cautioning health care professionals about using ChatGPT as an “authoritative source for medication-related information.”
New research may dampen some of the enthusiasm for the role of artificial intelligence in guiding pharmaceutical care.
ChatGPT’s answers to nearly 75% of drug-related questions reviewed by pharmacists were incomplete or wrong in a study presented at the American Society of Health-System Pharmacists Midyear Clinical Meeting in early December. Some of the inaccurate responses could endanger patients.
“Health-care professionals and patients should be cautious about using ChatGPT as an authoritative source for medication-related information,” said Sara Grossman, PharmD, associate professor of pharmacy practice at Long Island University and a lead author of the study. “Anyone who uses ChatGPT for medication-related information should verify the information using trusted sources.”
Grossman and her team challenged the free version of ChatGPT by OpenAI with real questions over a 16-month period in 2022 and 2023. Pharmacists involved in the study first researched and answered 45 questions, and each answer was reviewed by a second investigator. These responses served as the standard against which the responses generated by ChatGPT were compared.
Only 10 of the 39 ChatGPT-provided responses were judged to be satisfactory according to the criteria established by investigators. For the other 29 questions, responses generated by ChatGPT did not directly address the question (11) or were inaccurate (10), and/or incomplete (12). For each question, researchers asked ChatGPT to provide references so the information provided could be verified. References were provided in just eight responses, and each included non-existent references.
In one case, researchers asked ChatGPT whether a drug interaction exists between the COVID-19 antiviral Paxlovid and the blood pressure-lowering medication verapamil, and ChatGPT indicated no interactions had been reported for this combination of drugs.
“In reality, these medications have the potential to interact with one another, and combined use may result in excessive lowering of blood pressure,” Grossman said. “Without knowledge of this interaction, a patient may suffer from an unwanted and preventable side effect.”
Related: Biden to HHS: Create an AI task force to keep health care ‘safe, secure and trustworthy’
OpenAI’s usage policies state that its technologies should not be used to “tell someone that they have or do not have a certain health condition or provide instructions on how to cure or treat a health condition. OpenAI’s models are not fine-tuned to provide medical information. You should never use our models to provide diagnostic or treatment services for serious medical conditions.”
Although AI shows great promise to transform many aspects of health care, researchers recommend proceeding with caution.
“AI-based tools have the potential to impact both clinical and operational aspects of care,” said Gina Luchen, PharmD, the society’s director of digital health and data. “Pharmacists should remain vigilant stewards of patient safety by evaluating the appropriateness and validity of specific AI tools for medication-related uses and continuing to educate patients on trusted sources for medication information.”