Google has tested its medical chatbot with real patients for the first time, and the study was published Oct. 8 in the medical journal The Lancet. The chatbot, called AMIE, is a research system Google built to hold the kind of conversation a doctor has when working out what is wrong with someone. Until now it had been tried mostly in simulations, such as conversations with actors playing patients. Chatbots do well in those tests, but researchers at Beth Israel Deaconess Medical Center in Boston, who ran the study with Google, say little is known about how they do with real people.
The patients had each booked an urgent appointment at the hospital's primary care clinic for a new problem that was not an emergency. Before the visit, about 100 of them chatted with AMIE by text from home. It asked about their symptoms and medical history, offered possible explanations for them to go over with their doctor, and wrote a summary for the doctor to read beforehand. A doctor watched every conversation live over a video call, ready to step in if a patient seemed at risk, was upset or asked to stop.
No conversation had to be stopped. Patients said AMIE listened well, explained things clearly and put them at ease, and their opinion of AI rose after the chat and stayed up after they saw their doctor. Doctors who read AMIE's summaries said they usually helped them prepare. Separate doctors, who did not know which answers came from the chatbot, rated AMIE's lists of likely causes and its suggested next steps about as good overall as those from the patients' own doctors. When doctors checked patients' records weeks later, the true cause was usually somewhere on AMIE's list, though its first guess was right only about half the time.
The chatbot was not flawless. The watching doctors caught one time it stated something false as if it were true, and stepped in a handful of other times to clarify. One doctor also worried a patient had been made anxious when AMIE listed lymphoma, a cancer, among the possible causes. The patients' own doctors made care plans that reviewers rated more practical and better value for money, which the researchers say is expected: AMIE could not examine anyone or see their medical records. Patients also had doubts about how private their answers were and how honest the chatbot was.
This was a small first study at a single hospital that did not compare patients who used the chatbot with patients who didn't. It left out pregnant patients and people with mental health concerns, and no one in it needed emergency care, so how AMIE handles an emergency is still untested. It was designed to test whether such a chatbot can be used safely and whether people accept it, not whether it makes anyone healthier. Alphabet, Google's parent company, paid for the study. Google says larger trials are needed, and AMIE remains a research system, not a product. It is not yet clear whether a chatbot like this could work without a doctor watching or whether it saves doctors time. Adam Rodman, who directs AI programs at the hospital's research institute, says future studies also need to find out what builds patients' trust in it.