A few weeks ago, I was working in the ER, and I saw a patient with abdominal pain who had been referred by a telemedicine provider. The patient was very distraught because she had been told that she could have a life-threatening infection in her abdomen, and she was tearful about it.
Usually before seeing patients, I perform a “chart biopsy,” gathering some basic information from the chart on their medical history as represented in the electronic record. In this case, I could see a note from earlier today by the telemedicine provider. It was very well written, organized, and made a compelling case that the patient could have SBP, or spontaneous bacterial peritonitis, a feared complication of liver failure where bacteria from the gut infect ascitic fluid, or extra fluid around the bowels caused by liver dysfunction. I could see that this patient did have a history of liver disease, so I was immediately concerned, and began thinking through the next steps I would take, including performing diagnostic paracentesis, a procedure where a needle is introduced into the abdomen to sample the fluid to detect any infection.
When I went to bedside and interviewed the patient, however, it sounded like her pain had been going on for many months, and she didn’t really have other characteristic symptoms of infection, like fever or malaise. On exam, she did have some tenderness, but I didn’t detect the characteristic bloating and “fluid wave” that would indicate a large volume of extra fluid in the abdomen.
What’s Going On?#
Confused, I returned to the EHR and explored her clinical notes more thoroughly. It appeared that she had been seen earlier that week and had a CT scan for the same pain, and the scan showed no fluid in her abdomen. While it’s certainly possible that a small amount of fluid could have developed since then, it’s unlikely that it would be enough to put her at risk of a serious infection. Moreover, her story didn’t support a diagnosis of infected ascitic fluid. The pain should be new and different, not going on for months.
Puzzled, I went back to the note from the referring physician and realized my mistake.
A Perfect Clinical Note#
The note from the telemedicine provider looked excellent, and had led me to weigh it heavily. Clinical notes are hard to write, and even harder to make readable to other providers. A note of this quality must indicate a very competent clinician who was very concerned about this patient. Unconsciously, I was biased to trust the note not because of what it said, but how it smelled.
The Analogue of “Code Smell”#
In programming circles, the term “code smell” captures the phenomenon where competent programmers can judge the quality of code based on subtle signs in how the code is written. For nearly any computational task, there are many ways to perform it algorithmically, and still more flexibility in how the algorithm is expressed in code, right down to how variable names are chosen and how lines of code are indented, organized, and commented on.
In emergency medicine we are perhaps the experts on assimilating clinical notes, having to scan through the chart on a patient we have probably never met trying to understand what is going on and where the important information is located. On this day, I realized that I had unconsciously developed a sense of “documentation smell,” where I judged the veracity of human-written content (not machine-generated labs or structured data) based on the length, complexity, vocabulary, completeness, organization, and tone of what the author actually chose to type (or in many cases, dictate). Surely this sense had also been honed by authoring thousands of clinical notes myself, and considered how they might be read or used in the future, choosing my words carefully.
Fool Me Once#
Inspecting the well-organized note, I suddenly realized that it had almost certainly been written by an LLM, either on a bespoke healthcare platform from an enterprising startup, or perhaps a curious provider experimenting with ChatGPT.
The note was well-reasoned, but ignored obvious inconsistencies, like the patient’s clinical picture didn’t resemble SBP, and that she had no history of fluid accumulation in her abdomen that would place her at risk for this condition. She had a CT scan earlier in the week that had all but eliminated the possibility of such an infection, yet the note failed to address this key fact.
My suspicion is that the author of the note didn’t review the medical record in any detail, and whether the diagnosis of SBP originated from the clinician or an algorithm, they hadn’t thought critically about the differential diagnosis prior to sending the patient to the ER. But my oh my, their note looked and sounded beautiful!
A Serious and Pervasive Problem#
Clearly healthcare isn’t the only area where human-generated responses convey as much in their form as their content. From people writing their senator, to college admissions essays, even casual emails, the way we write conveys as much meaning as what we say, like body language on the page. And the formerly unfakeable traces of effort, “proof-of-work” to borrow a phrase, used to be something you could trust. We’re entering a world where written language, generated for centuries with great effort and by at least one human mind, now can’t be trusted by “smell.” Among the many disruptions ahead, this may be the most fundamental.
How to Respond?#
At the moment, a critical reading can detect the tone, organization, and soulless perfection of an LLM-generated response. But not always, and with even a little prompting models can easily pass the “Turing test” of human authorship, even for vigilant readers who work with AI all the time.
Clearly, we need new ways to separate human-inspired work from algorithmic slop. But “slop” isn’t always, or even usually, sloppy in ways we can notice. As long as copy-paste is legal, how are we ever to know if a jumble of words originated from a human brain?
The long-derided in-person interview may need to make a resurgence. And in healthcare, it remains vital to go to the bedside and gather data from the source.