Explainable Explainability for Clinical LLMs

Discussions of explainability for large language models (LLMs) tend to focus on methods that identify what parts of a neural network encode certain concepts or words, with the hope that by illuminating the meaning of parts of the LLM “black box” we can be certain of how it is making a decision. These methods resemble those used in convolutional neural network architectures processing imaging data from, say, a CT scan, where parts of the source image important to the label are highlighted to help humans understand how the categorization was performed.

But when we talk about “explainable” systems, who is owed the explanation? While one approach may satisfy an AI engineer, it might not make sense to the user, particularly a clinician who is making a high stakes decision. Ideally the model outputs could be explained in a way that resembles a clinician’s own decision-making process.

For medical imaging, a “heat map” of significance makes sense, because the dimensionality of the data is quite high – millions of billions of pixels are fed into the model, and they can be grouped according to their spatial relationships. But some types of clinical data are discrete, made of up of individual values like vital signs or laboratory values. Other clinical data, like a description of a pain symptom, are unstructured and occupy a very large semantic space.

A Simple Example

#

Imagine this vignette:

INPUT: 45 year-old auto mechanic with a history of hypothyroidism and peripheral vascular disease presents with left-sided chest pain that is sharp, non-radiating, not related to exertion...

OUTPUT: This patient is at elevated risk for coronary artery disease.

EXPLANATION: The following neural network activations led to this decision: "maleness" "middle-aged male" "chest pain" "vascular diseases"

While this explanation tells us something about the clinical example, it’s quite vague, and does little to illuminate the decision-making process. What was the significance of these activations? How were they combined to arrive at the conclusion? We can see that the model assumed the auto mechanic is male, but little else of value.

While it’s possible in principle to create a system that monitors the neural network activations and can provide detailed explanations of what is happening, we’re not there yet. In the meantime, is there a simpler method that would satisfy our curiosity?

Modeling Counterfactuals

#

Whether they admit it or not, most clinicians make decisions based on explicit or implicit algorithms based on clinical factors that are present in a case. A patient may have risk factor or clinical features that increase or decrease the suspicion for a disease. If you ask a doctor, “why did you order that test?” she can usually rattle of a few clinical features that raised her suspicion.

If you wanted to understand the decision more, what could you ask her? One simple approach would be to explore counterfactuals – “if the patient weren’t male, would you still order the test?” But understanding what inputs would ultimately change the output, you can infer the importance of each clinical detail.

Assuming $n$ binary input data elements (yes/no data points like “male/female”) and a binary output, you could re-run the model $n$ times with counterfactuals to determine the sensitivity of the output to each of these dimensions. This would require repeating the analysis $n$ times, or $O(n)$. You could explore counterfactuals involving two differences from reality with $\frac{n(n-1)}{2}$ iterations or $O(n^2)$ time, and in general the number of iterations is just the binomial:

Here $k$ is the number of simultaneous differences from reality you wish to explore. Now $O(n^k)$ isn’t impressive efficiency, but remember that the dimensionality of relevant clinical data is quite small, and counterfactuals up to order $k=3$ are probably sufficient to satisfy users.

We can further reduce the number of steps by asking the LLM itself, in a separate prompt and query, which clinical data are most important. The counterfactual explainer then would only need to consider the top clinical factors, so the effective $n$ to consider could be much smaller.

For $n=5$ clinical factors deemed important, with $k=3$, we calculate the number of repeated inferences as:

So in this reasonable clinical scenario, increasing the number of inferences by an order of magnitude is sufficient to explore the relevant counterfactuals. While this increase in compute is not a negligible cost, it seems quite reasonable if a high-stakes clinical decision is being considered.

The alternative, counterfactual explanation to a clinician would look like this:

INPUT: 45 year-old auto mechanic with a history of hypothyroidism, stroke, and peripheral vascular disease presents with left-sided chest pain that is sharp, non-radiating, not related to exertion...

OUTPUT: This patient is at elevated risk for coronary artery disease.
 
CALCULATED list of clinical factors that would reverse the decision: 
[ peripheral vascular disease AND stroke ] 

EXPLANATION: If a history of peripheral vascular disease were not present AND a history of stroke were not present, this patient would not be considered high risk for coronary artery disease.

As a clinician, I find this explanation much more understandable and relatable, and it would increase my comfort level in trusting the model’s outputs to know how it would respond in alternative clinical situations.