Validating Clinical Artificial Intelligence (AI) from the Perspective of a Neuroradiologist

In 2018, Florent Saint-Clair, EVP at Dicom Systems had a conversation with Dr. Alan Pitt, Neuroradiologist at the Barrow Neurological Institute in Phoenix, AZ on the challenges of implementing Artificial intelligence (AI) in healthcare. In this discussion, Dr. Alan Pitt shared his candid viewpoints from the perspective of a radiologist with 20 plus years on the frontline of healthcare and innovation. In response to the plethora of new AI solutions and applications in the market intended to support physicians, how can organizations ensure the algorithms are clinically relevant? The process of operationalizing an algorithm in live clinical workflows requires an enterprise-wide roadmap and cross-departmental buy-in and support. This conversation covers AI-related products for clinical relevance, a checklist developed for Dicom Systems in collaboration with Dr. Pitt, and strategies for evaluating AI solutions.

The excerpt from this webinar examines:

• How clinicians define transformative healthcare
• The need for AI technology in healthcare
• Possible barriers to entry and implementation

Please note: This conversation has been edited for length and clarity.

Transformative Healthcare Powered by AI

Florent: If there is any kind of AI that’s going to be transformative enough to be of interest to you as a physician in your job today at Dignity Health, what would a technology company consider to be low-hanging fruit that really needs to be addressed ASAP, that AI can solve?

Dr. Alan Pitt: In my core expertise as a neuroradiologist, the low-hanging fruit, are the big public health problems that I see current imaging not addressing very well. The first thing I would choose would be a rapid volumetric assessment of the brain that would allow me to correlate treatment with changes in the brain morphology. Number two would be the ability to look at multiple sclerosis plaques. An AI algorithm that could quickly assess those plaques and show me changes over time would probably be a relatively easy, but a very useful thing to do. A third thing would be to look at the cortical surface of the brain because I don’t think we do a very good assessment of many mental disorders. If we could get a better assessment of the cortical surface of the brain, which is something I cannot do as an imager, but an AI algorithm could possibly do, that would be a third place that I think AI can do something that I cannot do myself.

Florent: There are some very real use cases that are not really being tackled by AI or machine learning (ML). To be specific, the impetus for the development of those algorithms today generally comes from the technology side, right?

Dr. Alan Pitt: That’s right. As a general rule, if you want clinicians to adopt whatever technology you have, my advice is to be a ripple, not a wave. If I have to get out of my chair to go use your technology, I’m pretty much not gonna do it. I need to have everything within my workflow for me to really start using it at scale and to use it very frequently. This is really true of any AI algorithm. It really has to be something in the background that’s super easy for me to use. If you don’t do that, it doesn’t really matter how much value that AI algorithm brings to me, I’m pretty much not gonna do it at scale. I’m not gonna use it frequently. And so, the biggest barrier for me as a radiologist is not the AI algorithm itself, it’s how you’re gonna integrate that into my workflow if you really want it to be a successful offering.

Florent: Today, in your clinical environment, how many algorithms are actually being used clinically? Considering the amount of noise in the industry coming from all the large and small companies, why is that today?

Dr. Alan Pitt: Zero. Let me give you another story and it may give you some context. My friends developed a company, Montage Health, that allowed me to do data analytics on my own reports. Think of it as a Google crawler for radiology reports, where it would consume my reports, and allow me to see them by referring, by type of imaging, by age of patient, and all sorts of things. It would allow me to see all sorts of dashboards. And I knew that my EMR could not give me these reports. I wanted this application in my hospital environment for two purposes. I wanted it for quality assessment, which is a big driver for the healthcare system, but I also wanted it for research. I wanted to be able to figure out do I have enough patience to even do a research study on a particular problem? Can I identify those patients? I came back home to Dignity and I started going everywhere I could, asking people, “Will you help me get this thing?” It literally took me two and a half years to get through all the red tape within budgeting, but also the privacy and security within my hospital to bring this application to the institution. You cannot expect clinicians, even if they’re super passionate about your algorithm, to spend a couple of years knocking on doors, trying to break down all the silos to get the necessary budgeting, privacy, and security pieces that are gonna be required.

Barriers To Adoption of Healthcare AI

Florent: How to operationalize this technology is one of the key barriers to entry, right?

Dr. Alan Pitt: It’s a major barrier. The production of the algorithm is the easy part. But even if you have the algorithm, you have to figure out a way to get it deployed. And it’s really no different than a start-up company going to a venture capital firm to present a new enterprise. The things that a VC will look for in the technology is the secret sauce, the process and the experience of the people building the technology. All of those things are ingredients that go into the decision making, and so, before we even get to the operationalization of an algorithm, there are some real questions that need to be asked.

Florent: So when you get beyond all the noise in our industry, what is it that you could be doing as a physician to educate yourself on what’s possible in AI and be a contributor to the clinical input the technology side really needs from physicians?

Dr. Alan Pitt: Physicians make a lot of assumptions while the AI industry is really interested in promoting itself. AI companies kind of over-promise and under-deliver a little bit. I’d really like to get a good assessment of what is possible, and I think many clinicians would be happy to say, “Look, this is my problem. I’ve gotta count up the number of dots on a field and it’s just way too many dots to count and characterize. Can you accurately do this for me?” Can you measure the amount of volume loss in the brain? Can you come up with an accurate and rapid way to quantitate fluid and fat inside the brain? I can show you by imaging how to differentiate those, but I don’t really have a sense of how strong the algorithms are to correct for that. I think that getting a better dialogue would help clinicians frame their tasks for the tech community. I think we’re starting to have that dialogue, but it’s still pretty limited. I’m usually presented with an algorithm as opposed to having a conversation.

Florent: How many of those algorithms do you run into, Alan, that you go, “Okay, well, that’s nice, but I can’t use this today?”

Dr. Alan Pitt: If I’m gonna invest in technology, there’s the nice-to-have things and the have-to-have things. So there are certain things that make my world better and make it easier for me. There are certain things that I can’t do for myself, but the algorithms can provide. I want to emphasize that I believe it’s a true partnership between health and tech. Now, you may not be able to think like a doctor, but you can think like a patient and you can start to identify problems that you may have seen for you or a loved one that you think technology might be able to solve. And you can begin that conversation by bringing a clinician in early, to ask “My loved one has this problem. I noticed that they’re unable to do XYZ. What do you think about that?” And if I were able to help you better understand this tumor or better evaluate this multiple sclerosis and follow it over time. If I was able to do that for you, would that be of value for you in caring for this patient?” If you can get the clinician to say, “Yes, that would definitely be of value to me.” Then I think you begin a conversation around, should this be developed? How much value does it bring?