Using Generative AI for medical diagnostics is dangerous and irresponsible. The AI companies should have a visible disclaimer everywhere. But because they don't, even the so-called "AI experts" are being confused.
Case in point. As I mentioned in my review of the November 29, 2023, congressional hearing “Understanding How AI is Changing Health Care,” there was a 'covfefe' that everyone seemed to have missed: https://sergeiai.substack.com/p/what-if-a-physician-doesnt-use-ai.
Rep. Gus Bilirakis:
“Mr. Shen, can you tell us about the role of generative AI, what it is, and what its potential can be within the health care sector?”
Peter Shen, Head of Digital Health – North America, Siemens Healthineers:
“With generative AI here, we see the greatest potential in the ability for the AI to consume information about the patient themselves. So, when a patient goes to get an exam for a diagnosis, leveraging generative AI can help identify precisely what diagnosis should be looked for. Another area where generative AI benefits medical imaging is in interpreting the images themselves. It can translate complicated medical language into layman’s terms for the patient, helping them better understand the test results from their exam.”
Wrong! We don't use hallucinating AI for precision medicine. Shame on you, Mr. Shen.
If the experts of AI make such egregiously erroneous statements, what can you expect from the users of AI?
Yes, as per Sam Altman, AI can be magical, but in healthcare, we need more than magic. We need precision, accuracy, and reliability. The thought of using generative AI in medical diagnostics is as absurd as using a Magic 8-Ball for brain surgery. It’s not just irresponsible. It’s a gamble with human lives.
The statistical nature of these machines is revealed I've noticed, when trying out ChatGPT for low-level editing of text: it tends to wander away from the task the longer it's allowed to generate answers. It has no real internal coherence. I had to keep telling it over again what it's job was exactly.
This lack of true internal coherence was dramatically revealed the other day when it went bonkers for 6 hours. 😆
Big fan of your work! But not the best example (I’m a cardiologist). 1)Instructions not that crazy despite what pt said. trunk rotations are ok as long as arms close to chest. Nothing about arm exercises were recommended. 2)pts after cardiac surgery are inundated w instructions from OT, PT, written, etc. So why even ask a bot? 3) I could not come close to reproducing the output. And no matter how u ask, you always get the boiler plate “ but ask your provider for specific instructions…”. 4) what was on reference 5?? 5) activity recommendations change over time. 2 wks vs 2mos.
I've had Perplexity deliver these pastiche/mashup answers fairly consistently. I asked about the autopilot for a Saab JA37 and mingled in was a bunch of stuff about the C-1 autopilot for the B-17. Because they were both made by Honeywell.
The amount of examples one can have showing that 'stochastically constrained random next token generation' is not 'understanding' is practically infinite.
They are just ungrounded structuralist large language models that are manipulating tokens. They are as aware of the world as is my sofa. It is amazing that they perform as well as they do, they are incredibly interesting systems from my point of view as a NLP researcher, and they are (eventually) going to revolutionise human-machine interfaces. But people should stop trying to use them as some kind of search engine.
I've noticed those sorts of pastiche answers from GenAIs.
Since users may innocently trust the answer (as if it were aware, or like a human, or had common sense), if I were the product maker of an AI, I would also generate (additional) very large and stern warnings regarding medical advice, or anything similar that could cause harm, especially as these things become more mainstream. For example, imagine an elderly person, on medications, feeling feeble and told by a friend or caretaker to look things up on the computer, or given a caretaker app... why wait for things like this to happen? (My two cents...).
This is not a surprise if you have listened to Lex Fridman’s interview of Stephen Wolfram who explains very clearly the limitations of AI’s ability to generate accurate answers.
The comments so far here have been really useful. The need for caution extends beyond AI to all sources of information, including seemingly authoritative sites and medical peeps on YouTube. There's a broader challenge in working critically with what we consume, AI or otherwise. Advice can sometimes be dangerously incorrect, highly polarised, or not relevant to someone's specific needs/history.
The way AI mimics these complexities shows how difficult it is to work out what's going to provide benefit versus what might harm.
Chatbots will have their use in the medial field. Primarily for summarizing doctor visits, doing paperwork, etc. They can aid with diagnosis too, but for providing educated guesses. Those will be specialized chatbots that are trained on a lot of medical data and validated over a few years.
Then there should be a totally non-chatbot tool that checks if a drug order filed by a doctor is likely to be harmful for the current patient.
digitaurus said: "But people should stop trying to use them as some kind of search engine."
The problem, d, is that search engines Google and MS/Bing are promoting their AI/LLMs as an adjunct to their searches, putting the AI bot text on top of all other searches. Users have to actively skip over the LLM reply to get to the real information.
Gary, please keep kicking the Prophets of Ai in the nuts! As you well know (since I first quoted you on autonomous killer automobiles in 2017), the past decade has been a virtual Gulag for Ai-skeptics, myself included. And it is a relief to see the facade finally collapsing. Cheers!
I am vastly amused by the AI brigade claiming people using their Information System need to do their own research to validate and qualify the Information the Information System spewed forth. In straightforward words what they are saying is their vaunted System is unreliable, thus useless, and the people who use it are idiots.
Using Generative AI for medical diagnostics is dangerous and irresponsible. The AI companies should have a visible disclaimer everywhere. But because they don't, even the so-called "AI experts" are being confused.
Case in point. As I mentioned in my review of the November 29, 2023, congressional hearing “Understanding How AI is Changing Health Care,” there was a 'covfefe' that everyone seemed to have missed: https://sergeiai.substack.com/p/what-if-a-physician-doesnt-use-ai.
Rep. Gus Bilirakis:
“Mr. Shen, can you tell us about the role of generative AI, what it is, and what its potential can be within the health care sector?”
Peter Shen, Head of Digital Health – North America, Siemens Healthineers:
“With generative AI here, we see the greatest potential in the ability for the AI to consume information about the patient themselves. So, when a patient goes to get an exam for a diagnosis, leveraging generative AI can help identify precisely what diagnosis should be looked for. Another area where generative AI benefits medical imaging is in interpreting the images themselves. It can translate complicated medical language into layman’s terms for the patient, helping them better understand the test results from their exam.”
Wrong! We don't use hallucinating AI for precision medicine. Shame on you, Mr. Shen.
If the experts of AI make such egregiously erroneous statements, what can you expect from the users of AI?
Yes, as per Sam Altman, AI can be magical, but in healthcare, we need more than magic. We need precision, accuracy, and reliability. The thought of using generative AI in medical diagnostics is as absurd as using a Magic 8-Ball for brain surgery. It’s not just irresponsible. It’s a gamble with human lives.
The statistical nature of these machines is revealed I've noticed, when trying out ChatGPT for low-level editing of text: it tends to wander away from the task the longer it's allowed to generate answers. It has no real internal coherence. I had to keep telling it over again what it's job was exactly.
This lack of true internal coherence was dramatically revealed the other day when it went bonkers for 6 hours. 😆
Big fan of your work! But not the best example (I’m a cardiologist). 1)Instructions not that crazy despite what pt said. trunk rotations are ok as long as arms close to chest. Nothing about arm exercises were recommended. 2)pts after cardiac surgery are inundated w instructions from OT, PT, written, etc. So why even ask a bot? 3) I could not come close to reproducing the output. And no matter how u ask, you always get the boiler plate “ but ask your provider for specific instructions…”. 4) what was on reference 5?? 5) activity recommendations change over time. 2 wks vs 2mos.
I've had Perplexity deliver these pastiche/mashup answers fairly consistently. I asked about the autopilot for a Saab JA37 and mingled in was a bunch of stuff about the C-1 autopilot for the B-17. Because they were both made by Honeywell.
The amount of examples one can have showing that 'stochastically constrained random next token generation' is not 'understanding' is practically infinite.
They are just ungrounded structuralist large language models that are manipulating tokens. They are as aware of the world as is my sofa. It is amazing that they perform as well as they do, they are incredibly interesting systems from my point of view as a NLP researcher, and they are (eventually) going to revolutionise human-machine interfaces. But people should stop trying to use them as some kind of search engine.
I've noticed those sorts of pastiche answers from GenAIs.
Since users may innocently trust the answer (as if it were aware, or like a human, or had common sense), if I were the product maker of an AI, I would also generate (additional) very large and stern warnings regarding medical advice, or anything similar that could cause harm, especially as these things become more mainstream. For example, imagine an elderly person, on medications, feeling feeble and told by a friend or caretaker to look things up on the computer, or given a caretaker app... why wait for things like this to happen? (My two cents...).
This is not a surprise if you have listened to Lex Fridman’s interview of Stephen Wolfram who explains very clearly the limitations of AI’s ability to generate accurate answers.
The guy is an idiot.
The comments so far here have been really useful. The need for caution extends beyond AI to all sources of information, including seemingly authoritative sites and medical peeps on YouTube. There's a broader challenge in working critically with what we consume, AI or otherwise. Advice can sometimes be dangerously incorrect, highly polarised, or not relevant to someone's specific needs/history.
The way AI mimics these complexities shows how difficult it is to work out what's going to provide benefit versus what might harm.
Chatbots will have their use in the medial field. Primarily for summarizing doctor visits, doing paperwork, etc. They can aid with diagnosis too, but for providing educated guesses. Those will be specialized chatbots that are trained on a lot of medical data and validated over a few years.
Then there should be a totally non-chatbot tool that checks if a drug order filed by a doctor is likely to be harmful for the current patient.
digitaurus said: "But people should stop trying to use them as some kind of search engine."
The problem, d, is that search engines Google and MS/Bing are promoting their AI/LLMs as an adjunct to their searches, putting the AI bot text on top of all other searches. Users have to actively skip over the LLM reply to get to the real information.
well, if LLM is teached upon medical corpus, then its answers are correct. I tested this on our LLM and it gives very sound answer.
Gary, please keep kicking the Prophets of Ai in the nuts! As you well know (since I first quoted you on autonomous killer automobiles in 2017), the past decade has been a virtual Gulag for Ai-skeptics, myself included. And it is a relief to see the facade finally collapsing. Cheers!
Yet our media elites main focus this week has been a black pope on the image generator
I am vastly amused by the AI brigade claiming people using their Information System need to do their own research to validate and qualify the Information the Information System spewed forth. In straightforward words what they are saying is their vaunted System is unreliable, thus useless, and the people who use it are idiots.