Major publishers and aggregators are also under pressure to add chatbots to their products. Elsevier is offering Scopus AI, for example. It uses RAG and a knowledge graph, and the answers seem to come from the summaries of the papers it selects; however, it may still be very misleading. The answers are slightly different every time, depending on the papers it includes. This is not obvious to most users. Users are asking that the summaries be consistent, and Elsevier's reps were not making it obvious that this is not possible. They also said it was great for topics they did not understand, but underwhelmed when they were experts in the topic. This is a red flag to me - nearly any source looks good when you know very little, and it is hard to know what you don't know. Scopus AI may be fine if users understand that it is a place to generate ideas to explore further (and verify), but it may be a problem if users assume that it is trustworthy.
In my opinion, this sort of thing is expected, given that science has degenerated significantly from just a seeking of knowledge. A lot of science is more of a game now that is just about securing more funding for research that really won't help anyone. It's also become a religion, not in the way it operates, but regarding its societal role: people need something to believe in because religion is being displaced.
So, I wonder: does this AI fakery show the danger of AI (which is indeed a danger) or is it actually better at showing the farce of modern science?
Here's the thing... Hallucinations and mistakes from chatbots can degrade science. But given the evidence coming out about how much data is faked and images manipulated in papers submitted to even the best journals, my bigger fear is genAI being used to manipulate data, generate fake images, and just up your H-index. These are tools for the complete enshittification of science, until we come up with rigorous tools to detect genAI output.
"could quickly overwhelm their reviewing processes"
Could? It's already happening and I'm refusing article reviews by the dozens. Last year I'd not have refused a single one in fear of being dropped from the reviewer panel(s), but this is easily overwhelming every productive and tenured scientist with some reviewing track record RIGHT NOW[1].
[1] Especially if you work in anything related AI, duh.
The Ur problem is "Publish or Perish" where quantity is the sole analytical metric. Now add ChatBots spewing word salad at the speed of electrons and the necessary and sufficient conditions for the destruction of scientific publishing in this decade. Since we know the ChatBot manufacturer's attitude is "give me my money, who cares who gets my pox" we shouldn't expect any help there. My solution: anyone submitting a paper shown to be Chatbotted should have their name placed in a publicly accessible file at, say, the National Science Foundation and permanently banned from publication.
As LLMs start to ingest the output of other LLMS and their own work, possibly through RAG, the output will become ever more unreliable, bizarre and useless.
I gave a lecture on AI and Social Work last night. Highly recommended your subreddit and X. Thanks for this. Always good. Time for philosophy of science to weigh in.
Derek Lowe's "In the Pipeline" blog at Science wrote about this a year or so ago. He noted a slew of questionable papers of Chinese origin being published on organometallics (if I doth remember accurately.)
Honestly, I see this as a net good for the science community because Gen AI is just being used by authors who were previously just spewing out human generated garbage. What comes to mind are the Helen Pluckrose et. al. who published absolute garbage in journals to show how bad the journals really were.
The issue with bad writing has plagued the scientific community for years and if LLMs force them to be better then I think it will also clean up bad human writing too.
I thought that was a tumour at first, not (apparently) a rat with giant testicles looking up at its own penis that towers above its own body. It looks so wrong, I'm still somehow not confident in what I'm seeing, like as if I'm seeing something supernatural and want a rational explanation for it. Not sure a polite way to put it, but just wtf? Imagine coming across that looking at the paper on your own and seeing that, without any knowledge of it being from AI, trying to process what you're looking at!
I concur that reputation and integrity are critical.
"FWIW both articles came from China, locked in a race with the US to dominate the science journal productivity statistics."
Then "science journal productivity statistics" must be pretty simple to game. Is it the lack of consequences that have teeth when a paper is shown to be "smoke & mirrors"? Maybe an economist could use game theory to sort this out.
Stop and consider the current state of affairs in the media. Rathergate in 2004 cost Dan Rather his job and the producer he was working with never worked in the industry again. Today it would be ignored or get a Pulitzer Prize.
Major publishers and aggregators are also under pressure to add chatbots to their products. Elsevier is offering Scopus AI, for example. It uses RAG and a knowledge graph, and the answers seem to come from the summaries of the papers it selects; however, it may still be very misleading. The answers are slightly different every time, depending on the papers it includes. This is not obvious to most users. Users are asking that the summaries be consistent, and Elsevier's reps were not making it obvious that this is not possible. They also said it was great for topics they did not understand, but underwhelmed when they were experts in the topic. This is a red flag to me - nearly any source looks good when you know very little, and it is hard to know what you don't know. Scopus AI may be fine if users understand that it is a place to generate ideas to explore further (and verify), but it may be a problem if users assume that it is trustworthy.
In my opinion, this sort of thing is expected, given that science has degenerated significantly from just a seeking of knowledge. A lot of science is more of a game now that is just about securing more funding for research that really won't help anyone. It's also become a religion, not in the way it operates, but regarding its societal role: people need something to believe in because religion is being displaced.
So, I wonder: does this AI fakery show the danger of AI (which is indeed a danger) or is it actually better at showing the farce of modern science?
Remember the 1990's? How internet would be a fountain of good and true? Boy, history does rhyme.
Here's the thing... Hallucinations and mistakes from chatbots can degrade science. But given the evidence coming out about how much data is faked and images manipulated in papers submitted to even the best journals, my bigger fear is genAI being used to manipulate data, generate fake images, and just up your H-index. These are tools for the complete enshittification of science, until we come up with rigorous tools to detect genAI output.
"could quickly overwhelm their reviewing processes"
Could? It's already happening and I'm refusing article reviews by the dozens. Last year I'd not have refused a single one in fear of being dropped from the reviewer panel(s), but this is easily overwhelming every productive and tenured scientist with some reviewing track record RIGHT NOW[1].
[1] Especially if you work in anything related AI, duh.
The Ur problem is "Publish or Perish" where quantity is the sole analytical metric. Now add ChatBots spewing word salad at the speed of electrons and the necessary and sufficient conditions for the destruction of scientific publishing in this decade. Since we know the ChatBot manufacturer's attitude is "give me my money, who cares who gets my pox" we shouldn't expect any help there. My solution: anyone submitting a paper shown to be Chatbotted should have their name placed in a publicly accessible file at, say, the National Science Foundation and permanently banned from publication.
Excellent.
Hopefully that title will go viral.
Maybe should submit this post to a slew of journals.
As LLMs start to ingest the output of other LLMS and their own work, possibly through RAG, the output will become ever more unreliable, bizarre and useless.
It is called Model Collapse.
I gave a lecture on AI and Social Work last night. Highly recommended your subreddit and X. Thanks for this. Always good. Time for philosophy of science to weigh in.
Derek Lowe's "In the Pipeline" blog at Science wrote about this a year or so ago. He noted a slew of questionable papers of Chinese origin being published on organometallics (if I doth remember accurately.)
Honestly, I see this as a net good for the science community because Gen AI is just being used by authors who were previously just spewing out human generated garbage. What comes to mind are the Helen Pluckrose et. al. who published absolute garbage in journals to show how bad the journals really were.
The issue with bad writing has plagued the scientific community for years and if LLMs force them to be better then I think it will also clean up bad human writing too.
https://www.theatlantic.com/ideas/archive/2018/10/new-sokal-hoax/572212/
Metrics, targets, you know the drill.
I wish I lived in the world where the techbros were held to account....
I thought that was a tumour at first, not (apparently) a rat with giant testicles looking up at its own penis that towers above its own body. It looks so wrong, I'm still somehow not confident in what I'm seeing, like as if I'm seeing something supernatural and want a rational explanation for it. Not sure a polite way to put it, but just wtf? Imagine coming across that looking at the paper on your own and seeing that, without any knowledge of it being from AI, trying to process what you're looking at!
Science is so yesterday. Obedience is the new thing.
It is not possible to keep a chatbot maker accountable because somebody used it to write a fake paper. Or wrote a fake story that went viral.
Chatbots are tools that have their uses. They will get smarter, and that surely result in both more opportunities and more problems.
Need to focus on solid regulation, not on recalling chatbots, holding makers "accountable", etc.
I concur that reputation and integrity are critical.
"FWIW both articles came from China, locked in a race with the US to dominate the science journal productivity statistics."
Then "science journal productivity statistics" must be pretty simple to game. Is it the lack of consequences that have teeth when a paper is shown to be "smoke & mirrors"? Maybe an economist could use game theory to sort this out.
Stop and consider the current state of affairs in the media. Rathergate in 2004 cost Dan Rather his job and the producer he was working with never worked in the industry again. Today it would be ignored or get a Pulitzer Prize.