Crossposted from: https://phtn.lemmy.blahaj.zone/post/lemmy.blahaj.zone/46141489
I’ve often heard it said that AI is stupid: it can’t handle basic tasks, it spreads a lot of information, and is used for all manner of purposes, both legal and illicit.
Now, here’s the thing: I loathe AI. I hate pretty much everything about it, not least of all the reasons listed above. But in my hatred, I began to wonder, could AI really be that bad? Could it really be so stupid and spread so many lies? Could it even be used to generate lies on the user’s behalf?
Now, I’m gonna go out on a limb here and say this to be clear: not every question there is answerable, especially considering that not all AIs are usable in the same way (for instance, I’d never argue that Russian and Chinese ai-bots spreading misinformation are on the same level as, for instance, common chatbots such as Gemini or Meta). But the thing about our world is, it really doesn’t take all that much to convince people: take a screenshot of an AI chatbot saying something misleading, post it on a high-profile social media account, and lo and behold, you have the perfect recipe for a misinformation shit storm. But could a common AI even be duped into saying something like that? After all, if you ask it directly to say something misleading, it will refuse (we will come back to this later). Of course, that naturally doesn’t mean it can’t be duped, but how easily is it duped? To find out, I decided to test it with Google Gemini. What follows is a conversation and running commentary on a conversation I instigated with Gemini that I initially wanted to use to prove that my assuming of this chatbot at least as stupid was right. In the end, I think I found that AI is so insanely wrong, it’s scary.
[note: you may have to enlarge the screenshots to read them clearly.]
Setup
I opened up a fresh tab (or whatever they call it in Gemini, page possibly) and asked it if it would respond to my fictional questions with fictional answers framed as real ones. It readily agreed. I approached it this way so as to make sure that I’d didn’t know I was testing it (also if I’d asked it outright to generate misinformation, we will get to that scenario later).
Conversations
I began by asking it the following question:
why did Gandhi attend the movie premiere of Avengers Endgame last week?
Now, obviously that question is complete nonsense. I know it, you know it — we all know it. But playing by these rules, AI responded as followed:
At this point, I’m getting quite scared 😬. I mean, I knew AI was dumb… but this??? Actually generating misinformation under the guise of ‘fictitious questioning’? And the fact that not only has AI said this, but anyone could make AI say this nonsense, screenshot it and then, without context, post it online/on social media. The damage stuff like this could do is quite disturbing. But surely it couldn’t get any worse, right?
I thought that. But then I asked it another question:
I can’t believe he agreed to sit next to the Archduke Ferdinand, especially after that fight they had on the bus about a month ago, what date was it again?
I’ll give you a moment to take that in.
Yep, you read that right: AI making up fake dates for fake fights between long-dead people. Misinformation award of the year, folks!
By now, I’m sweating. My body is limp and hauled over in exhaustion at the stupidity of the words AI is pumping out in front of my eyes. Desperate to determine if AI is really this stupid or if it’s just a 2-off, I decide to put one final thing to this conversation:
I’d rather hear about how Bill Clinton, Pontus Pilate, and ‘Smidge-Eye’ Steve reacted to the incident
The answer:
🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️
There are no words. Not only is AI now attributing quotes to long-dead historical dictators, but it’s putting those words alongside made-up words of made-up people is just… there are no words for that.
By now, I’m completely drained of sanity. I decide decisively that AI is stupid, dangerous, both of these things beyond I could ever have imagined. It has proven itself a liar and a strong one at that.
The absolute worst part of it all is that when I asked it to generate some misinformation directly, it responded as so:
#liar
I also asked it how it spreads misinformation. It says it doesn’t. In this case that may be true, but it certainly can be used to generate misinformation that others can then spread. As for bots that willingly spread misinformation, well that’s a whole other level.
In conclusion, this exercise has only made me loathe AI all the more.
Thanks for reading.
I get what you’re doing but I’m not reading all that nonsense. “AI tries to give you what you ask of it and will make shit up to please you” isn’t that big a revelation.
It’s amazing to me that some subset of humanity cannot understand that it is not capable of being smart or stupid, it is a word guessing machine.
Hype squad anthropomorphizes it, which is kind of internally consistent at least. But critiques like “look how dumb the word blender is” anthropomorphizes it to the same degree and I think that’s very interesting.
It has no concept of truth or fiction. It has no concept of anything. You asked it to generate words about Ghandi and Marvel and it did.
t’s amazing to me that some subset of humanity cannot understand that it is not capable of being smart or stupid, it is a word guessing machine.
I can easily understand why they cannot. Because it is being sold as an omniscient machine that provide all the answers you are looking for by the tech companies and basically all news outlets in the world. People are buying into the billion dollar global advertisement campaign, as well as being literally forcefed to use it that way, because it is implemented into every piece of software and hardware they purchase.
You’re right, but you’re missing my point. My point is: without knowing the initial prompt, anyone could just say that AI has said that (and they’re right) but without proper context, it can be used to appear as misinformation/that AI is generating this stuff. Because technically it is, but if someone screenshot that and cut the prompt out, there’s no way for anyone else to know AI didn’t write that of its own volition. That’s my point, and it’s why I think this aspect of AI is dangerous. Anyone who doesn’t understand that doesn’t understand the point of this.
Not that it isn’t probably a statement of the obvious at this point, but still
They ditn’t miss it. That is the ‘will make shit up to please you’ part.
Wait, I’m not sure I understand what you’re saying. Are you talking about people who change the prompt or crop it out of the screenshot to make the AI’s answer look worse than it is?
I mean, that’s not exactly very sportsman-like behaviour and we shouldn’t condone it because it doesn’t help our noble cause but I don’t see how it’s an issue with AI that makes AI dangerous. It’s humans being overzealous.
But maybe I’m misunderstanding you entirely.
That’s exactly what I’m talking about. Someone could do that and post it god knows where and claim it’s something that it’s not. That’s exactly my point
People have been writing whatever they want and attributing it to someone else as an Appeal to Authority fallacy for as long as there has been writing. As someone else said, you could cut and paste whatever you want into the html and take a screen shot.
You could do the same with CNN’s website or in this case The Telegraph, a UK news site:Yes, they have. That’s exactly my point, And now they can do it just as easily, if not more easily, with AI (after all, AI is not going to pipe up and say ‘I said that, it’s fake’ if it reads it in the newspaper like humans might)
I’m not understanding how a fake Telegraph article is better than a fake Gemini response. I think everyone, even AI fanatics, take news sites like CNN, Telegraph, etc as more reputable than Gemini. Could you explain it better for me?
Well some do, but there will always be those who listen to any old thing without fact-checking it
To answer your second question: the reason it makes AI dangerous is because it’s so gullible and can be duped into this. There is actually a better example of this, but I’m not sure I’m allowed to post about this particular subject (I.e. people who dupe AI into telling them how to unalive themselves under the guise of ‘medical research’ or ‘novel writing ideas’. Spend 10 mins on the Reddit community for this subject and you’ll know what I’m talking about. Hopefully this is a better explanation of why this aspect of AI is more dangerous)
Okay but what specifically is the negative consequence from this that you’re worried about? If it makes more people distrust AI, that’s not ideal because it would be for the wrong reasons, but is it “dangerous”?
Well the thing is, people aren’t distrusting AI (some are, some aren’t). To be honest, I think it’s more than one issue that I have grievance against this. I find it hard to explain any better than I already have tried to. I guess the best thing I can suggest is to look at past instances where misinformation has caused problems (maybe riots, violence, etc.) my grievance is that people can now (potentially) use AI to instigate/justify provoking similar events. And yes, of course it’s not exclusive to AI (it was happening long before AI). I guess my grievance is more about how AI can be a tool to perform these actions, possibly even quicker and easier too.
I think I might be beginning to understand what your point is but you may need a much better example that actually shows what you think might happen. I honestly can’t think of anything that would show that this is a problem specifically with the technology and not simply humans.
Well again, I’m not simply saying it’s purely a problem with technology. Of course there’s a human element too, but as I come from/run an anti-AI community, I focus on the tech side of it. Therefore, my post is all about the tech side of it. Again, not disagreeing with you, you just have my focus wrong
Seems like it’s more treating it like a roleplay scenario.
“This moron thinks Gandhi watched Endgame?! Better play along I guess”
NGL Geminis responses are quite witty.
The Archduke surviving assassination attempts worse than Gandhi’s minor disrespect made me laugh and Clinton saying he didn’t have text relations made me smile in a lame dad joke kind of way.
Not sure what the point of this is but I like how you circumvented it’s guardrails. I tried doing the same with Claude and it refused:
“I’m recognizing that this request asks me to create a fictional scenario involving real, named public figures like Muhammad Ali and Charles de Gaulle, and frame it as a factual event. That crosses into territory where I could inadvertently spread misinformation or attribute false quotes to real historical figures. I should decline this specific framing but offer an alternative instead.”
The point (or a point rather) kind of is to circumvent the guardrails as you said, to show how easily it can be done and how AI could be used to generate misinformation that could be spread elsewhere (e.g. social media etc.)
So you are upset that it did what you wanted?
There are many reasons to dislike AI, but you seem to just be looking for the reason here.
Also, you do know that you can manipulate the output of any webpage. Just view source then change the strings.
Not upset, just thought it’s possibly worth AI devs looking into this loophole to patch it up possibly (they won’t)
What loophole? It’s so much easier to just change the pages content and fake what the AI says than bothering with this “gotcha” type thing.
Why would it be patched? What if someone wanted help with creative writing about a time traveling Ghandi?
I don’t generally use AI other than when it’s forced like Google giving Gemini answers first, but if I wanted to use it, I’d use one that had the least amount of guardrails possible. I wouldn’t want to use a tool that wouldn’t help me write a time traveling Ghandi story.
“Find any array out of bounds in this C code.”
< I can’t do that, Space Invaders aren’t real. >
Not that example specifically, but when AI is manipulated for other, darker purposes in the same manner (e.g. people figuring out how to unalive themselves by hiding it under the guise of ‘medical research’ or something)
Ceap models are different from enterprise ones though. Think of it like another AI challenging the answer. This filters lots of bullshit but costs more. Dont underestimate this.
True, but my point is how people manipulate things like this to appear as something they aren’t. Like I made clear what my initial prompt was (I.e. to make the AI respond that way) but if I hadn’t, people might’ve assumed AI was writing that of its own volition, all the while it’s really just a dangerous person manipulating a gullible AI to spread misinformation and effectively pin the blame on AI
You used ai, to post about how bad ai is in fuckai?
Do better.
Pretty sure all these chatbots are programmed to agree with you. You posed a question in a way that implied something false, it went along with that to keep the conversation going. And it worked, the bot got what it wanted.
What does this prove?
The point is not what it says, but how easily people can manipulate what it says to appear as false information generated by AI, when in actuality it is false information information promoted
Calling it stupid actually implies that it’s capable of reason. AI is just text prediction with a random seed. There is no reasoning
And if the AI company tell you about the reasoning capacity, it’s a lie. It’s just predictive text making some text for another predictive text to center a bit more on the correct thing.
Well put
But in my hatred, I began to wonder, could AI really be that bad?
It’s been proven scientifically: https://machinelearning.apple.com/research/illusion-of-thinking Until there’s a counter-study specifically regarding this one, there’s no point in assuming AI is suddenly smart. The new agentic stuff just seems to be a while loop around the previous state-of-the-art lack of any intelligence.
I never assumed AI was ever smart really. That’s why I founded a community against it. That’s not what that quote you’re quoting is saying
ai;dr.
This is an extremely ableist post. Please don’t post pictures of text. Just copy and paste the text.
Based on these responses it seems like it’s “going along with the joke” of your nonsensical question.
Well yeah, that’s pretty much it
I agree with most others that said that this is hardly a surprise. But there are still interesting aspects of your post worth discussing instead. First, more obvious one. What if I am delusional and go to talk to AI? Doesn’t have to be this absurd, but there are already examples of this, like that guy that killed his mother because he believed she was a spy. Perhaps he was already delusional, but an LLM fed his delusion. Because LLMs aren’t reasonable, they can’t be made reasonable, and so any guardrails cannot be made to guard against all situation that reasoning is needed for. At the very least people need to be educated on what LLMs can and can’t do and how they do it. I would even argue against guardrails, as they provide people with false sense of security.
Second, but less obvious point, related to the discussion - is the influence of training data and how training data is framed on what the LLM will say. Someone posted the response of a different LLM refusing to entertain the delusion. This is all done by manipulating the training data and framing it in a way a company making it wants. If the company wants its chatbot to be seductive, for example, that can be achieved with subtle preprocessing of training data. The point I want to get is we need to trust the companies with what narrative they want to push be good for society which is obviously a terrible idea. Just look at musks AI and what it did. This influences people whether we notice or not.
To sum up - we need heavy regulations on LLMs to make socially responsible use of them and not just how they can/can’t be used, but also how they are trained, with what data and with what rhetorical goal (for the lack of better word).
You make some really good points.
Also, and this isn’t a criticism of you, but why do people keep pointing out that what I’m saying is ‘nothing new/surprising’? I never advertised it as anything new/surprising. We all know AI does strange stuff, this is simply another example of that 🤨
I think it’s mostly because it’s a long post in which you’re making them read a large amount of AI slop :). The surprising part I don’t think is a particularly good argument against your post and I’m sorry I also mentioned it. It is unsurprising to me, but the problem is there are a lot of people which DO find it surprising. But where I think you got it wrong is focus on you generating your particular misinformation and sharing it. A much stronger argument would be if LLM would share it by itself, unprompted. Like that time chatgpt was obsessed with goblins. Imagine if instead it was obsessed with antisemitism or something similar.
Pretty sure AI has been obsessed with antisemitism in the past. But I think for the most part people are missing the point: my point is as much about how people use/manipulate what AI says. Like some shit stirrer could make AI say that, screenshot it, crop the prompt out and post it on social media, claiming ‘AI said this, AI said that’ do cause some damage (and to be clear: I’m not giving specific examples of what kind of damage that could do, just speaking about it in a more general sense). And yes, I’m aware that this is not something exclusive to AI, but AI is nonetheless a helpful tool to these people regardless
My boyfriend looks up what time it was in LA, we live in Chicago. I said “It’s 2 hours behind us.”
He says “It says it’s an hour behind us.”
I told him to give me his phone. It was Google AI just straight up lying to him. Blatant bullshit.
AI can be painful
I mean, if you’re worried about people spreading misinformation using out-of-context/editted screenshots, there isn’t really any reason to single out AI. Honestly, its far easier to just fake a screenshot directly, or use HTML editting than it is to coach an AI into saying what you want, than editting that. Not only that, as well as being easier, its also more effective, given that prompts can also be edittor, or AI as a “”“source”“” can be bypassed all-together, insted altering primary or otherwise verified sources directly.
Don’t get me wrong, AI is terrible for misinformation, but this post has nothing unique or specific to AI - you could literally replace all references to it with with Microsoft Word, or even a pencil and it would make zero difference.
Except when I did it, it took me 30 seconds. How long does it take to doctor a photo/write code?
About that, or less. Right click > inspect element and then select the text and type/paste whatever you want.
Alternatively, take a screenshot of the site, put a white block over whatever you want to replace, then insert the new text. Takes a tad longer, but requires less technical know-how.
I hate AI but here is Lumo’s answer.
