ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers.
Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.
The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.
That's happening across the board when it comes to models that feed off online information. Create a website with a false claim, have AI slurp it up and spit it back out when a user asks a question.
That's very troubling. The scariest part is how easy it is, one Instagram post and you may change the customer phone number of Airbnb, or who is leading the election poles.
It’s not just hackers - since legitimate sites block AI crawlers, the AI are just grabbing whatever will let them read anything - killing legit sites and feeding slop farms and feeding misinformation! The future is so bright!
Did takedowns for a brand's fake support numbers and the hard part was never finding them, it was that killing one PDF just moved it to a new Medium post by morning.
There's an age-old SEO / spamming thing happening here, but it's made worse by the LLMs just being unbelievably credulous. They'll wrap anything up in a veneer of authenticity, and Google's search AI box only adds to that.
I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.
ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers.
Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.
The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.
What’s the specific example and how does it work
That's happening across the board when it comes to models that feed off online information. Create a website with a false claim, have AI slurp it up and spit it back out when a user asks a question.
Happy elections.
That's very troubling. The scariest part is how easy it is, one Instagram post and you may change the customer phone number of Airbnb, or who is leading the election poles.
It’s not just hackers - since legitimate sites block AI crawlers, the AI are just grabbing whatever will let them read anything - killing legit sites and feeding slop farms and feeding misinformation! The future is so bright!
Did takedowns for a brand's fake support numbers and the hard part was never finding them, it was that killing one PDF just moved it to a new Medium post by morning.
There's an age-old SEO / spamming thing happening here, but it's made worse by the LLMs just being unbelievably credulous. They'll wrap anything up in a veneer of authenticity, and Google's search AI box only adds to that.
I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.