What’s the specific example and how does it work

That's very troubling. The scariest part is how easy it is, one Instagram post and you may change the customer phone number of Airbnb, or who is leading the election poles.

I don't think they did. Or those who did have been "encouraged" to leave a long ago.

Because their leaderships have defunded / fired all ethics researchers looking into their ongoing crimes and failures in favor of a bs "AI safety" narrative based around a creepypasta about an AI monster time traveling backwards to torture their staff for not maximizing shareholder value.

They don’t assume all data is of equal quality

Recent political discourse has all but proved that a majority of the populace in many countries is quite removed from facts in general. I don’t think that there is much hope for critical thinking.

Because AI companies select out the kinds of people who would ask silly questions like "wait, is this data good" that hamper the eternally "profitable" quest for "scaling".

You can't count on that, but quantity increases visibility don't it, and it takes a single prompt to spit it all out. Couple that with social media bots and you've got a decent modern and super cheap propaganda machine. Now imagine you toss millions of dollars at that idea.

It’s not just hackers - since legitimate sites block AI crawlers, the AI are just grabbing whatever will let them read anything - killing legit sites and feeding slop farms and feeding misinformation! The future is so bright!

Did takedowns for a brand's fake support numbers and the hard part was never finding them, it was that killing one PDF just moved it to a new Medium post by morning.

There's an age-old SEO / spamming thing happening here, but it's made worse by the LLMs just being unbelievably credulous. They'll wrap anything up in a veneer of authenticity, and Google's search AI box only adds to that.

I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.

It's happening at the startup sphere too. There are companies now creating fake company pages to recommend and advertise products so that the LLMs can consume them.

...carried out by malicious actors, through automated campaigns...

Back at benevolence or malevolence.

I really think the only solution is "don't put google AI summaries at the top of every search".

AI tools are notoriously unreliable, and people that use them *on purpose* generally know and recognize that. If grandma, who is not super-internet-literate, types a query into google and gets a phone number, she's going to assume that it's the correct number and not the result of a lossy data store that's under active (and constant) attack.

It used to be that you could reasonably verify your source with an HTTPS cert (so you can probably trust the number that's on Delta Airlines' website), and pop up scary warnings for grandma if those certs failed. Now every piece of misinformation has a valid cert from google or meta or X, so determining the truth is much more difficult and time consuming.

I would not assume the invisible hand of the market is gonna solve this problem for us. It will be tricky to get this problem to even register at the labs, who are worried about existential risk first and paying (coding) customers 2nd.

I've been seeing the back-side of this as a mod at the Julia Discourse. Pagerank/SEO spam is ridiculously obvious and trivial to detect/block — links are all that matter. Many GEO spam posts are similarly obvious: someone posting about a cryptocurrency customer support hotline on a programming language forum is definitely spam. But recently spammers/scammers been using AI to _tailor_ posts to look much more authentic, posting "How to use Julia to analyze market statistics" referencing (without links!) a particular crypto exchange and then (sometimes) going back later to edit in phone numbers or the like.

What makes this even more painful is seeing all the good faith answers that such GEO posts spur from the community. It's far more abusive than SEO spam.

My wife runs the Facebook page for our community pool. It's a really nature oriented community, and as a April Fools joke, she posted something like "Announcing our new landscaping: we cut down all the pesky sun blocking trees, and replaced the grass with beautiful AstroTurf."

A few weeks later she searched for the pool on Google, and the AI review has taken the post seriously and mentioned that our pool features beautiful AstroTurf landscaping.

IIRC, the AI review repaired itself a few weeks later.