In your opinion, do you buy into the panic about AI eliminating humanity or potentially disrupting and taking down the internet? Is this a marketing ploy?
Give me your best thoughts on this most recent call for AI slowdown.
This sounds like a nice idea for a day or two, but I think how it presents in reality could be rough. The power grid might be down for a few days causing food spoilage, it could be taking down banks so people don't have access to their money, it could be screwing up IT at a hospital which interferes with treatment and care, we may not be able to fill up our gas tanks... I don't think it'll be "nefarious-gun-on-a-robot-dog" any time soon, but rather something that still causes major major issues.
The first thing on the top of my head is maybe that too large of a percentage of GDP is being spent on infrastructure for these companies that we don’t really need that could go to more useful investments.
No. Frontier AI companies need the government to shield them from becoming commodity. That's why they're begging for regulation.
On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.
So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
From my perspective it looks like working to deep in AI makes you crazy. No other way to explain all the weird bullshit these billionaires throw around.
If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.
If you look around I don't think we're at the point where even if AI go rogue it could just disrupt us for a while. No real major concern to humanity but yeah I guess having rules and regulations would be plus point. So for now there shouldn't be anything to panic about.
Agree. Ais gone rogue now could be bad, but not humanity-ending bad. Just a hiccup. Of course if it somehow ends with human deaths then even if one human dies because of a rogue AI, that's a failure. But I can't imagine rogue AIs with enough real world power to harm humans in some way. What I do believe however is that humans will find a way to do harm with AI without AI going rogue, as with any other thing humanity comes close to. So the scenario is somehow similar to the nuclear bomb in the 40s, I don't fear our AIs going rogue and killing everyone, I fear a bad guy behind a very smart AI with very bad intentions.
I don't consider this a "yes/no" question. It seems quite clear to me that the idea of "AI doom" is neither certain (100% probable) nor impossible (0% probable). The question is where on the continuum is the truth. And I don't know. And I'm pretty sure nobody else does either.
That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.
So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.
This reminds me a lot of Y2K. There were dire warnings about planes falling out of the sky, financial system meltdown, all systems being broken, etc. In the end nothing happened.
You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.
- They have internal models that still show there's gas in the tank, in terms of improvements
- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.
- The true demand for high-intelligence tasks is lower than what they need to justify their spending
- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again
- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture
Astra, Claude -- these aren't "Jesus on a keyboard" somewhere. They don't assemble into one big brain and then answer everyone at the speed of light.
It's weights. Spun up billions of times over.
If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.
But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.
Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.
Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).
Give me your best thoughts on this most recent call for AI slowdown.
at this moment? no. one day, perhaps soon? yes
also, taking down the internet for a couple days doesn't sound so bad. we could all hang out again.
i wish internet had a day off on sunday
Can't lie, I'd be all for the internet dropping for a few days... Except no Amazon purchases lol
This sounds like a nice idea for a day or two, but I think how it presents in reality could be rough. The power grid might be down for a few days causing food spoilage, it could be taking down banks so people don't have access to their money, it could be screwing up IT at a hospital which interferes with treatment and care, we may not be able to fill up our gas tanks... I don't think it'll be "nefarious-gun-on-a-robot-dog" any time soon, but rather something that still causes major major issues.
No, I don't buy into that at all.
I think that the genAI companies themselves present a greater danger to us (arguably not an existential danger, though) than genAI itself.
What's the difference?
The first thing on the top of my head is maybe that too large of a percentage of GDP is being spent on infrastructure for these companies that we don’t really need that could go to more useful investments.
No. Frontier AI companies need the government to shield them from becoming commodity. That's why they're begging for regulation.
On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.
So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.
They’re not giving any details about why or how.
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
Yes, they are.
https://www.anthropic.com/threat-intelligence-report-septemb...
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
if AGI will come in versions i think we are safe
From my perspective it looks like working to deep in AI makes you crazy. No other way to explain all the weird bullshit these billionaires throw around.
Sure AI sucks, but not to that degree (yet)
I'm not panicking but there are very real risks.
If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.
If you look around I don't think we're at the point where even if AI go rogue it could just disrupt us for a while. No real major concern to humanity but yeah I guess having rules and regulations would be plus point. So for now there shouldn't be anything to panic about.
Agree. Ais gone rogue now could be bad, but not humanity-ending bad. Just a hiccup. Of course if it somehow ends with human deaths then even if one human dies because of a rogue AI, that's a failure. But I can't imagine rogue AIs with enough real world power to harm humans in some way. What I do believe however is that humans will find a way to do harm with AI without AI going rogue, as with any other thing humanity comes close to. So the scenario is somehow similar to the nuclear bomb in the 40s, I don't fear our AIs going rogue and killing everyone, I fear a bad guy behind a very smart AI with very bad intentions.
I don't consider this a "yes/no" question. It seems quite clear to me that the idea of "AI doom" is neither certain (100% probable) nor impossible (0% probable). The question is where on the continuum is the truth. And I don't know. And I'm pretty sure nobody else does either.
That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.
So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.
[1]: https://en.wikipedia.org/wiki/Neuromancer
What is the concrete scenario that is based on reality which results in the percentage increasing above zero?
This reminds me a lot of Y2K. There were dire warnings about planes falling out of the sky, financial system meltdown, all systems being broken, etc. In the end nothing happened.
You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.
All of the following are true:
- They have internal models that still show there's gas in the tank, in terms of improvements
- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.
- The true demand for high-intelligence tasks is lower than what they need to justify their spending
- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again
- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture
No. At least not with an LLM. Maybe in 50 years with real AI. A lot of non technical people entering the conversation which is very annoying.
Astra, Claude -- these aren't "Jesus on a keyboard" somewhere. They don't assemble into one big brain and then answer everyone at the speed of light.
It's weights. Spun up billions of times over.
If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.
But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.
Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.
Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).
No, just a push for regulatory capture, nothing more