Continuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?
A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.
I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.
Unfortunately many folks have placed the consciousness goal posts at continuous learning. Such an advancement would be devastating for their conclusion.
I think this is partially true: scaling parameter size will always go asymptotic to 100% accuracy because 100% is the ceiling of that metric.
However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.
And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.
It will eventually be super useful, and so disruptive that it will make today's LLMs look like nothing particularly special IMHO.
As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.
Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.
Serving requests where every user has their own set of self-updating weights will absolutely murder whatever minimal margin the AI providers have today
Who says the weights have to be duplicated in their entirety? Even that will likely be worth it. Imagine an OpenAI owning the ERP and CRM of a big chunk of the Fortune 500.
There are two questions about that stability I have.
One, things like catastrophic forgetting and falling into incoherence.
Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.
Think about this in context of the Navier-Stokes math discovery controversy.
Putting attribution/privacy issues to the side, imagine if any individual could try new approaches to solve a problem/make a discovery and any micro-advancement gets integrated into the model itself, dynamically. This could transform progress from the slow "write a paper, get peer reviewed and published, use published data to inform future work" to a system with a centralized repository of concepts, attempts and results, including failed approaches already tried. How much work do humans waste replicating failed approaches?
Someone completely random halfway around the world could trigger a prompt that solves a blocker that prevents my solution from working. Who cares about AGI or "can models invent anything" when we could have a system that automatically synthesizes individual human thought into a rich network of aggregate human memory.
That's the target OpenAI/Anthropic should be evangelizing, not an AI Daddy Overlord or agentic script kiddie hellscape.
You could try out some version of this today, with a wiki. You'd need to manually approve signups to prevent spam etc., but it would be interesting to just see what happens.
Continuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?
A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.
I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.
Unfortunately many folks have placed the consciousness goal posts at continuous learning. Such an advancement would be devastating for their conclusion.
Which is a choice, not a "law":
https://arxiv.org/abs/2510.13786
https://www.alphaxiv.org/abs/2512.20264
https://arxiv.org/abs/2607.05155
I think this is partially true: scaling parameter size will always go asymptotic to 100% accuracy because 100% is the ceiling of that metric.
However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.
And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.
I wonder how (and if) continuous learning models will achieve stability.
They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run
It will eventually be super useful, and so disruptive that it will make today's LLMs look like nothing particularly special IMHO.
As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.
Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.
Serving requests where every user has their own set of self-updating weights will absolutely murder whatever minimal margin the AI providers have today
Who says the weights have to be duplicated in their entirety? Even that will likely be worth it. Imagine an OpenAI owning the ERP and CRM of a big chunk of the Fortune 500.
There are two questions about that stability I have.
One, things like catastrophic forgetting and falling into incoherence.
Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.
I do it all the time. Am I stable? Depends who you talk too.
So would it be a 42 trillion parameter model, because that's how many tokens there are in the training data?
Does that mean, it's not compressed anymore?
Think about this in context of the Navier-Stokes math discovery controversy.
Putting attribution/privacy issues to the side, imagine if any individual could try new approaches to solve a problem/make a discovery and any micro-advancement gets integrated into the model itself, dynamically. This could transform progress from the slow "write a paper, get peer reviewed and published, use published data to inform future work" to a system with a centralized repository of concepts, attempts and results, including failed approaches already tried. How much work do humans waste replicating failed approaches?
Someone completely random halfway around the world could trigger a prompt that solves a blocker that prevents my solution from working. Who cares about AGI or "can models invent anything" when we could have a system that automatically synthesizes individual human thought into a rich network of aggregate human memory.
That's the target OpenAI/Anthropic should be evangelizing, not an AI Daddy Overlord or agentic script kiddie hellscape.
You could try out some version of this today, with a wiki. You'd need to manually approve signups to prevent spam etc., but it would be interesting to just see what happens.