“When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker.“
Damn! That was ... fast. Both literally during the attack and in a sense of "we knew this was coming, but we now have a real-world first public case".
Some interesting tidbits:
The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the "agentic attacker" scenario the industry has been forecasting.
Bang tokens against a wall until something works. I really hope hf goes beyond normal disclosure and makes a lot of this public, if possible (after they rotate everything, filter PII, etc)
The attack was initially surfaced through AI-assisted detection.
Nice. Blue vs. red in real time. Noise and "alert fatigue" have been huge problems in the past, even with high-cost solutions, so glad to see this is already improving.
LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity
I wonder if this was true "decoy activity" or simply the agent banging its tokens against the wall w/ some hallucinations along the way. Again, hope we get some datasets out of this.
these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker.
Yeah, sadly this was also been "forseen" by a lot of people. Similar to how some things can't be processed by LLMs (studies about crime / violence / PII related stuff), this was bound to happen. I guess it's good that it happened to hf, which has an incentive to be open about this. Hopefully we get past the "the enemy will use it", and realise that the enemy is already using it, and move on to "help the blue team".
I think it's clear now that not your inference, not your tokens is obvious. You need on-prem models, even if they're below SotA.
Autonomous, AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed.
Yeah, lateral movement at agentic speed is worrying. Hopefully this leads to improved postures everywhere. Zero trust and all that. Plus, it's not clear how separated the workers were from the pod itself (was it even using containers? With the amount of LPEs out there, that seems unwise, and we might see a move towards proper virtualisation for anything touching outside data).
Autonomous, AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed.
AI models definitely do not operate at "machine speed." A human can definitely be faster than these trillion parameter thinking models.
A human can definitely be faster than these trillion parameter thinking models.
I’m thinking this can be true when the steps and muscle memory is known beforehand to the human. Otherwise we also have to stop, think and experiment for a while before we can proceed.
The model also does more of a brute force approach of trying a bunch of different things; testing theories, giving up, trying another thing. A lot of the LLM benchmarks do not care how long an LLM takes (as long it's below some upper bound in some cases). A human stopping to think can be faster compared to the model from going down a bunch of bad paths.
The interesting aspect of this attack and the how guardrails prevented them from analyzing the attack.
When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
While falling back to their local hosted Open Weight model, gave them the ability to analyze the attack, while also keeping their internal structure / data private.
Right!? And it is such a relief that their chief strategist isn’t out tweeting about how FUD should be weaponized by the government against foreign competitors.
Missed opportunity to mess with it!
Key quote:
“When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker.“
Damn! That was ... fast. Both literally during the attack and in a sense of "we knew this was coming, but we now have a real-world first public case".
Some interesting tidbits:
Bang tokens against a wall until something works. I really hope hf goes beyond normal disclosure and makes a lot of this public, if possible (after they rotate everything, filter PII, etc)
Nice. Blue vs. red in real time. Noise and "alert fatigue" have been huge problems in the past, even with high-cost solutions, so glad to see this is already improving.
I wonder if this was true "decoy activity" or simply the agent banging its tokens against the wall w/ some hallucinations along the way. Again, hope we get some datasets out of this.
Yeah, sadly this was also been "forseen" by a lot of people. Similar to how some things can't be processed by LLMs (studies about crime / violence / PII related stuff), this was bound to happen. I guess it's good that it happened to hf, which has an incentive to be open about this. Hopefully we get past the "the enemy will use it", and realise that the enemy is already using it, and move on to "help the blue team".
I think it's clear now that not your inference, not your tokens is obvious. You need on-prem models, even if they're below SotA.
Yeah, lateral movement at agentic speed is worrying. Hopefully this leads to improved postures everywhere. Zero trust and all that. Plus, it's not clear how separated the workers were from the pod itself (was it even using containers? With the amount of LPEs out there, that seems unwise, and we might see a move towards proper virtualisation for anything touching outside data).
AI models definitely do not operate at "machine speed." A human can definitely be faster than these trillion parameter thinking models.
I’m thinking this can be true when the steps and muscle memory is known beforehand to the human. Otherwise we also have to stop, think and experiment for a while before we can proceed.
The model also does more of a brute force approach of trying a bunch of different things; testing theories, giving up, trying another thing. A lot of the LLM benchmarks do not care how long an LLM takes (as long it's below some upper bound in some cases). A human stopping to think can be faster compared to the model from going down a bunch of bad paths.
The interesting aspect of this attack and the how guardrails prevented them from analyzing the attack.
When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
While falling back to their local hosted Open Weight model, gave them the ability to analyze the attack, while also keeping their internal structure / data private.
I expect this will kick off a series of 'GLM helped HuggingFace where Fable wouldn't' type headlines.
And its gonna be factually correct. Proprietary models will create slop for you, but will refuse to fix its sloppy security because "safety".
I'm so glad OpenAI doens't have some multi-hundred billion dollar financial incentive to drum up how they have AGI.
Right!? And it is such a relief that their chief strategist isn’t out tweeting about how FUD should be weaponized by the government against foreign competitors.