So the first incident was on March 19th and the second incident is March 22nd —- evidently the attackers maintained persistence through maybe two separate credential rotation efforts.
Friendly reminder that just because someone is building security software it doesn't mean they are competent and won't cause more harm than good.
Every month the security team wants me to give full code or cloud access to some new scanner they want to trial. They love the fancy dashboards and lengthy reports but if I allowed just 10% of what they wanted we would be pwned on the regular...
Most of corporate security nowadays involves "endpoint security solutions" installed on all devices, servers and VMs, piping everything into an AI-powered dashboard so we can move fast and break everything.
I audited Trivy's GitHub Actions a while back and found some worrying things, the most worrying bit was in the setup-trivy Action where it was doing a clone of main of the trivy repo and executing a shell script in there. There was no ref pinning until somebody raised a PR a few months ago. So a security company gave themselves arbitrary code execution in everyone's CI workflows.
Aqua were breached earlier this month, failed to contain it, got breached again last week, failed to contain it again, and now the attackers have breached their Docker Hub account. Shit happens but they're clearly not capable of handling this and should be enlisting outside help.
The ref pinning part is almost worse than no pinning. You can pin the action itself to a commit SHA, sure. But half the actions out there clone other repos, curl binaries, or run install scripts internally. Basically none of that is covered by your pin. You're trusting that the action author didn't stick a `curl | bash` somewhere in their own infra.
Audited our CI a few months back and found two actions doing exactly that. Pinned to SHA on our end, completely unpinned fetches happening inside.
My hypothesis is that generally, there's no quality floor at which security departments are "allowed" to say "actually, none of the options on the market in this category are good enough; we're not going to use any of them". The norm is to reflexively accept extreme invasiveness and always say yes to adding more software to the pile. When these norms run deeply enough in a department, it's effectively institutionally incapable of avoiding shitty security software.
Fwiw w/r/t Trivy in particular,I don't think Trivy is bad software and I use it at work. We're unaffected by this breach because we use Nix to provide our code scanning tools and we write our own Actions workflows. Our Trivy version is pinned by Nix and periodically updated manually, so we've skipped these bad releases.
From having worked at and consulted with security software producing companies as well as security software consuming ones, I would say the security companies are worse than average at security.
And their security teams more cynical.
Sometimes they deliberately hire lower aptitude candidates to run internal security to prevent them from getting distracted by the product.
In other cases they are getting high on their own supply, more or less.
Jack Welch style management seems to take a deeper toll in this sector.
It doesn't help that a lot of security software is pretty niche. It's unreasonable to expect most candidates to know it or have experience.
In one case I was one of exactly two people out of 500 that had used the product as a paying customer. Neither of us was in management.
After a year or two the CISO drifted over and asked me to show him how to use the product, but he was more interested in soundbytes than actually using the system.
It became a powerpoint exercise and I collected my attaboy.
Wasn't this discovered already last week, on Friday, that the threat actor had replaced the legit images with malware images? And republished 75 out of 76 tags?
No, the actor reappeared. This article is not fully updated. On March 22nd, the actor compromised their DockerHub account and published new Docker images.
Whenever possible, credentials shouldn't be inside the sandbox either. Credential proxying, or transparent credential injection, for example with Sandcat: https://github.com/VirtusLab/sandcat
The sandbox will need internet access (to update data) and you will need to send code to test into it; so compromise already equals leaking all your code, without even breaking the sandboxing
The sandbox will need internet access (to update data) and you will need to send code to test into it; so compromise already equals leaking all your code, without even breaking the sandboxing
Compromising all code in one directory is bad.
Compromising all my data in all other directories, including mounted cloud drives, is worse.
I restrict most dev tools to access only the current directory.
second breach in a month from the same initial credential compromise. the first rotation didn't fully revoke access. the attacker walked right back in. no persistence needed.
My initial thought is that if this isn't a new compromise, Trivy must not have rotated the old credentials. They claim, however,
We rotated secrets and tokens, but the process wasn't atomic and attackers may have been privy to refreshed tokens
… does anyone know what exactly they're talking about, here? To my knowledge, GH does not divulge new tokens after they're issued, but it depends on the exact auth type we're talking about, and GH has an absurd number of different types of tokens/keys one can use.
OpenClaw creator made some related claims, that as soon as he created a GitHub organization with a new name, somehow it was stolen from him, and he had to ask Github people to do it for him atomically.
Well, not my best 2 weeks at work, now I have to fill out a dozen forms and sit trough a shitload of meeting, just because they got pwned (twice, or once, but really badly :D )
"GitHub's own security guidance recommends pinning actions to full commit SHAs as the only truly immutable way to consume an action"
Why doesn't GitHub just enforce immutable versioning for actions? If you don't want immutable releases, you don't get to publish an Action. They could decide to enforce this and mitigate this class of issue.
Why doesn't GitHub just enforce immutable versioning for actions?
I always wish these arguments came with a requirement to include a response to "well, what about the other side of the coin?", otherwise, you've now forced me to ask: well?
The two sides of the coin: Security wants pinned versions, like you have, so that compromises aren't pulled in. Security does not want¹ pinned versions, so that security updates are pulled in.
The trick, of course, is some solution that allows the latter without the former, that doesn't just destroy dev productivity. And remember, …there is no evil bit.
(… I need to name this Law. "The Paradox of Pinning"?)
(¹it might not be so explicitly state, but a desire to have constant updated-ness w/ security patches amounts to an argument against pinning.)
Their question isn't about pinned versions, it's about immutable versions. The question is why it is possible to change what commit "v5" refers to, not "why would you want to write v5".
You already don't get updates pulled in with the system unless they swap the version out from under you, which is not a normal way to deploy.
It amounts to an argument against pinning in a (IMO) weird world view where the package maintainer is responsible for the security of users' systems. That feels wrong. The user should be responsible for the security of their system, and for setting their own update policy. I don't want a volunteer making decisions about when I get updates on my machine, and I'm pretty security minded. Sure, make the update available, but I'll decide when to actually install it.
In a more broad sense I think computing needs to move away from these centralised models where 'random person in Nebraska'[0] is silently doing a bunch of work for everyone, even with good intentions. Decisions should be deferred to the user as much as possible.
it might not be so explicitly state, but a desire to have constant updated-ness w/ security patches amounts to an argument against pinning
When you want to update, you update the hashes too. This isn’t an issue in any other packaging ecosystem, where locking (including hashing) is a baseline expectation. The main issue is developer ergonomics, which comes back to GitHub Actions providing very poor package management primitives out of the box.
(This is the key distinction between updating and passively being updated because you have mutable pointers to package state. The latter gets confused for the former, but you almost always want the former.)
This isn't a bad distinction that you've made, I just think even lockfiles (what you're suggesting, essentially) still fall prey to the same paradox I'm suggesting.
Yes, lockfiles prevent "inadvertent" upgrades, in the sense that you get the "pinned" version in the lockfile. So if we go with the lockfile, we're now on the "pinned" side of the paradoxical coin. Yes, we no longer get auto-pwned by supply chain, but security's problem is "why are we not keeping up to date with patches?" now, since the lockfile effectively prevents them.
And then you see tooling get developed, like what Github has in the form of Dependabot, which will automatically update that lockfile. Now we're just back to the other side of the paradoxical coin, just with more steps.
(This isn't to say we shouldn't do lockfiles. Lockfiles bring a lot of other benefits, and I am generally in favor of them. But I don't think they solve this problem.)
I don’t think this is a paradox, it’s just a process. You use lockfiles to establish consistent resolutions, and then you use dependency management tooling to update those lockfiles according to various constraints/policies like compatibility, release age, known vulnerabilities, etc.
(Another framing is that you might want floating constraints for compatibility reasons, but when actually running software you basically never want dependencies changing implicitly beneath you, even if they fix things. Fixes should always be legible, whether they’re security relevant or not.)
Version tags should obviously be immutable, and if you want to be automatically updated you can select 1.0.*, if you don't you just pick the version tag.
Auto upgrade to version deemed OK by security team. Basically you need to get updates that patch exploits then wait and be more patient for feature upgrades.
So, in the context of me questioning "yes, but exactly how is this supposed to work", you're essentially punting the question into a black box that won't betray us.
In the real world, though, we don't have a magic little black box: we have to actually implement that.
The only answer I have seen from real world security teams is variations of "why wouldn't we be keeping up with updates?", and that's an unpinned dep.
I assume this is because it is modeled after git tags, and at this point it would be a major change to move away from this. But it should probably get started at some point.
Because the true name of the feature is VisualSourceSafe actions. It's all over the code of the runner if you take a second to look, and the runner, like the rest of the feature, is of typical early 2000s Microsoft quality, which is to say, none at all.
Even then, that's only immutable for the workflow config. Many workflows then go on to pull in mutable inputs downstream (eg: default to "latest" version).
A better question perhaps is why we’ve allowed ourselves to be so vulnerable by a single provider (GitHub). Supply chain attacks would have a significantly smaller blast radius if people start using their own forges. GitHub as a social network is no longer a good idea
You can't. They can execute arbitrary code. They can download another bash file via Curl and execute that.
Presumably you'd check the code of the action before you include it (and then don't use an action with non-pinned versions). This way you know the action won't execute arbitrary code for this version and won't get any other code because of version pinning.
The docker action you linked is ironic in this regard since every other version in the code seems to be pinned except the one you linked to.
This recommendation is currently broken. Even when you pin the full commit SHA for an action, that action may still pull in transitive dependencies (other actions) that aren't pinned.
I think that GitHub should set up Actions so that whenever you run a Github Actions step, it checks to see if either you have pinned it to a SHA or if the repository has immutable tags configured. If not, put a giant warning at the top of every pipeline run so that people are aware of the issue.
This allowed the threat actor to perform authenticated operations, including force-updating tags
Hey look, infrastructure underpinning the security of thousands of products, being compromised in a way a simple setting could have prevented (Do not allow overriding tags is an old GH setting). Yet another reason we need a Software Building Code. I wonder how many more of these reasons we'll find in 2026.
This is a good wake-up call (or reminder) that many “supply chain security” products are no more secure or responsibly engineered than the stacks they’re intended to protect. This is a characteristic of security software in general, but the rise of these kinds of “run us everywhere” tools/products invite new and exciting ways for an attacker to compromise large numbers of users in a single campaign.
This has always been my big "WTH?" whenever I see people using github actions. "You're literally taking someone else's script and ruining it against your codebase"
Why do people still use others untrusted Actions, especially without hashes? Just have an LLM write whatever script you need to do it yourself using the necessary tools.
Granted, if the underlying CLI tool itself is compromised, then avoiding the associated Action won't help you.
Recent and related:
Trivy ecosystem supply chain temporarily compromised - https://news.ycombinator.com/item?id=47450142 - March 2026 (35 comments)
temporarily might be a bit of an euphemism here
temporally
So the first incident was on March 19th and the second incident is March 22nd —- evidently the attackers maintained persistence through maybe two separate credential rotation efforts.
As far as I understood it, their entire repo got pwnd in February, and this now is the third successful attack by the same actor.
Friendly reminder that just because someone is building security software it doesn't mean they are competent and won't cause more harm than good.
Every month the security team wants me to give full code or cloud access to some new scanner they want to trial. They love the fancy dashboards and lengthy reports but if I allowed just 10% of what they wanted we would be pwned on the regular...
Most of corporate security nowadays involves "endpoint security solutions" installed on all devices, servers and VMs, piping everything into an AI-powered dashboard so we can move fast and break everything.
I audited Trivy's GitHub Actions a while back and found some worrying things, the most worrying bit was in the setup-trivy Action where it was doing a clone of main of the trivy repo and executing a shell script in there. There was no ref pinning until somebody raised a PR a few months ago. So a security company gave themselves arbitrary code execution in everyone's CI workflows.
Aqua were breached earlier this month, failed to contain it, got breached again last week, failed to contain it again, and now the attackers have breached their Docker Hub account. Shit happens but they're clearly not capable of handling this and should be enlisting outside help.
It seems they did end up contracting with Sygnia
The ref pinning part is almost worse than no pinning. You can pin the action itself to a commit SHA, sure. But half the actions out there clone other repos, curl binaries, or run install scripts internally. Basically none of that is covered by your pin. You're trusting that the action author didn't stick a `curl | bash` somewhere in their own infra.
Audited our CI a few months back and found two actions doing exactly that. Pinned to SHA on our end, completely unpinned fetches happening inside.
In this case I'm talking about what the Action did internally. The git clone inside was not pinned, but is now.
My hypothesis is that generally, there's no quality floor at which security departments are "allowed" to say "actually, none of the options on the market in this category are good enough; we're not going to use any of them". The norm is to reflexively accept extreme invasiveness and always say yes to adding more software to the pile. When these norms run deeply enough in a department, it's effectively institutionally incapable of avoiding shitty security software.
Fwiw w/r/t Trivy in particular,I don't think Trivy is bad software and I use it at work. We're unaffected by this breach because we use Nix to provide our code scanning tools and we write our own Actions workflows. Our Trivy version is pinned by Nix and periodically updated manually, so we've skipped these bad releases.
From having worked at and consulted with security software producing companies as well as security software consuming ones, I would say the security companies are worse than average at security.
And their security teams more cynical.
Sometimes they deliberately hire lower aptitude candidates to run internal security to prevent them from getting distracted by the product.
In other cases they are getting high on their own supply, more or less.
Jack Welch style management seems to take a deeper toll in this sector.
It doesn't help that a lot of security software is pretty niche. It's unreasonable to expect most candidates to know it or have experience.
In one case I was one of exactly two people out of 500 that had used the product as a paying customer. Neither of us was in management.
After a year or two the CISO drifted over and asked me to show him how to use the product, but he was more interested in soundbytes than actually using the system.
It became a powerpoint exercise and I collected my attaboy.
/s But I thought npm was the issue, and all of this couldn't happen anywhere else?!
What if we just rebuild everything from scratch with AI? No more supply chain attacks!
Just use OpenClaw. Oh wait, I think Microslop already did...
Don't underestimate the prowess of Microslop to fuck up. I'm just glad I saw all of this coming and abandoned this hellscape long ago.
No one has ever said that supply chain attacks are limited to npm.
fatiguing
Wasn't this discovered already last week, on Friday, that the threat actor had replaced the legit images with malware images? And republished 75 out of 76 tags?
No, the actor reappeared. This article is not fully updated. On March 22nd, the actor compromised their DockerHub account and published new Docker images.
I always run such tools inside sandboxes to limit the blast radius.
I don't think it would help here, they were stealing credentials
Whenever possible, credentials shouldn't be inside the sandbox either. Credential proxying, or transparent credential injection, for example with Sandcat: https://github.com/VirtusLab/sandcat
So, stealing credentials in the current directory and in all other directories are the same thing?
The sandbox will need internet access (to update data) and you will need to send code to test into it; so compromise already equals leaking all your code, without even breaking the sandboxing
Compromising all code in one directory is bad. Compromising all my data in all other directories, including mounted cloud drives, is worse.
I restrict most dev tools to access only the current directory.
You only need internet access to grab the image, I don't think trivy requires internet access itself. All of my image scanning tools run in isolation.
It needs internet access for upgrading the check bundle and for full Java library resolution (pom.xml). See e.g. https://github.com/aquasecurity/trivy/discussions/9698
Nice, thanks! Yeah, so exfil is definitely still a thing to watch out for, even if you run in an unprivileged env.
You're supposed to scan for vulnerabilities, not become one!
When you stare into the vuln, the vuln stares back at you.
a security scanner that cant secure its own credentials. at this point the irony writes itself
GG
How the heck are credential compromises still a thing with 2FA and refresh tokens???
How bugs are still possible now when we all write everything in Rust?
Since there is no requirement for anyone to use them... people don't use them. If people aren't forced to do the right thing, they do the lazy thing.
Usually through service accounts. Those are single factor.
second breach in a month from the same initial credential compromise. the first rotation didn't fully revoke access. the attacker walked right back in. no persistence needed.
My initial thought is that if this isn't a new compromise, Trivy must not have rotated the old credentials. They claim, however,
… does anyone know what exactly they're talking about, here? To my knowledge, GH does not divulge new tokens after they're issued, but it depends on the exact auth type we're talking about, and GH has an absurd number of different types of tokens/keys one can use.
OpenClaw creator made some related claims, that as soon as he created a GitHub organization with a new name, somehow it was stolen from him, and he had to ask Github people to do it for him atomically.
It is a bit different. What happened to openclaw:
He created a new org “openclaw” to reserve the name. Then he wanted to swap it with “moltbot” org.
So he opened two browser windows, one with “moltbot” repo settings another with “openclaw” repo settings.
Then he renamed “openclaw” to whatever, and quickly tried to rename “moltbot” to now available “openclaw”.
But in a second when “openclaw” was available, a bot snatched the repo.
Well, not my best 2 weeks at work, now I have to fill out a dozen forms and sit trough a shitload of meeting, just because they got pwned (twice, or once, but really badly :D )
"GitHub's own security guidance recommends pinning actions to full commit SHAs as the only truly immutable way to consume an action"
Why doesn't GitHub just enforce immutable versioning for actions? If you don't want immutable releases, you don't get to publish an Action. They could decide to enforce this and mitigate this class of issue.
I always wish these arguments came with a requirement to include a response to "well, what about the other side of the coin?", otherwise, you've now forced me to ask: well?
The two sides of the coin: Security wants pinned versions, like you have, so that compromises aren't pulled in. Security does not want¹ pinned versions, so that security updates are pulled in.
The trick, of course, is some solution that allows the latter without the former, that doesn't just destroy dev productivity. And remember, …there is no evil bit.
(… I need to name this Law. "The Paradox of Pinning"?)
(¹it might not be so explicitly state, but a desire to have constant updated-ness w/ security patches amounts to an argument against pinning.)
Their question isn't about pinned versions, it's about immutable versions. The question is why it is possible to change what commit "v5" refers to, not "why would you want to write v5".
You already don't get updates pulled in with the system unless they swap the version out from under you, which is not a normal way to deploy.
One of the only useful things my previous employer did was disallow moving tags via hg hooks.
It amounts to an argument against pinning in a (IMO) weird world view where the package maintainer is responsible for the security of users' systems. That feels wrong. The user should be responsible for the security of their system, and for setting their own update policy. I don't want a volunteer making decisions about when I get updates on my machine, and I'm pretty security minded. Sure, make the update available, but I'll decide when to actually install it.
In a more broad sense I think computing needs to move away from these centralised models where 'random person in Nebraska'[0] is silently doing a bunch of work for everyone, even with good intentions. Decisions should be deferred to the user as much as possible.
[0]: https://xkcd.com/2347/
When you want to update, you update the hashes too. This isn’t an issue in any other packaging ecosystem, where locking (including hashing) is a baseline expectation. The main issue is developer ergonomics, which comes back to GitHub Actions providing very poor package management primitives out of the box.
(This is the key distinction between updating and passively being updated because you have mutable pointers to package state. The latter gets confused for the former, but you almost always want the former.)
Honestly what I really want is the latter (mutable references), but pointing to aliases that I own and update manually (the former).
Yeah, that’s essentially what a lockfile would provide. I think GitHub Actions should really have an (official) one.
So, fork the action repository and pull from upstream at your own pace?
So JFrog
This isn't a bad distinction that you've made, I just think even lockfiles (what you're suggesting, essentially) still fall prey to the same paradox I'm suggesting.
Yes, lockfiles prevent "inadvertent" upgrades, in the sense that you get the "pinned" version in the lockfile. So if we go with the lockfile, we're now on the "pinned" side of the paradoxical coin. Yes, we no longer get auto-pwned by supply chain, but security's problem is "why are we not keeping up to date with patches?" now, since the lockfile effectively prevents them.
And then you see tooling get developed, like what Github has in the form of Dependabot, which will automatically update that lockfile. Now we're just back to the other side of the paradoxical coin, just with more steps.
(This isn't to say we shouldn't do lockfiles. Lockfiles bring a lot of other benefits, and I am generally in favor of them. But I don't think they solve this problem.)
I don’t think this is a paradox, it’s just a process. You use lockfiles to establish consistent resolutions, and then you use dependency management tooling to update those lockfiles according to various constraints/policies like compatibility, release age, known vulnerabilities, etc.
(Another framing is that you might want floating constraints for compatibility reasons, but when actually running software you basically never want dependencies changing implicitly beneath you, even if they fix things. Fixes should always be legible, whether they’re security relevant or not.)
Version tags should obviously be immutable, and if you want to be automatically updated you can select 1.0.*, if you don't you just pick the version tag.
Auto upgrade to version deemed OK by security team. Basically you need to get updates that patch exploits then wait and be more patient for feature upgrades.
So, in the context of me questioning "yes, but exactly how is this supposed to work", you're essentially punting the question into a black box that won't betray us.
In the real world, though, we don't have a magic little black box: we have to actually implement that.
The only answer I have seen from real world security teams is variations of "why wouldn't we be keeping up with updates?", and that's an unpinned dep.
I assume this is because it is modeled after git tags, and at this point it would be a major change to move away from this. But it should probably get started at some point.
GitHub has an opt-in option to enforce immutable tags as part of immutable releases.
https://docs.github.com/en/code-security/concepts/supply-cha...
Nice, yeah I think they should start to migrate to that behavior by default.
what if you pin it to a version that is compromised for years before finding out?
Allowing it to be updated can also fix security problems.
It’s basically all the same arguments as static vs dynamic linking.
Plus, I believe I saw that the one action was getting the latest version of trivy anyway.
Because the true name of the feature is VisualSourceSafe actions. It's all over the code of the runner if you take a second to look, and the runner, like the rest of the feature, is of typical early 2000s Microsoft quality, which is to say, none at all.
GitHub Actions, the feature that was years in the making, and launched in August 2018. Which Microsoft then acquired 2 months later.
Yep, once you start looking into the inner workings, you realize it's just a rebranded Azure Pipelines with a ton of technical debt attached.
Even then, that's only immutable for the workflow config. Many workflows then go on to pull in mutable inputs downstream (eg: default to "latest" version).
A better question perhaps is why we’ve allowed ourselves to be so vulnerable by a single provider (GitHub). Supply chain attacks would have a significantly smaller blast radius if people start using their own forges. GitHub as a social network is no longer a good idea
You can pin a GitHub Action to a SHA, but the GitHub Action can be a Docker one pointing to a mutable Docker image label.
Example:
https://github.com/github-community-projects/issue-metrics/b...
You can't. They can execute arbitrary code. They can download another bash file via Curl and execute that.
Presumably you'd check the code of the action before you include it (and then don't use an action with non-pinned versions). This way you know the action won't execute arbitrary code for this version and won't get any other code because of version pinning.
The docker action you linked is ironic in this regard since every other version in the code seems to be pinned except the one you linked to.
This recommendation is currently broken. Even when you pin the full commit SHA for an action, that action may still pull in transitive dependencies (other actions) that aren't pinned.
TL;DR Github Actions is the NPM of CI systems.
I think that GitHub should set up Actions so that whenever you run a Github Actions step, it checks to see if either you have pinned it to a SHA or if the repository has immutable tags configured. If not, put a giant warning at the top of every pipeline run so that people are aware of the issue.
People have been warning about giant security holes in GitHub Actions dependency but MS did nothing.
They warned you!
Hey look, infrastructure underpinning the security of thousands of products, being compromised in a way a simple setting could have prevented (Do not allow overriding tags is an old GH setting). Yet another reason we need a Software Building Code. I wonder how many more of these reasons we'll find in 2026.
This is a good wake-up call (or reminder) that many “supply chain security” products are no more secure or responsibly engineered than the stacks they’re intended to protect. This is a characteristic of security software in general, but the rise of these kinds of “run us everywhere” tools/products invite new and exciting ways for an attacker to compromise large numbers of users in a single campaign.
So by wanting to improve the security of my application, I ended up lowering the security of my application? Nice.
This post is from March 20 and update on 22! Not today!!!
Please don’t scare people like this!
This has always been my big "WTH?" whenever I see people using github actions. "You're literally taking someone else's script and ruining it against your codebase"
Why do people still use others untrusted Actions, especially without hashes? Just have an LLM write whatever script you need to do it yourself using the necessary tools.
Granted, if the underlying CLI tool itself is compromised, then avoiding the associated Action won't help you.