Both versions were published using the compromised npm credentials of a lead axios maintainer, bypassing the project's normal GitHub Actions CI/CD pipeline.
Doesn’t npm mandate 2FA as of some time last year? How was that bypassed?
Other languages have package managers (perl) and there are package managers in existence that are not so vulnerable to this issue. IMO, it stems from one place: Transitive dependencies and general opaqueness of the issue.
In package managers like pacman, apt, apk,... it's easier to catch such issue. They do have postinstall scripts, but it's part of the submission to the repo, not part of the project. Whatever comes from the project is hashed, and that hash is also visible as part of the submission. That makes it a bit difficult to sneak something. You don't push a change, they pull yours.
Log4Shell was hardly a supply-chain attack - just a latent bug in a widely-used library. That can happen anywhere.
Maven to this day represents my ideal of package distribution. Immutable versions save so much trouble and I really don't understand why, in the age of left-pad, other people looked at that and said, "nah, I'm good with this."
Completely agree. NPM has the only registry where massive supply chain attacks happen several times a year. Mainly the fault lies with NPM itself, but much of it is just a terrible opsec culture in the community.
Most package.jsons I see have semver operators on every dependency, so patches spread incredibly quickly. Package namespacing is not enforced, so there is no way of knowing who the maintainer is without looking it up on the registry first; for this reason many of the most popular packages are basically side projects maintained by a single developer*. Post-install scripts are enabled by default unless you use pnpm or bun.
When you combine all these factors, you get the absolute disaster of an ecosystem that NPM is.
*Not really the case for Axios as they are at least somewhat organized and financed via sponsors.
The semantics are irrelevant. The effect is what's important: Hijacking widely used software to exploit systems. The OC is somehow under the illusion that avoiding JS altogether is a silver bullet for avoiding this.
The semantics are very relevant, since you presented it as a supply-chain attack. If you call a library vulnerability a supply-chain attack, then your argument has lost coherence.
The OC is somehow under the illusion...
Avoiding package managers with shitty policies is the silver bullet for this attack vector. I get that it can be useful in the moment to retract published artifacts, or update them in-place, or run some code after your artifact is downloaded, but all of these are false economies in our hostile environment.
The issues have everything to do with npm as a platform and nothing with JS as a language. You can use JS without npm. Saying you'll escape supply chain attacks by not using JS is like saying you'll be saved from an car crash with a parachute.
JS as a language is part of the problem because the standard library is so minimal that people need to use a lot more 3rd party libraries than they would in most popular languages.
It’s only a losing strategy if you assume everyone universally adopts the slow strategy, and no research teams spot it in the interim. For things with large splash radius, that’s unrealistic, so defenders have an information advantage.
Makes actual security patches tougher to roll out though - you need to be vigilant to bypass the slowdown when you’re actually fixing a critical flaw. But nobody said this would be easy!
How much do you want to bet me that the credential was stolen during the previous LiteLLM incident? At what point are we going to have to stop using these package managers because it's not secure? I've got to admit, it's got me nervous to use Python or Node.js these days, but it's really a universal problem.
it’s got me nervous to use Python or Node.js these days
My feelings precisely. Min package age (supported in uv and all JS package managers) is nice but I still feel extremely hesitant to upgrade my deps or start a new project at the moment.
I don’t think this is going to stabilize any time soon, so figuring out how to handle potentially compromised deps is something we will all need to think about.
I would be avoiding npm itself on principle in the JS ecosystem. Use a package manager that has a history of actually caring about these issues in a timely manner.
It almost doesn't matter, because you can get pwned by a transitive dependency. If someone doesn't have the same scruples as you have, you're still at risk.
PNPM makes you approve postinstall scripts instead of running them by default, which helps a lot. Whenever I see a prompt to run a postinstall script, unless I know the package normally has one & what it does, I go look it up before approving it.
(Of course I could still get bitten if one of the packages I trust has its postinstall script replaced.)
I suppose you would have to commit your node_modules, or otherwise cache your setup so that all prerequesite modules are built and ready to install without running post-install scripts?
I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
Also from the report:
Neither malicious version contains a single line of malicious code inside axios itself. Instead, both inject a fake dependency, plain-crypto-js@4.2.1, a package that is never imported anywhere in the axios source, whose only purpose is to run a postinstall script that deploys a cross-platform remote access trojan (RAT)
Good news for pnpm/bun users who have to manually approve postinstall scripts.
nearly every other project uses it for some reason instead of fetch (I never understood why).
Fetch wasn't added to Node.js as a core package until version 18, and wasn't considered stable until version 21. Axios has been around much longer and was made part of popular frameworks and tutorials, which helps continue to propagate it's usage.
There are pretty much two usage patterns that come up all the time:
1- automatically add bearer tokens to requests rather than manually specifying them every single time
2- automatically dispatch some event or function when a 401 response is returned to clear the stale user session and return them to a login page.
There's no reason to repeat this logic in every single place you make an API call.
Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
Finally, there's some nice mocking utilities for axios for unit testing different responses and error codes.
You're either going to copy/paste code everywhere, or you will write your own helper functions and never touch fetch directly. Axios... just works. No need to reinvent anything, and there's a ton of other handy features the GP mentioned as well you may or may not find yourself needing.
Don't be silly, this is the JS ecosystem. Why use your brain for a minute and come up with a 50 byte helper function, if you can instead import a library with 3912726 dependencies and let the compiler spend 90 seconds on every build to tree shake 3912723 out again and give you a highly optimized bundle that's only 3 megabytes small?
IMO interceptors are bad. they hide what might get transformed with the API call at the place it is being used.
Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
This is not true unless you are not interfacing with your own backends. even then why not just make a helper that unwraps as json by default but can be passed an arg to parse as something else
That fetch requires so many users to rewrite the same code - that was already handled well by every existing node HTTP client- says something about the standards process.
It could also be trivially written for XMLHttpRequest or any node client if needed. Would be nice if they had always been the same, but oh well - having a server and client version isn't that bad.
Because it is so few lines it is much more sensible to have everyone duplicate that little snippet manually than import a library and write interceptors for that...
(Not only because the integration with the library would likely be more lines of code, but also because a library is a significantly liability on several levels that must be justified by significant, not minor, recurring savings.)
One more use case for Axios is it automatically follows redirects, forwarding headers, and more importantly, omiting or rewriting the headers that shouldn't be forwarded for security reasons.
fetch automatically follows redirects, fetch will forward your headers, omitting or rewriting headers is how security breaks… now a scraper got through because it’s masquerading as Chrome.
The fetch API is designed for browsers. It's not designed for servers. Fetch may work for a particular use case on the server, it may not. Servers have needs over and above what a browser allows the client to do.
Now I'm curious, because we have a big server side code base using fetch(). What are you using that doesn't work with fetch? Especially since axios nowadays has a fetch adapter.
Good news for pnpm/bun users who have to manually approve postinstall scripts.
Would they not have approved it for earlier versions? But also wouldn't the chance of addition automatic approval be high (for such a widely used project)?
The prompt would be to approve the new malicious package (plain-crypto-js)'s scripts, too, which could tip users off that something was fishy. If they were used to approving one for axios and the attackers had just overwrote axios's own instead of making a new package, it would probably catch people out.
Can't speak for other devs but I like to read postinstall scripts or at least put them through an LLM if they're too hard to grok.
It's also a little context dependent, for example if I was using Axios and I see a prompt to run the plain-crypto-js postinstall script, alarm bells would instantly ring, which would at least make me look up the changelog to see why this is happening.
In most cases I don't even let them run unless something breaks/doesn't work as expected.
Assuming axios didn't have a postinstall script before, it wouldn't have been approved for a previous version. If you ignore it, you ignore it, but postinstall scripts are relatively rare in npm deps, so it would seem a bit out of place when the warning pops up.
I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
You can remember this answer for every time you ask same question again:
"Coz whatever else/builtin was before was annoying enough for common use cases"
Because axios existed before the builtin fetch, and so there's a lot of stackoverflow answers explaining how to use fetch, and the llm models are trained on that, so they will write axios requests instead of fetch
I spent two years trying to get it out of a project that began long after Axios had become redundant but it's very hard to go back and challenge decisions like this because every business priority is aligned against this kind of work.
I expect libraries built on top of fetch will be the next to be compromised, because why would you use fetch without an arbitrary layer of syntactic sugar...
Has anyone tested general purpose malware detection on supply chains ? Like clamscan . I tried to test the LiteLLM hack but the affected packages had been pulled. Windows Defender AV has an inference based detector that may work when signatures have not yet been published
I second this question. I usually scan our containers with snyk and guarddog, and have wondered about guarddog in particular because it adds so much build time.
absolutely massive help. i love HN community and thanks. If you do see an archive of axios or other compromised artifacts, please send those my way for continued testing. i'm going to test automation and see if this actually has utility.
the primitive clamscan experiment worked! it detected Txt.Trojan.TeamPCP-10059839-1 from the .tar.gz archive. I'll continue testing to see if it's viable
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default.
Here's how to set global configs to set min release age to 7 days:
(Side note, it's wild that npm, bun, and pnpm have all decided to use different time units for this configuration.)
If you're developing with LLM agents, you should also update your AGENTS.md/CLAUDE.md file with some guidance on how to handle failures stemming from this config as they will cause the agent to unproductively spin its wheels.
Their analysis was triggered by open source projects upgrading en-masse and revealing a new anomalous endpoint, so, it does require some pioneers to take the arrows. They didn't spot the problem entirely via static analysis, although with hindsight they could have done (missing GitHub attestation).
A security company could set up a honeypot machine that installs new releases of everything automatically and have a separate machine scan its network traffic for suspicious outbound connections.
The problem is what counts as suspicious. StepSecurity are quite clear in their post that they decide what counts as anomalous by comparing lots of open source runs against prior data, so they can't figure it out on their own.
Can you elaborate? Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
Attackers going "low and slow" when they know they're being monitored is just standard practice.
Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
I'm sure they will, but attackers will adapt. And I'm really unconvinced that these delays are really going to help in the real world. Imagine you rely on `popular-dependency` and it gets compromised. You have a cooldown, but I, the attacker, issue "CVE-1234" for `popular-dependency`. If you're at a company you now likely have a compliance obligation to patch that CVE within a strict timeline. I can very, very easily pressure you into this sort of thing.
I'm just unconvinced by the whole idea. It's fine, more time is nice, but it's not a good solution imo.
There are many options. Here's a post just briefly listing a few of the ones that would be handled by package managers and registries, but there are also many things that would be best done in CI pipelines as well.
I suspect most packages will keep a mix of people at 7 days and those with no limit. That being said, adding jitter by default would be good to these features.
This became evident, what, perhaps a few years ago? Probably since childhood for some users here but just wondering what the holdup is. Lots of bad press could be avoided, or at least a little.
About the use of different units: next time you choose a property name in a config file, include the unit in the name. So not “timeout” but “timeoutMinutes”.
At that point, you're making all your configuration fields strings and adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either; either you write a bunch of parsing code (not terribly difficult but not something I wanna do when I can just not), or you use some time library to parse a duration string, in which case the programming language and time library you happen to use suddenly becomes part of your config file specification and you have to exactly re-implement your old time handling library's duration parser if you ever want to switch to a new one or re-implement the tool in another language.
I don't think there are great solutions here. Arguably, units should be supported by the config file format, but existing config file formats don't do that.
adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either
I'd argue that it is ideal, in the sense that it's the sweet spot for a general config file format to limit itself to simple, widely reusable building blocks. Supporting more advanced types can get in the way of this.
Programs need their own validation and/or parsing anyway, since correctness depends on program-specific semantics and usually only a subset of the values of a more simply expressed type is valid. That same logic applies across inputs: config may come from files, CLI args, legacy formats, or databases, often in different shapes. A single normalization and validation path simplifies this.
General formats must also work across many languages with different type systems. More complex types introduce more possible representations and therefore trade-offs. Even if a file parser implements them correctly (and consistently with other such parsers), it must choose an internal form that may not match what a program needs, forcing extra, less standard transformation and adding complexity on both sides for little gain.
Because acceptable values are defined by the program, not the file, a general format cannot fully specify them and shouldn’t try. Its role is to be a medium and provide simple, human-usable (for textual formats), widely supported types, avoid forcing unnecessary choices, and get out of the way.
All in all, I think it can be more appropriate for a program to pick a parsing library for a more complex type, than to add one consistently to all parsers of a given file format.
Another parsing step is the common case. Few parameters represent untyped strings where all characters and values are valid. For numbers as well, you often have a limited admissible range that you have to validate for. In the present case, you wouldn’t allow negative numbers, and maybe wouldn’t allow fractional numbers. Checking for a valid number isn’t inherently different from checking for a regex match. A number plus unit suffix is a straightforward regex.
Yes!! This goes for any time you declare a time interval variable. The number of times I've seen code changes with a comment like "Turns out the delay arg to function foo is in milliseconds, not seconds".
I actually think it is not too bad a design, because seconds are the SI base unit for time. Putting something like "x days" requires additional parsing steps and therefore complexity in the implementation. Either knowing or calculating how many seconds there are in a day can be expected of anyone touching a project or configuration at this level of detail.
Seconds are also unambiguous. Depending on your chosen definition, "X days" may or may not be influenced by leap seconds and DST changes.
I doubt anyone cares about an hour more or less in this context. But if you want multiple implementations to agree talking about seconds on a monotonic timer is a lot simpler
I came here to argue the opposite. Expressing it in seconds takes away questions about time zones and DST.
I think you're incorrect to say that second are also ambiguous. Maybe what you mean is that days are more practical, but that seems very much a personal preference.
I understand the [flawed] reasoning behind "x seconds from now is going to be roughly now() + x on this particular system", but how does defining the cooldown from an external timestamp save you from dealing with DST and other time shenanigans? In the end you are comparing two timestamps and that comparison is erroneous without considering time shenanigans
Could you explain what you mean re: ambiguity? I understand why “calendar units” like months are ambiguous, but minutes, hours, days, and weeks all have fixed durations (which is why APIs like Python’s `timedelta` allows them).
In the UK last Sunday was 23 hours long because we switched to BST, and occasionally leap seconds will result in a minute being something other 60 seconds.
The minute between December 31, 2016 23:59 and January 1st 2017 is 61 seconds, not 60 seconds. The hour that contains that minute is 3601 seconds, the day that contains that hour is 43201 seconds, etc. If you assume a fixed duration and simply multiply by 43200, your math will be wrong compared to the rest of the world.
Daylight savings time makes a day take 23 hours or 25 hours. That makes a week take 7254000 seconds or 7261200 seconds. Etc.
that kind of complexity is always worth it. Every single time. It's user time that you're saving and it also makes config clearer for readers and cuts out on "too many/little zeroes on accident" errors
It's just library for handling time that 98% of the time your app will be using for something else.
Probably went with the simplest implementation, if starting from the current “seconds since epoch” value. Let the user do any calculations needed to translate three days into that measurement.
It also efficiently annoys the most people at once: those what want hours will complain if they set it to days, thought that want days will complain if hours are used. By using minutes or seconds you can wind up both segments while not offend those who rightly don't care because they can cope with a little arithmetic :)
Though doing what sleep(1) does would be my preference: default to seconds but allow m/h/d to be added to change that.
Hence the way I would do it (and have for other purposes), as stated in my final sentence. Have the human state the intent and convert to your own internally preferred units as needed.
No no no, see now we just say "computer! do tedious math!", and it will do some slightly different math for us and compliment us on having asked it to do so.
The one true unit of time is hexadecimal encoded nanoseconds since the unix epoch. (I'm only half joking because I actually have authored code that used that before.)
Workdays! Think about it, if you set the delay in regular days/seconds the updated dependency can get pulled in on a weekend with only someone maybe on-call.
(Hope your timezones and tzdata correctly identifies Easter bank holiday as non-workdays)
Don't forget about regional holidays, which might follow arbitrary borders that don't match any of the official subdivisions of the country. Or may even depend on the chosen faith of the worker
In JavaScript something entirely new would be invented, to solve a problem that has long been solved and is documented in 20+ year old books on common design patterns. So we can all copy-paste `{ or: [{ days: 42, months: 2, hours: "DEFAULT", minutes: "IGNORE", seconds: null, timezone: "defer-by-ip" }, { timestamp: 17749453211*1000, unit: "ms"}]` without any clue as to what we are defining.
In Java, a 6000LoC+ ecosystem of classes, abstractions, dependency-injectables and probably a new DSL would be invented so we can all say "over 4 Malaysian workdays"
But you know that Java solution will continue working even after we no longer use the Gregorian Calendar, the collapse and annexation of Malaysia to some foreign power, and then us finally switching to a 4-day work week; so it'd be worth it.
... and since it was architectured to allow runtime injection-patching of events before they hit the enterprise-service-bus, everyone using this library must first set fourteen ENV vars in their profile, and provide a /etc/java/springtime/enterprise-workday-handling/parse-event-mismatch.jar.patch. Which should fix the bug for you.
You can find the patch files for your OSs by registering at Oracle with a J3EE8.4-PatchLibID (note, the older J3EE16-PatchLib-ids aren't compatible), attainable from your regional Oracle account-manager.
JavaScript Temporal. Not sure knowing what a "workday" is in each timezone is in it's scope but it's the much needed and improved JS, date API (granted with limited support to date)
In before someone thinks it's a joke, the most commonly used logging library in Java had LDAP support in format scripts enabled by default" (which resulted, of course in CVE)
…now imagine a list of instruments, some of which have durations specified in days/weeks/months (problems already with the latter) and some in workdays, and the user just told your app to display it sorted by duration.
Nah, working hours and make global assumptions of 0900-1230/1330-1730, M-F, and have an overly convoluted way to specify what working ours actually are in the relevant location(s).
They made a movie to make money. I doubt anyone holding the purse strings cared one iota if that bit were corrected or not. It’s not really a retcon either because they didn’t change anything.
That had more or less been the explanation in the books for decades, and even in George Lucas' notes from 1977:
It's a very simple ship, very economical ship, although the modifications he made to it are rather extensive – mostly to the navigation system to get through hyperspace in the shortest possible distance (parsecs).
For Star Wars, they retconned it to mean he found the shortest possible route through dangerous space, so even for Han Solo's quote, it's still distance.
To me it sounds safer to have different big infra providers with different delays, otherwise you still hit everyone at the same time when something does inevitably go undetected.
And the chances of staying undetected are higher if nobody is installing until the delay time ellapses.
It's the same as not scheduling all cronjobs to midnight.
The config for uv won't work. uv only supports a full timestamp for this config, and no rolling window day option afaik. Am I crazy or is this llm slop?
Define a dependency cooldown by specifying a duration instead of an absolute value. Either a "friendly" duration (e.g., 24 hours, 1 week, 30 days) or an ISO 8601 duration (e.g., PT24H, P7D, P30D) can be used.
I think it should work at the user config level too:
If project-, user-, and system-level configuration files are found, the settings will be merged, with project-level configuration taking precedence over the user-level configuration, and user-level configuration taking precedence over the system-level configuration.
This is what tripped me up. I added that config and then got this error:
error: Failed to parse: `.config/uv/uv.toml`
Caused by: TOML parse error at line 1, column 17
|
1 | exclude-newer = "7 days"
| ^^^^^^^^
failed to parse year in date "7 days": failed to parse "7 da" as year (a four digit integer): invalid digit, expected 0-9 but got
I was on version 0.7.20, so I removed that line, ran "uv self update" and upgraded to 0.11.2 and then re-added the config and it works fine now.
Yeah, that error message isn’t ideal on older versions, but unfortunately there’s no way to really address that. But I’m glad it’s working for you on newer versions.
min release age to 7 days about patch releases exposes you to the other side of the coin, you have an open 7 days window on zero-day exploits that might be fixed in a security release
Urgent fix, patch released, invisible to dev team cause they put in a 7 day wait. Now our app is vulnerable for up to 7 days longer than needed (assuming daily deploys. If less often, pad accordingly). Not a great excuse as to why the company shipped an "updated" version of the app with a standing CVE in it. "Sorry we were blinded to the critical fix because set an arbitrary local setting to ignore updates until they are 7 days old". I wouldn't fire people over that, but we'd definitely be doing some internal training.
At least with pnpm, you can specify minimumReleaseAgeExclude, temporarily until the time passes. I imagine the other package managers have similar options.
The packages that are actually compromised are yanked, but I assume you're talking about a scenario more like log4shell. In that case, you can just disable the config to install the update, then re-enable in 7 days. Given that compromised packages are uploaded all the time and zero-day vulnerabilities are comparatively less common, I'd say it's the right call.
Is there a way to do that per repo for these tools ? We all know how user sided configuration works for users (they usually clean it whenever it goes against what they want to do instead of wondering why it blocks their changes :))
Fairly sure every single one has a repo level config that you can add these settings to. Others have pointed out the pnpm and npm, and I bunfig can also be repo level.
If everyone or a majority of people sets these options, then I think issues will simply be discovered later. So if other people run into them first, better for us, because then the issues have a chance of being fixed once our acceptable package/version age is reached.
Because everyone got updates immediately. If the default was 7 days, almost no one would get updates immediately but after 7 days, and now someone only finds about after 7 days. Unless there is a poor soul checking packages as they are published that can alert the registry before 7 days pass, though I imagine very few do that and hence a dedicated attacker could influence them to not look too hard.
If I remember correctly, in all the recent cases it was picked up by automated scanning tools in a few hours, not because someone updated the dependency, checked the code and found the issue.
So it looks like even if no one actually updates, the vast majority of the cases will be caught by automated tools. You just need to give them a bit of time.
While this explicitly calls out "postinstall", I'm pretty sure it affects other such lifecycle scripts like preinstall in dependencies.
The --ignore-scripts option will ignore lifecycle scripts in the project itself, not just dependencies. And it will ignore scripts that you have previously allowed (using the "allowBuilds" feature).
And when you actually need a super hot fix for a 0-day, you will need to revert this and keep it that way for some time to then go back to minimum age.
While this works, we stillneed a permanent solution which requires a sort of vetting process, rather than blindly letting everything through.
I think my vetting would settle for a repo diff against the previous version, confirming the only difference was the security fix (though that doesn't cover all the bases).
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
The solution is not moar toolz. That's the problem—this crazy mindset that the problems endemic to bad tooling have a solution in the form of complementing them with another layer, rather than fewer.
Git and every sane SCM already allow you to manage your source tree without jumping through a bunch of hoops to go along with wacky overlay version control systems like the one that the npmjs.com crew designed, centering around package.json as a way to do an end-run around Git. You don't need to install and deploy anything containing never-before-seen updates just because the NodeJS influencer–developers say that lockfiles are the Right Way to do things. (It's not.)
Opting in to being vulnerable to supply chain attacks is a choice.
But also have a regular review of your dependencies to update them when necessary, because as bad as compromised packages may be things do have vulnerabilities occasionally, and upgrading things that are a long way out-of-date can be quite hard.
I recommend everyone to use bwrap if you're on linux and alias all package managers / anything that has post build logic with it.
I have bwrap configured to override: npm, pip, cargo, mvn, gradle, everything you can think of and I only give it the access it needs, strip anything that is useless to it anyway, deny dbus, sockets, everything. SSH is forwarded via socket (ssh-add).
This limits the blast radius to your CWD and package manager caches and often won't even work since the malware usually expects some things to be available which are not in a permissionless sandbox.
You can think of it as running a docker container, but without the requirement of having to have an image. It is the same thing flatpak is based on.
As for server deployments, container hardening is your friend. Most supply chain attacks target build scripts so as long as you treat your CI/CD as an untrusted environment you should be good - there's quite a few resources on this so won't go into detail.
AFAIK maven doesn’t support post install logic like npm does. You have to explicitly optin with build plugins. It doesn’t let any arbitrary dependency run code on your machine.
Maybe I misunderstood this point. But the ssh socket also gives access to your private keys, so I see no security gain in that point. Better to have a password protected key.
It's so your private key is not stolen, but you're right passphrase protected keys win anyway. I use hardware keys so this isn't a problem for me to begin with.
This only works for post-install script attacks. When the package is compromised, just running require somewhere in your code will be enough, and that runs with node/java/python and no bwrap.
node is also sandboxed within bwrap I have sandbox -p node if I have to give node access to other folders, I also have sandbox -m to define custom mountpoints if necessary and UNSAFE=1 as a last resort which just runs unsandboxed.
Check also https://github.com/wrr/drop which is a higher-level tool than bwrap. It allows you to make such isolated sandboxes with minimal configuration.
This looks nice but I wouldn't trust a very fresh tool to do security correctly.
As a higher-level alternative to bwrap, I sometimes use `flatpak run --filesystem=$PWD --command=bash org.freedesktop.Platform`. This is kind of an abuse of flatpaks but works just fine to make a sandbox. And unlike bwrap, it has sane defaults (no extra permissions, not even network, though it does allow xdg-desktop-portal).
Shame it's not a bit more mature, it does look like more the sort of thing I want. I use firejail a bit, but it's a bit awkward really.
To be honest - and I can't really believe I'm saying it - what I really want is something more like Android permissions. (Except more granular file permissions, which Android doesn't do at all well.) Like: start with nothing, app is requesting x access, allow it this time; oh alright fine always allow it. Central place to manage it later. Etc.
I like the idea of bubblewrap, but my pain point is that it is work to set it up correctly with bind mounts and forwarding necessary environment variables to make the program actually work usefully. Could you share your pip bwrap configuration? It sounds useful.
This may not be popular, but is there a place for required human actions or just timed actions to slow down things like this? For instance, maybe a GH action to deploy requires a final human click and to change that to cli has a 3 day cooling period with mandatory security emails sent out. Similarly, you switch to read only for 6 hrs after an email change. There are holes in these ideas but the basic concept is to treat security more like physical security, your goal isn't always to 100% block but instead to slow an attacker for xxx minutes to give the rest of the team time to figure out what is going on.
Hi, security here. We've tried, but the amount of people you need for this vs the amount of people you have trying to review and click the big button always means that this step will be a bottleneck. Thus this step will be eliminated.
A much better approach would be to pin the versions used and do intentional updates some time after release, say a sprint after.
Yeah, I am looking at that on the use end. It sounds like on the python side this type of thing will be more standard (uv now and soon pip supported with version date requirements). I think time is a big missing element in many security in depth decisions. It can be time until you adopt like use no package newer than xx days or time it takes to deploy etc etc. Unfortunately the ecosystem is getting really diverse and that means ever more sophisticated attacks so we may need to do things that are annoying just to survive.
Why not just release escrow? If I try to push a new release version another developer or developers have to agree to that release. In larger projects you would expect the release to be coordinated or scheduled anyways. Effectively we're just moving "version pinning" or "version delay" one layer up the release chain.
xz has dozens of contributors and two active maintainers. It was the actual example I was thinking of. The code was submitted by a third party and not a result of a developer machine compromise.
left pad wasn't a security incident. It was a capitalism incident.
Pinning, escrowing, and trailing all help, but I'm not sure "this step will be eliminated" is inevitable.
Package manager ecosystems are highly centralized. npm.org could require MFA (or rate limit, or email verification, or whatever) and most packagers would gripe but go along with this. A minority would look for npm competitors that didn't have this requirement, and another minority would hack/automate MFA and remove the added security, but the majority of folks would benefit from a centralized requirement of this sort.
Yup. As someone who's been on both the eng and security side, you cannot improve security by blocking the product bus. You're just going to get run over. Your job is to find ways of managing risk that work with the realities of software development.
And before anyone gets upset about that, every engineering discipline has these kind of risk tradeoffs. You can't build a bridge that'll last 5,000 years and costs half of our GDP, even though that's "safer". You build a bridge that balances usage, the environment, and good stewardship of taxpayer money.
Min release age sucks, but we’ve been here before. Email attachments used to just run wild too, then everyone added quarantine delays and file blocking and other frictions... and it eventually kinda/sorta worked. This does feel worse, though, with fewer chokepoints and execution as a natural part of the expectation.
Edit: bottom line is installs are gonna get SOOO much more complicated. You can already see the solution surface... Cooling periods, maintainer profiling, sandbox detonation, lockfile diffing, weird publish path checks. All adds up to one giant PITA for fast easy dev.
Min release age might just postpone vulnerability to be applied few days later in non trivial cases like this. More I think about it, Odin lang approach of no package manager makes senses. But, for that approach won't work for Javascript as it needs npm package even for trivial things. Even vendoring approach like golang won't work with Javascript with the amount of churn and dependencies.
Genuinely how are you supposed to make sure that none of the software you have on your system pulls this in?
It’s things like this that make me want to swap to Qubes permanently, simply as to not have my password manager in the same context as compiling software ever.
This sounds like satire but isn't - I just make sure the nodejs/npm packages don't exist on my system. I've yet to find a crucial piece of software that requires it. As much as I love that cute utility that turns maps into ascii art, it's not exactly sqlite in terms of usefulness.
I don't deny that node/npm is useful for building servers, devtools for JS development itself, etc. but as an end user I haven't encountered anything useful which requires having it on my machine.
We run everything NPM related inside Apple containers, and are looking to do the same with Python and Rust soon. Bwrap on Linux does the same.
I like to think of it like working with dangerous chemicals in the lab. Back in the days, people were sloppy and eventually got cancer. Then dangers were recognized and PPE was developed and became a requirement.
We are now at the stage in software development where we are beginning to recognizing the hazards and developing + mandating use of proper PPE.
A couple of years ago, pip started refusing to install packages outside of a virtualenv. I'm guessing/hoping package managers will start to have an opt-in flag you can set in a system-wide config file, such that they refuse to run outside of a sandbox.
The problem is that package managers are a distraction. You have to sandbox everything or else it doesn't work. These attacks use post-install hooks for convenience but nothing would have stopped them patching axios itself and just waiting for devs to run the app on their local workstation. So you end up needing to develop in a fully sandboxed environment.
While it's not perfect, pinning specific versions and managing all updates directly has been a solid solution for my team. Things can of course still slip through, but we're never vulnerable to these just because there was a new package release and we opted into it by default.
Updating packages takes longer, but we try to keep packages to a minimum so it ends up not being that big deal.
Supply chain attacks are so scary that I think most companies are going to use agents to hard fork their own versions of a lot of these core libraries instead. It wasn’t practical before. It’s definitely much more doable today.
I was thinking about this as a bull case for human developers. Seems if you're worried enough to do this you're not going to have LLMs write the new code.
The flip side of that is now you're running old software and CVEs get published all the time. Threat actors actively scan the internet looking for software that's vulnerable to new CVEs.
Large companies already maintain a clone of their packages. Very large ones actually bundle their own build system (Google Bazil, AWS Brazil). If you want to update a package, you have to fetch the sources and update the internal repository. It slows down the opportunities for a supply chain attack down to a crawl.
Once you fork it, you are then on the hook for forking every future update and security patch. You can automate a lot of the testing, but its still adding an extra failure point that you are now responsible for.
And if you pin it, then when you inevitably get a CVE for an old version, the upgrade path is harder and more time consuming. And that's when the security teams come knocking to pass their audit.
There are so many scanners these days these things get caught pretty quick. I think we need either npm or someone else to have a registry that only lets through packages that pass these scanners. Can even do the virustotal thing of aggregating reports by multiple scanners. NPM publishes attestation for trusted build environments. Google has oss-rebuild.
All it takes is an `npm config set` to switch registries anyways. The hard part is having a central party that is able to convince all the various security companies to collaborate rather than having dozens of different registries each from each company.
Rather than just a hard-coded delay, I think having policies on what checks must pass first makes sense with overrides for when CVEs show up.
The ones you hear about are caught quickly, I’m more worried about the non obvious ones. So far none of these have been as simple as changing a true to a false and bypassing all auth for all products or something, and would that be caught by an automated scanner?
There are definitely levels to this. Yes I think it can be caught by automated scanners in theory. Either commit by commit scanning and reproducible builds or fuzzing and getting the behavioral differences between versions
Not to beat a dead horse but I see this again and again with dependencies. Each time I get more worried that the same will happen with rust. I understand the fat std library approach won’t work but I really still want a good solution where I can trust packages to be safe and high quality.
Linux distros and BSD ports did that since the 90's. When Linux distros had barely a PM or just tarballs, Infomagic sold 4 CD full of libre software. When I had no internet at home, back in the day I bought 3 DVD's of Debian Sarge for 20 euros, about $20. A bargain, it was the price of a hard-cover best seller book.
GB's of libre software, graphical install, 2.6 kernel, KDE3 desktop, very light on my Athlon 2000 with 256MB of RAM. It was incredible compared to what you got with Windows XP and 120 Euro per seat. Nonfree software and almost empty.
And, well, if for instance I could get read only, ~16TB durable USB drive with tons of Guix packages offline (in a two yearly basis with stable releases) for $200 I would buy them in the spot.
You would say that $200 for a distro it's expensive, but for what it provides, if you are only interested in libre gaming and tools, they amount you save can be huge. I've seen people spend $400 in Steam games because of the Holyday sales...
- copy the dependencies' tests into your own tests
- copy the code in to your codebase as a library using the same review process you would for code from your own team
- treat updates to the library in the same way you would for updates to your own code
Apparently, this extra work will now not be a problem, because we have AI making us 10x more efficient. To be honest, even without AI, we should've been doing this from the start, even if I understand why we haven't. The excuses are starting to wear thin though.
I don't know where you've worked but a hostile and intelligent actor or internal red team would succeed under each of those cases at every job I've worked at.
Defending against a targeted attack is difficult, yes. But these recent campaigns were all directed at everyone. Auditing and inspecting your dependencies does absolutely help thwart that because there will always be people who don't.
They succeeded in poisoning the whole supply chain and making everyone distrust package management to a degree never seen before, and people who aren't reviewing their dependencies are already getting hit. You seem to suggest that we all accept that.
That attitude might be the reason why the places you've worked would be under threat. The places I've worked would also be under threat, because several of my colleagues had that attitude, and this is why red teaming works.
As you should. Also, the constant complaint from devs on these very boards is that quality and security are relegated behind new features that are often described as useless but pushed by management.
Why wouldn't the "fat std" thing work? Yes it's hard to design properly, both in scope and actual design (especially for an unstandardized language still moving fast), but throwing the towel and punting the problem to the "free market" of uncurated public repos is even worse.
It's what we call in France "la fête du slip".
PS: that's one reason I try to use git submodules in my Common Lisp projects instead of QuickLisp, because I really see the size of my deptree this way.
Not necessarily, but let's agree that some design faults would happen: you still get the option to use the solid, boring and slightly rusty std instead of another 100 dependencies from the supply chain supermarket.
At work, we're happy with Python's included batteries when we need to make scripts instead of large programs.
NPM should have a curation mechanism, via staff review or crowdsourcing, where versions of popular packages are promoted to a stable set, like linux distros do. I would only use curated versions if they had such a thing.
This is why Node.js is completely unsuitable as backend.
Until recently, there wasn’t even a standard Promise-based HTTP client. Why should we need to download a library just to make a simple HTTP request? It’s because Node.js’s standard library is too limited, leading to an explosive growth in third-party libraries. As a result, it’s vulnerable to security attacks, and maintaining it in an enterprise environment becomes a major challenge.
Let’s use .NET or Go. Why use JavaScript outside of the browser when there are excellent backend environments out there?
All these supply chain attacks make me nervous about the apps I use. It would be valuable info if an app used such dependencies, but on the other hand, programmers would cut their sales if they gave you this info.
This is just going to get worse and worse as agentic coding gets better. I think having a big dependency tree may be a thing of the past in the coming years. Seems like eventually new malware will be coming out so fast it will basically be impossible to stop.
Axios has a long history, and is included in a lot of code, also in indirect dependencies. Just check its npm page: it has 174025 dependents as of this moment, including a lot of new packages (I see openclaw and mcp related packages in the list).
And with LLMs generating more and more code, the risk of copying old setups increases.
We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
The issue is not dependencies themselves, it's transitive ones. Nobody installs left-pad or is-even-number directly, and "libraries" like these are the vast majority of the attack surface. If you get rid of transitive dependencies, you get rid of the need of a package manager, as installing a package becomes unzipping a few files into a vendor/ folder.
There's so many C libraries like this. Off the top of my head, SQLite, FreeType, OpenSSL, libcurl, libpng/jpeg, stb everything, zlib, lua, SDL, GLFW... I do game development so I'm most familiar with the ones commonly used in game engines, but I'm sure other fields have similarly high quality C libraries.
They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust, and C#/Java/JS/Python libraries are never used outside their respective language (aside form Java ones in other JVM langs).
If you're developing for the web your attack surface is quite a bit bigger. Your proposed solution of copying a few files might work but how do you keep track of updates? You might be vulnerable to a published exploit fixed a few months ago. A package manager might tell you a new version is available. I don't know how that would work in your scenario.
I'd really like to see package managers organized around rings where a very small core of incredibly important stuff is kept in ring 0, ring 1 gets a slightly wider amount of stuff and can only depend on ring 0 dependencies and then ring 2+ is the crapware libraries that infect most ecosystems.
But maybe that's not the right fit either. The world where package managers are just open to whatever needs to die. It's no longer a safe model.
In practice, "ring 0" is whatever gets merged into your language's standard library. Node and python both have pretty expansive standard libraries at this point, stepping outside of those is a choice
The OS distro model is actually the right one here. Upstream authors hate it, but having a layer that's responsible for picking versions out of the ecosystem and compiling an internally consistent grouping of known mutually-compatible versions that you can subscribe to means that a lot of the random churn just falls away. Once you've got that layer, you only need to be aware of security problems in the specific versions you care about, you can specifically patch only them, and you've got a distribution channel for the fixes where it's far more feasible to say "just auto-apply anything that comes via this route".
That model effectively becomes your ring 1. Ring 0 is the stdlib and the package manager itself, and - because you would always need to be able to step outside the distribution for either freshness or "that's not been picked up by the distro yet" reasons - the ecosystem package repositories are the wild west ring 2.
In the language ecosystems I'm only aware of Quicklisp/Ultralisp and Haskell's Stackage that work like this. Everything else is effectively a rolling distro that hasn't realised that's what it is yet.
Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
What is a problem is library quality. Which is downstream of nobody getting paid for it, combined with an optimistic but unrealistic "all packages are equal" philosophy.
High quality C libraries
OpenSSL
OpenSSL is one of the ones where there's a ground up rewrite happening because the code quality is so terrible while being security critical.
On the other end, javascript is uniquely bad because of the deployment model and difficulty of adding things to the standard library, so everything is littered with polyfills.
Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
Absolute nonsense. What does automated world even mean? Even if one could infer reasonably, it's no justification. Appealing to "the real world" in lieu of any further consideration is exactly the kind of mindlessness that has led to the present state of affairs.
Automation of dependency versions was never something we needed it was always a convenience, and even that's a stretch given that dependency hell is abundant in all of these systems, and now we have supply chain attacks. While everyone is welcome to do as they please, I'm going to stick to vendoring my dependencies, statically compiling, and not blindly trusting code I haven't seen before.
Relax, while mentioning the real world without any criticism for the soundness of the solution is absolute nonsense, some would say idiotic, thinking only in the absolute best solution given your narrow world view is not any better.
While I agree that my view is narrow, the "best solution" in question is what we used to do, and it was fine. There are still many places that manually manage dependencies. Fundamentally automatic software versioning is an under-developed area in need of attention, and technologies like semantic versioning which are ubiquitous are closer to suggestions, and not true indicators of breaking changes. My personal view is that fully automatic dependency version management is an ongoing experiment and should be treated as such.
The culture within the npm/js community has mainly been one of using the package manager rather than "re-inventing the wheel", as such the blast radius of a compromised package is much greater
It’s not just NPM, though. Every Rails project and every Rust project I’ve seen ended up with massive numbers of dependencies vs what an equivalent project in Go or C# would have needed.
CPAN too, just try Hailo under Perl to test an old-fashioned chatbot based on Markov chains where very small LLM's and Hailo converge if used with the advanced training options for it. Yes, it will pull tons of dependencies, (less with cpanminus if run with 'cpanm -n Hailo'), but contrary to NPM, Pip and the like CPAN's repos are highly curated and before PHP and ubiquitoous Python Perl was used everywhere, from a sysadmin language (better than Bash/Sh for sure) to CGI, IRC bots and whatnot. How many issues did we have? Zero or near zero.
It's more to do with the standard library being so barren of common application needs, and looking for a solution that the community has gotten behind. Axios has been a common dependency in many codebases, because it is a solid solution that many have already used. Every developer could try building all the libraries that they would reach for themselves, but then each company has now taken on the task of ensuring their own (much larger) codebase is free from security issues, on top of taking care of their own issues and bugs.
Rust libraries are infrequently used outside of Rust because if you have the option, you'd just use Rust, not the ancient featureless language intrinsically responsible for 70% of all security issues. C libraries are infrequently used in Rust outside of system libc, for the same reason; I go and toggle the reqwest switch to use rustls every time, because OpenSSL is horrendous. This is also why you say 'rarely' instead of 'never', when a few years ago it was 'never'; a few years from now you'll say 'uncommonly', and so on. The reason C libraries are used is because you don't feel like reimplementing it yourself, and they are there; but that doesn't apply more to C libraries than Rust libraries, and the vast majority of crates.io wouldn't be usefully represented in C anyway, or would take longer to bind to than to rewrite. (No, nobody uses libcurl.) Finally, this only happens in NPM, and the Rust libraries you pull in are all high-quality. So this sounds like a bunch of handwaving about nonsense.
Rust is terrible for pulling in hundreds of dependencies though. Add tokio as a dependency and you'll get well over 100 packages added to your project.
pin-project-lite is the only base dependency, which itself has no dependencies. If you enable the "full" feature, ie all optional doodads turned on (which you likely don't need), it's 17: bytes, cfg-if, errno, libc, mio, parking_lot+parking_lot_core+lock_api, pin-project-lite, proc_macro2+quote+syn+unicode-ident, scopeguard, signal-hook-registry, smallvec, and socket2. You let me know which ones you think are bloat that it should reimplement or bind to a C library about, and without the blatant fabrication this time.
Even side stepping that tokio no longer pulls multiple packages, it used to be split into multiple packages, in the same way that KDE in Rust would be hundreds of packages.
Rust projects tend to take their project and split it into many smaller packages, for ease of development, faster compiles through parallelization, ensuring proper splitting of concerns, and allowing code reuse by others. But the packages are equivalent to a single big package. The people that write it are the same. They get developed in tandem and published at the same time. You can take a look at the del tree for ripgrep, and the split of different parts of that app allows me to reuse the regex engine without dealing with APIs that only make sense in the context of a CLI app or pulling in code I won't ever use (which might be hiding an exploit too).
Counting 100 100 line long crates all by the same authors as inherently more dangerous than 1 10000 line long crate makes no sense to me.
It's worth noting that Rust packages (crates) are all single compilation units, and every compilation unit is a package. It's the equivalent of complaining that OpenSSL pulls in hundreds of `.c` files.
I don't think this community of professionals is going to come around to a solution which requires marginally more effort.
If no one checks their dependencies, the solution is to centralize this responsibility at the package repository. Something like left-pad should simply not be admitted to npm. Enforce a set of stricter rules which only allow non-trivial packages maintained by someone who is clearly accountable.
Another change one could make is develop bigger standard libraries with all the utilities which are useful.
For example in Rust there are a few de facto standard packages one needs very often, which then also force you to pull in a bunch of transitive dependencies.
Those could also be part of the standard library.
This all amounts to increasing the minimal scope of useful functionality a package has to have to be admitted and increasing accountability of the people maintaining them. This obviously comes with more effort on the maintainers part, but hey maybe we could even pay them for their labor.
We have libraries like SQLite, which is a single .c file that you drag into your project
You are just swapping a package manager with security by obscurity by copy pasting code into your project. It is arguably a much worse way of handling supply chain security, as now there is no way to audit your dependencies.
If you get rid of transitive dependencies, you get rid of the need of a package manager
This argument makes no sense. Obviously reducing the amount of transitive dependencies is almost always a good thing, but it doesn't change the fundamental benefits of a package manager.
There's so many C libraries like this
The language with the most fundamental and dangerous ways of handling memory, the language that is constantly in the news for numerous security problems even in massively popular libraries such as OpenSSL? Yes, definitely copy-paste that code in, surely nothing can go wrong.
They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust
This is a WILD assumption, doing C-style bindings is actually quite common. YOu will of course then also be exposing a memory unsafe interface, as that is what you get with C.
What exactly is your argument here? It feels like what you are trying to say is that we should just stop doing JS and instead all make C programs that copy paste massive libraries because that is somhow 'high quality'.
This seems like a massively uninformed, one-sided and frankly ridiculous take.
You are just swapping a package manager with security by obscurity by copy pasting code into your project
You should try writing code, and not relying on libraries for everything, it may change how you look at programming and actually ground your opinions in reality. I'm staring at company's vendor/ folder. It has ~15 libraries, all but one of which operate on trusted input (game assets).
fundamental benefits of a package manager.
I literally told you why they don't matter if you write code in a sane way.
doing C-style bindings is actually quite common
I know bindings for Rust libraries exist. Read the literal words you quoted. "Rust libraries are very rarely used outside of Rust". Got some counterexamples?
Gamedev is its own weird thing, and isn't a model you want to generalize to other industries. It has to optimize for things a lot of software does not, and that skews development.
Vendoring libraries is almost always a terrible idea because it immediately starts to bitrot and become a footgun.
Sometimes it's necessary, but it's not desirable, and you almost always just want to pin your dependencies instead.
I think you can do copy paste in most languages. But it will be a pain to update when there are improvements / security fixes.
You got a project with 1-2 depencies? Sure. But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs) good luck.
But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs
So don’t?
With manual deps management, everyone soon gravitates to a core set of deps. And libraries developer tends to reduce their deps needs, That’s why you see most C libraries deals with file formats, protocols, and broad concerns. Smaller algorithms can be shared with gists and blog articles.
It's not there isn't good libraries everywhere. It just the practice around NPM that people are appalled with. It's all about lowering the barrier for developers, even in spite of security and quality.
We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
I'm not sure why you believe this is more secure than a package manager. At least with a package manager there is an opportunity for vetting. It's also trivial that it did not increase your executable's size. If your executable depends on it, it increases its effective size.
Package managers are older than some users here. From CPAN/CTAN to ports under BSD's.
Some pm's are badly maintained (Pip/NPM), while others are curated enough.
Again, if you have GNU/Linux installed, install Guix, read the Info manual on 'guix import' and just create a shell/container with 'guix shell --container' (and a manifest package created from guix import) and use any crap you need for NPM in a reproducible and isolated way. You $HOME will be safe, for sure.
They're not a failed experiment. No one has ever "experimented" by making a safe package manager for their new language. And it is not that insane to do so. Very basic things will get you very far:
1. Packages should carry a manifest that declares what they do at build time, just like Chrome extensions do. This manifest would then be used to configure its build environment.
2. Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
3. Every package registry should produce a detailed audit log that contains a "who, what, when". Every build/ command should be producing audit logs that can be collected by endpoint agents too.
4. Every package registry should support TUF.
5. Typosquatting defenses should be standard.
etc etc etc. Some of this is hard, some of this is not hard. All of this is possible. No one has done it, so it's way too early to say "package managers can't be made safe" when no one has tried.
Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
How is this enforced when it's pushed via a pipeline?
Your account is separate from your publishing. That is, in order to go to my account to change configuration values, 2FA must be required.
Publishing should be handled via something like Trusted Publishing, which would leverage short lived tokens and can integrate with cryptographic logs for publish information (ie: "Published from the main branch of this repo at this time").
I don't understand commercial aspect of large OSS like package managers but i was wondering for years why this was missing from npm.
I think typosquatting was handled by npm last year but only after some popular miss typed packages started stealing developer creds.
The people building package managers are unaware of these problems going into it and it becomes extremely disruptive to start adding these things later on since your entire ecosystem is built on the assumption that they can do these things.
It's also shockingly controversial to suggest typosquatting suggestions. I made this suggestion ages ago for cargo, demonstrated that basic distance checks would have impacted <1% of crates over all time, and people still didn't want it.
Sorry, I should have clarified that - you're correct. `cosign` is an example of a tool that makes this quite straightforward and proves that this sort of system can work today.
Then you've got ecosystems like Clojure where many projects are just considered done and used by many. You can pin these (and be warned if a new version still comes out, say for an actual security fix). There are Clojure projects so stable, without any know exploit (we're certainly not talking about daily npm exploits here), that haven't been updated in years because they are... Done. Simply done. Perfection.
SQLite had 6 CVE's last year. FreeType had an RCE bug published last year. libcurl's last CVE reported was 2 weeks ago. Libpng had 4 vulnerabilities published this year....
Just sanity checking - if I only ever install axios in a container that has no secrets mounted in to its env, is there any real way I can get pwned by this kind of thing?
Essential steps to minimise your exposure to NPM supply chain attacks:
— Run Yarn in zero-installs mode (or equivalent for your package manager). Every new or changed dependency gets checked in.
— Disable post-install scripts. If you don’t, at least make sure your package manager prompts for scripts during install, in which case you stop and look at what it’s going to run.
— If third-party code runs in development, including post-install scripts, try your best to make sure it happens in a VM/container.
— Vet every package you add. Popularity is a plus, recent commit time is a minus: if you have this but not that, keep your eyes peeled. Skim through the code on NPM (they will probably never stop labelling it as “beta”), commit history and changelog.
— Vet its dependency tree. Dependencies is a vector for attack on you and your users, and any new developer in the tree is another person you’re trusting to not be malicious and to take all of the above measures, too.
Exactly. Yarn uses a yarn.lock file with the sha256 hashes of each npm package it downloads from the repo (they are .tgz files). If the hash won't match, install fails. No need to commit the dependencies into your git.
Number 1 would only be a win for zero-installs if it happened that registry was up when you made the security hotfix, since you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy. Seems like a highly unlikely scenario to me. Also, cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Most npm CVEs are stuff like DDoS vulnerabilities, and you should have mitigations for those in place for at the infra-level anyway (e.g. request timeouts, rate limits, etc), or you are pretty much guaranteed to be cooked sooner or later anyway. The really dangerous stuff like arbitrary command execution from a library that takes end user input is much much more rare. The most recent big one I remember is React2shell.
Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
I can see how zero-installs are useful under some specific constraints where you want to minimize dependencies to external services, e.g. when your CI runs under strict firewalls. But for most, nah, not worth it.
you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy.
Which dependency? It sounds like you are assuming some specific scenario, whereas the fix can take many forms. In immediate term, the quickest step could be to simply disable some feature. A later step may be vendoring in a safe implementation.
The registry doesn’t need to be actually down for you, either; the necessary condition is that your CI infrastructure can’t reach it.
cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Not sure what you mean by “npm CVEs”. The registry? The CLI tool?
As I wrote, if you are running compromised software in production, you want to fix it ASAP. In first moments you may not even know whether bad things will happen or not, just that you are shipping malicious code to your users. Even if you are lucky enough to determine with 100% confidence (putting your job on the line) that the compromise is inconsequential, you don’t want to keep shipping that code for another hour because your install step fails due to a random CI infra hiccup making registry inaccessible (as happened in my experience at least half dozen times in years prior, though luckily not in a circumstance where someone attempted to push an urgent security fix). Now imagine it’s not a random hiccup but part of a coordinated targeted attack, and somehow it becomes something anticipated.
Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Those rare conditions exist. Also, you are making it sound as if the registry is infallible, and no humans and/or LLMs there accept untrusted input from their environment.
The key aspect of modern package managers, when used correctly, is that even when the registry is compromised you are fine as long as integrity check crypto holds up and you hold on to your pre-compromise dependency tree. The latter is not a technical problem but a human problem, because conditions can be engineered in which something may slip past your eyes. If this slip-up can be avoided at little to no cost—in fact, with benefits, since zero-installs shortens CI times, and therefore time-to-fix, due to dramatically shorter or fully eliminated install step—it should be a complete no-brainer.
Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
As I wrote, I suspect it’d complicate such attacks or make them easier to spot, not make them impossible.
Are you saying it replaces my package manager, or that I should add another tool to my stack, vet yet another vulnerable dependency for critical use, to do something my package manager already does just as well?
You ~never want to vendor libraries.
I just explained why you should, and you are yet to provide a counter-argument.
It’s a subjective question, but in one of the zero-installs projects I definitely remember that when I added a couple of particular GUI libraries there suddenly a very, very long list of new files to track, since those maintainers to keep things decoupled. I wouldn’t stop using that library at that point (there were deadlines), but I would definitely try to find something lighter or more batteries-included next time.
There can be a tiny project with just one dependency that happens to have an overgrown, massive graph of further transitive dependencies (a very unpleasant scenario which I would recommend to avoid).
With zero installs turned on, such a codebase could indeed qualify as “big repo”, which I think would reflect well its true nature.
Without zero installs it could be tiny but with a long long lockfile that nobody really checks when committing changes.
Personally I store vendored dependencies in a submodule
I don’t like the added mental overhead of submodules, and so prefer to avoid them when possible, which I guess is a subjective preference.
Since this is, coneptually speaking, the matter of package management more so than it is the matter of version control in genetal, I prefer to rely on package manager layer to handle this. I can see how your approach could make sense, but honestly I would be more anxious about forgetting something when keeping vendored dependencies up-to-date in that scenario.
Your approach could be better in a sense that you can spot-check not just the list of changed packages, but also the actual code (since you presumably vendor them as is, while Yarn checks in compressed .tgz files). Not sure whether that justifies the added friction.
In light of these nonstop supply chain attacks:
Tonight I created /supply-chain-audit
-- A simple claude code skill that fetches info on the latest major package vulnerability, then scans your entire ~/ and gives you a report on all your projects.
The NPM ecosystem is a joke. I don't even want anything to do with it, because my stack is fully Elixir. But, just because of this one dependency that is used in some interfaces within my codebase, I need to go back to all my apps and fix it. Sigh.
JavaScript, its entire ecosystem is just a pack of cards, I swear. What a fucking joke.
The attacker went through the hassle to compromise a very widely used package, but use a non standard port (8000) on their C2...
If you plan to do something like that, use 443 at least, many corporate network do not filter this one ;)
I wonder if this has any connection with the recent string of attacks including the FBI director getting hacked. The attack surface is large, executed extremely cleanly - almost as if done by a high profile state sponsored actor, just like in Hollywood movies.
I'm working on a multi signature solution that helps to detect unauthorized releases in the case of an account hijack. It is open source, self hostable, accountless and I am looking for feedback!
I have a few projects which rely on npm (and react) and every few months I have to revisit them to do an update and make sure they still build, and I am basically done with npm and the entire ecosystem at this point.
Sure, its convenient to have so much code to use for basic functionality - but the technical debt of having to maintain these projects is just too damn high.
At this point I think that, if I am forced to use javascript or node for a project, I reconsider involvement in that project. Its ecosystem is just so bonkers I can't justify the effort much longer.
There has to be some kind of "code-review-as-a-service" that can be turned on here to catch these things. Its just so unproductive, every single time.
I am glad I don't need to touch JS or web dev at all.
Now, I tend to use Python, Rust and Julia. With Python I am constantly using few same packages like numpy and matplotlib. With Rust and Julia, I try as much as possible to not use any packages at all, because it always scares me when something that should be pretty simple downloads half of the Internet to my PC.
Julia is even worse than Rust in that regard - for even rudimentary stuff like static arrays or properly namespaced enums people download 3rd party packages.
Isn't Rust just as susceptible to this issue? For example, how do you deal with Rust's lack of support for HTTP in the standard library? Importing hyper pulls in a couple dozen transitive libraries which exposes you to the exact same kind of threats that compromised axios.
Given how HTTP is now what TCP was during the 90s and almost all modern networked applications needing to communicate in it one way or another, most rust projects come with an inherent security risk.
These days, I score the usability of programming languages by how complete their standard library is. By that measure, Rust and Javascript get an automatic F.
It is, therefore I have stated I avoid any dependencies while writing Rust, unless they are self-contained. And I said I am glad I don't do web, so I don't have need for HTTP implementations.
I believe compartmentalized operating systems like Qubes are the future for defending against these kinds of attacks.
Storing your sensitive data on a single bare-metal OS that constantly downloads and runs packages from unknown maintainers is like handing your house key out to a million people and hoping none of them misuse it.
The frustrating thing here is that axios versions display on npmjs with verified provenance. But they don’t use trusted publishing: https://github.com/axios/axios/issues/7055 - meaning the publish token can be stolen.
I wrongly thought that the verified provenance UI showed a package has a trusted publishing pipeline, but seems it’s orthogonal.
NPM really needs to move away from these secrets that can be stolen.
I am now migrating all my unencrypted secrets on my machines to encrypted ones. If a tool supports scripted credential providers (e.g. aws-cli or Ansible), I use that feature. Otherwise, I wrap the executable with a script that runs gpg --decrypt and injects an environment variable.
That way, I can at least limit the blast radius when (not if) I catch an infostealer.
In case it helps, we open-sourced a tool to audit dependencies for this kind of supply-chain issue. The motivation was that there is a real gap between classic “known vulnerability” scanning and packages whose behavior has simply turned suspicious or malicious. We also use AI to analyze code and dependency changes for more novel or generic malicious behavior that traditional scanners often miss.
Hopefully desktop Linux users will start to understand that malware actually does exist for Linux and that their operating system is doing nothing to protect them from getting RATed.
Yet npm isn't using them allowing this RAT to work. It is not secure by default. It requires every app to manually opt in to being secure. This opt in approach to security puts desktop Linux decades behind in regards to security. Not ahead.
Take for example iOS and Android. All apps are sandboxed by default. You can't make a program that just steals all of your credentials like you can on desktop Linux. Having security tools means nothing if they aren't being used.
They aren't because Node haven't developed a Node app for them. Desktop Linux does not offer more security than macOS. macOS has proper security around stuff like apps using the mic and camera.
Absolutely. If you ever did a npm install on a project using one of the affected axios versions, your entire system may be compromised.
The malicious versions inject a new dependency, plain-crypto-js@4.2.1, which is never imported anywhere in the axios source code. Its sole purpose is to execute a postinstall script that acts as a cross platform remote access trojan (RAT) dropper, targeting macOS, Windows, and Linux. The dropper contacts a live command and control server and delivers platform specific second stage payloads. After execution, the malware deletes itself and replaces its own package.json with a clean version to evade forensic detection.
I lost respect for Axios when they made a breaking change in a patch release. Digging into the root cause, I found the maintainer had approved an outside PR with an obvious AI slop PR description: https://github.com/axios/axios/issues/7059
Looks like the maintainer wasn't just careless when reviewing PRs.
So the root cause was again a developer's opsec. For improving things, I haven't seen many new initiatives on that side (beyond 2FA, but even that seems unenforced in these repositories, I reckon).
To have an initial smoke test, why not run a diff between version upgrades, and potentially let an llm summarise the changes? It’s a baffling practice that a lot of developers are just blindly trusting code repos to keep the security standards. Last time I installed some npm package (in a container) it loaded 521 dependencies and my heart rate jumped a bit
Is this the first time you have ever thought about the idea of supply chain attacks? This is the first thought 90% of people have and it doesn't work. Too much work to manually verify diffs and LLMs aren't good enough at this yet.
No, I think about it all the time. It’s just baffling that this kind of attack is still a thing, after a decade+ of this happening over and over again.
"Batteries included" ecosystems are the only persistent solution to the package manager problem.
If your first party tooling contains all the functionality you typically need, it's possible you can be productive with zero 3rd party dependencies. In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography. These are beacons for attackers. Everything depends on this stuff so the motivation for attacking these common surfaces is high.
I can literally count on one hand the number of 3rd party dependencies I've used in the last year. Dapper is the only regular thing I can come up with. Sometimes ScottPlot. Both of my SQL providers (MSSQL and SQLite) are first party as well. This is a major reason why they're the only sql providers I use.
Maybe I am just so traumatized from compliance and auditing in regulated software business, but this feels like a happier way to build software too. My tools tend to stay right where I left them the previous day. I don't have to worry about my hammer or screw drivers stealing all my bitcoin in the middle of the night.
This is a rather superlative and tunnel vision, "everything is a nail because I'm a hammer" approach. The truth is this is an exceedingly difficult problem nobody has adequately solved yet.
I think the AI tooling is, if not completely solving sandboxing, at least making the default much better by asking you every time they want to do something and providing files to auto-approve certain actions.
Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
Tell me about it. Using AI Chatbots (not even agents), I got a MVP of a packaging system[1] to my liking (to create packages for a proprietary ERP system) and an endpoint-API-testing tool, neither of which require a venv or similar to run.
------------------------------
[1] Okay, all it does now is create, sign, verify and unpack packages. There's a roadmap file for package distribution, which is a different problem.
at least making the default much better by asking you every time they want to do something
Really? I thought 'asking you every time they want to do something' was called 'security fatigue' and generally considered to be a bad thing. Yes you can concatenate files in the current project, Claude.
Fully agree with this! I think today .NET is probably the most batteries included platform you can get. This means that even if you use third-party libraries, these typically depend only on first-party dependencies, making it much less likely for something shady to sneak in.
C#'s LINQ (code as data, like LISP) wins over golang for any type of data access. Strongly-typed, language-native queries. Go has its own advantages though.
The other thing that keeps coming up is the github-code-is-fine-but-the-release-artifact-is-a-trojan issue. It really makes me question if "packages" should even exist in JavaScript, or if we could just be importing standard plain source code from a git repo.
I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
Why wouldn't that work well with legacy projects? In fact, the projects I was a part of that I'd call legacy nowadays, was in fact built by copy-and-pasting .js libraries into a "vendor/" directory, and that's how we shipped it as well, this was in the days before Bower (which was the npm of frontend development back in the day), vendoring JS libs was standard practice, before package managers became used in frontend development too.
Not sure why it wouldn't work, JavaScript is a very moldable language, you can make most things work one way or another :)(
Yes - the postinstall hook attack vector goes away. You can do SHA pinning since Git's content addressing means that SHA is the hash of the content. But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
This is basically what Deno's import maps tried to solve, and what they ended up with looked a lot like a package registry again.
At least npm packages have checksums and a registry that can yank things.
But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
The irony is that this is actually the current best practice to defend against supply chain attacks in the github actions layer. Pin all actions versions to a hash. There's an entire secondary set of dev tools for converting GHA version numbers to hashes
You can just git submodule in the dependencies. Super easy. Also makes it straightforward to develop patches to send upstream from within your project. Or to replace a dependency with a private fork.
In my experience, this works great for libraries internal to an organization (UI components, custom file formats, API type definitions, etc.). I don't see why it wouldn't also work for managing public dependencies.
Plus it's ecosystem-agnostic. Git submodules work just as well for JS as they do for Go, sample data/binary assets, or whatever other dependencies you need to manage.
This is where attestation/sigstore comes into play. Github has a first-party action for it and I wish more projects would use it. Regarding javascript specifically, I believe npm has builtin support for sigstore.
Honestly, you can get pretty far with just Bun and a very small number of dependencies. It’s what I love most about Bun. But, I do agree with you generally. .NET is about as good as I’ve ever seen for being batteries included. I just hate the enterprisey culture that always seems to pervade .NET shops.
I agree about the culture. If I take my eye off the dev team for too long, I'll come back and we'll be using entity framework and a 20 page document about configuring code cleanup rules in visual studio.
In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography
Unless you are Python, where the standard library includes multiple HTTP libraries and everyone installs the requests package anyways.
Few languages have good models for evolving their standard library, so you end up with lots of bad designs sticking around forever. Libraries are much easier to evolve, giving them the advantage in terms of developer UX and performance.
What type of developer chooses UX and performance over security? So reckless.
I removed the locks from all the doors, now entering/exiting is 87% faster!
After removing all the safety equipment, our vehicles have significantly improved in mileage, acceleration and top speed!
Better developer UX can directly lead to better safety. "You are holding it wrong" is a frequent source of security bugs, and better UX reduces the ways you can hold it wrong, or at least makes you more likely to hold it the right way
Friction is helpful. Putting seatbelts on takes more time than just driving, but it’s way safer for the driver. Current dev practices increase speed, not safety.
Better developer UX can directly lead to better safety.
Depends. If you had to add to a Makefile for your dependencies, you sure as hell aren't going to add 5k dependencies manually just to get a function that does $FOO; you'd write it yourself.
Now, with AI in the mix, there's fewer and fewer reasons to use so many dependencies.
What type of developer chooses UX and performance over security? So reckless.
Initially I assumed this is sarcastic, but apparently not. UX and performance is what programmers are paid to do! Making sure UX is good is one of the most important things in programmer job.
While security is a moving target, a goal, something that can never be perfect, just "good enough" (if NSA wants to hack you, they will). You make it sound like installing third party packages is basically equivalent to a security hole, while in practice the risk is low, especially if you don't overdo it.
Wild to read extreme security views like that, while at the same time there are people here that run unconstrained AI agents with --dangerous-skip-confirm flags and see nothing wrong with it.
Even more wild to read that sarcasm about "removing locks from doors for 87% speedup" is considered extreme...
And yes, we agree that running unconstrained AI agents with --dangerous-skip-confirm flags and seeing nothing wrong with it is insane. Kind of like just advertising for burglars to come open your doors for you before you get home - yeah, it's lots faster to get in (and to move about the house with all your stuff gone).
"Security" is often more about corporate CYA than improving my actual security as a user, and sometimes in opposition, and there is often blatant disregard for any UX concession at all. The most secure system is fully encrypted with all copies of the encryption key erased.
Personally I've heard Odin [1] to do a decent job with this, at least from what I've superficially learned about its stdlib and included modules as an "outsider" (not a regular user).
It appears to have things like support for e.g. image file formats built-in, and new things are somewhat liberally getting added to core if they prove practically useful, since there isn't a package manager in the traditional sense.
Here's a blog post by the language author literally named "Package Managers are Evil" [2]
(Please do correct me if this is wrong, again, I don't have the experience myself.)
Scala could be one example? When I upgraded to a newer version of the standard library (the Scala 2.13 or Scala 3 collections library), there was a tool, Scalafix [1], that could update my source code to work with the new library. Don't think it was perfect (don't remember), but helpful.
For a lot of code, I switched to generating code rather than using 3rd party libraries.
Things like PEG parsers, path finding algorithms, string sanitizers, data type conversion, etc are very conveniently generated by LLMs. It's fast, reduces dependencies, and feels safer to me.
Lol. My most recent comment before this one is here: https://news.ycombinator.com/item?id=47583593. You judge if AI threatens my identity. But hey, don't let the facts get in the way of a slick narrative.
Or find the best third party library and copy the code from a widely used version that has been out long enough to have been well tested into your source tree.
The problem is not third party libraries. It is updating third party libraries when the version you have still works fine for your needs.
Don't do this. Use a package manager that let's you specify a specific version to pin against. Vendoring side steps most automated tooling that can warn you about vulnerabilities. Vendoring is a signal that your tooling is insufficient, 99% of the time.
Vendoring means you don't have to fetch the internet for every build, that you can work offline, that you're not at the mercy of the oh-so-close-99.999 availability, that it will keep on working in 10 years, and probably other advantages.
If your tooling can pull a dependency from the internet, it could certainly check if more recent version from a vendored one is available.
Is there any package manager incapable of working offline?
I think you've identified the problem here: package management and package distribution are two different problems. Both tools have possibilities for exploits, but if they are separate tools then the surface area is smaller.
I'm thinking that the package distribution tool maintains a local system cache of packages, using keys/webrings/whatever to verify provenance, while the package management tool allows pinning, minver/maxver, etc.
This is only true if you aren’t internally mirroring those packages.
Most places I’ve worked have Artifactory or something like it sitting between you and actual PyPI/npm/etc. As long as someone has pulled that version at some point before the internet goes out, it’ll continue to work after.
And this is exactly why we see noise on HN/Reddit when a supply-chain cyberattack breaks out, but no breach is ever reported. Enterprises are protected by internal mirroring.
What are some examples of batteries-included languages that folk around here really feel productive in and/or love? What makes them so great, in your opinion?
(Leaving aside thoughts on language syntax, compile times, tooling etc - just interested in people's experiences with / thoughts on healthy stdlibs)
These are the big ones I use, specifically because of the standard libraries:
Python (decent standard library) - It's pretty much everywhere. There's so many hidden gems in that standard library (difflib, argparse, shlex, subprocess, cmd)
C#/F# (.NET)
C# feels so productive because of how much is available in .NET Core, and F# gets to tag along and get it all for free too. With C# you can compile executables down to bundle the runtime and strip it down so your executables are in the 15 MiB range. If you have dotnet installed, you can run F# as scripts.
Do you worry at all about the future of F#? I've been told it's feeling more and more like a second-class citizen on .NET, but I don't have much personal experience.
I used to, but the knowledge of .NET seems mostly transferrable to C#. It's super useful to do `dotnet fsi` and then work out the appropriate .NET calls in the F# repl.
I work in a NIS2 compliance sector, and we basically use Go and Python for everything. Go is awesome, Python isn't as such. Go didn't always come with the awesome stllib that it does today, which is likely partly why a lot of people still use things like Gin for web frameworks rather than simply using the standard library. Having worked with a lot of web frameworks, the one Go comes with is nice and easy enough to extend. Python is terrible, but on the plus side it's relatively easy to write your own libraries with Python, and use C/Zig to do so if you need it. The biggest challenges for us is that we aren't going to write a better MSSQL driver than Microsoft, so we use quite a bit of dependencies from them since we are married with Azure. These live in a little more isolation than what you might expect, so they aren't updated quite as often as many places might. Still, it's a relatively low risk factor that we can accept.
Our React projects are the contrast. They live in total and complete isolation, both in development and in production. You're not going to work on React on a computer that will be connected to any sort of internal resources. We've also had to write a novel's worth of legal bullshit explaining how we can't realistically review every line of code from React dependencies for compliance.
Anyway, I don't think JS/TS is that bad. It has a lot of issues, but then, you could always have written your own wrapper ontop of Node's fetch instead of using Axios. Which I guess is where working in the NIS2 compliance sector makes things a little bit different, because we'd always chose to write the wrapper instead of using one others made. With the few exceptions for Microsoft products that I mentioned earlier.
We used to have some C# but we moved away from it to have fewer languages and because it was a worse fit for us than Go and Python. I'm not sure .NET would really give us any advantages though. Microsoft treats most major languages as first class citizens in Azure, and since we build everything to be sort of platform agnostic, we wouldn't have the tie-ins that you could have with .NET. I'm not saying it would be fun to switch cloud, but all our services are build so that there is a decoupled "adapter" between our core logic and Azure. We use a lot of Azure functions as an example, but they run in container apps on a managed k8s, so the Azure function part is really just an ingress that could be swapped for anything else.
It's been a while since I worked with an "actual" function app in Azure. We did have a few .NET ones that weren't using containers. At the time they were pretty good, but today I'm not sure what the benefit over a managed container envrionment with container apps would be. Similarily with sqlserver. We use it because of governance and how it ties into data factory and I guess fabric, but we don't use ORM's so something like Entity Framework wouldn't really be something we'd benefit from with .NET.
I think the only thing we couldn't realistically replace and get something similar is the governance, but that's more to do with how Management Groups, Policies, Subscriptions and EntraID works than anything else.
Eventuallyt everything will probably be Python and then C/Zig for compute heavy parts. Not because Python is great, it's terrible, but it's what everyone uses. We're an energy company and with the internal AI tools we've made widely available we now have non-SWE employees writing code. It's Business Intelligence, it's Risk analysys, it's powerplant engineers, it's accountants. They're all working with AI code in their sandboxed environments and it's all Python. Since some of it actually turns out to generate great value, it's better for us (and the business) if our SWE teams can easily take over when "amateur hour" needs to meet operational compliance for the more "serious" production envrionments. I put things in "'s because I'm still not entirely sure how to express this. A lot of what gets build is great, and would have never been build without AI because we don't have the man power, but it's usually some pretty bad software. Which is fine, until it isn't.
AI was trained on Axios wrappers, so it's just going to be wrappers all the way down. Look inside any company "API Client" and it's just a branded wrapper around Axios.
I have never consciously wrapped Axios or fetch, but a cursory search suggests that there was a time when it was impossible for either to force TLS1.3. It's easy to imagine alternate implementations exist for frivolous reasons, but sometimes there are hard security or performance requirements that force you into them.
Fetch has also lacked support for features that xhr has had for over a decade now. For example upload progress. It's slowly catching up though, upload progress is the only thing I'd choose xhr for.
That is a way to approximate it, though I'd be curious to know the semantics compared to xhr - would they both show the same value at the same network lifecycle of a given byte?
That would show how quickly the data is passing into the native fetch call but doesn’t account for kind of internal buffer it might have, network latency etc
I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
I'm not saying that axios is unmaintained, I'm saying that if you want something like axios from the standard lib, fetch is the closest thing you get to official
I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
Nor is fetch a good client-side API either; you want progress indicators, on both upload and download. Fetch is a poor API all-round.
You can pass to `fetch` an `AbortSignal` like `AbortSignal.timeout(5000)` as a simple and easy guard.
If you also want to guard on size, iterating the `response.body` stream with for/await/of and adding a counter that can `abort()` a manual `AbortSignal` is relatively straightforward, though sounds complicated. You can even do that as a custom `ReadableStream` implementation so that you can wrap it back into `Response` and still use the `response.json()` shortcut. I'm surprised I'm not seeing a standard implementation of that, but it also looks straightforward from MDN documentation [1].
Browser fetch can lean on the fact that the runtime environment has hard limits per tab and the user will just close the tab if things get weird. on the server you're right
It doesn't have a need _now_. Axios is more than 10 years old now, and even before axios other libraries did the same utility of making requests easier
It doesn't matter. We pulled axios out of our codebase, but it still ends up in there as a child or peer from 40 other dependencies. Many from major vendors like datadog, slack, twilio, nx (in the gcs-cache extension), etc...
Because native fetch lack retries, error handling is verbose, search and body serialization create ton of boilerplate. I use KY http client, small lib on top of fetch with great UX and trusted maintainer.
So, youre on Microsoft then, judging by ScottPlot you write .NET desktop apps. If you use Dapper, you probably use Microsoft.Data.SqlClient, which is... distributed over NuGet and vulnerable to supply chain attack. You may not need many deps as a desktop dev. Modern day line of business apps require a lot more deps. CSVHelper, ClosedXML, AutoMapper, WebOptimizer,
NetEscapades.AspNetCore.SecurityHeaders.
Yes less deps people need the better but it doesn't fix trhe core problem. Sharing and distrib uting code is a key tenant of being able to write modern code.
I agree that dependencies are a liability, but, sadly, "batteries included" didn't work out for Python in practice (i. e. how do I even live without numpy? No, array aren't enough).
To the extend that Python is indeed "batteries included," that seems true. But just how "batteries included" is it? I'd argue that its batteries are pretty limited. Exhibit A: everybody uses the third-party requests instead of the stdlib urllib. Exhibit B: http.server isn't a production-ready webserver, so people use Flask or something beefier.
I'd contrast Python with Go, which has an amazing stdlib for the domains that Go targets. This last part is key--Go has a more focused scope than Python, and that makes it easier for its stdlib to succeed.
http.server isn't a production-ready webserver, so people use Flask [...]
Nit, but relevant nit: Flask is also not a production-grade webserver. You could say it is also missing batteries ... and those batteries are often missing batteries too. Which is why you don't deploy flask, you deploy flask on top of gunicorn on top of nginx. It's missing batteries all the way down (or at least 3 levels down).
Appreciate the nit. Had no idea that Flask wasn't production-grade. Yeesh.
I really don't miss this part of the Python world. When I started on backend stuff ~10 years ago, the morass of runtime stuff for Python webservers felt bewildering. uWSGI? FastCGI? Gunicorn? Twisted? Like you say, missing batteries all the way down, presumably due to async/GIL related pains.
Then you step into the Go world and it's just the stdlib http package.
Anyway, ranting aside, batteries included is a real thing, and it's great. Python just doesn't have it.
Frankly inventing a new language is irresponsible these days unless you build on-top of an existing ecosystem because you need to solve all these problems.
or you don't use a package manager where anyone can just publish a package (i.e. use your system package manager). There is still some risk, but it is much smaller. Like, if xz were distributed by PyPI or NPM, everyone would have been pwned, but instead it was (barely) found.
It's true that system repos doesn't include everything, but you can create your own repositories if you really need to for a few things. In practice Fedora/EPEL are basically sufficient for my needs. Right now I'm deploying something with yocto, which is a bit more limited in slection, but it's pretty easy to add my own packages and it at least has hashes so things don't get replaced without me noticing (to be fair, I don't know if the security practices of open-embedded recipes are as strong as Fedora...).
it's muddying what a package is. A package, or a distro, is the people who slave and labor over packaging, reviewing, deciding on versions to ship, having policies in place, security mailing lists, release schedules, etc.
just shipping from npm crap is essentially the equivelant of running your production code base against Arch AUR pkgbuilds.
Batteries included systems are still susceptible to supply chain attacks, they just move slower so it’s not as attractive of a target.
I think packages of a certain size need to be held to higher standards by the repositories.
Multiple users should have to approve changes. Maybe enforced scans (though with trivy’s recent compromise that wont be likely any time soon)
Basically anything besides lone developer can decide to send something out on a whim that will run on millions of machines.
While technically true, it's so much slower that it's essentially a different thing. Third party packages being attacked is a near daily occurrence. First party attacks happens on the timescale and frequency of decades.
It's like the difference in protecting your home from burglars and foreign nation soldiers. Both are technically invaders to your home, but the scope is different, and the solutions are different.
they just move slower so it’s not as attractive of a target.
Well, there’s other things. Maven doesn’t allow you to declare “version >= x.y.z” and doesn’t run arbitrary scripts upon pulling dependencies, for one thing. The Java classpath doesn’t make it possible to have multiple versions of the same library at the same time. That helps a lot too.
NPM and the way node does dependency management just isn’t great. Never has been.
This is exactly the world I'm working towards with packaging tooling with a virtual machine i.e. electron but with virtual machines instead so the isolation aspect comes by default.
"Batteries included" ecosystems are the only persistent solution
Or write your own stuff. Yes, that's right, I said it. Even HTTP. Even cryptography. Just because somebody else messed it up once doesn't mean nobody should ever do it. Professional quality software _should_ be customized. Professional developers absolutely can and should do this and get it right. When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
This post is modded down (I think because of the "roll your own crypto vibe", which I disagree with), but this is actually spot on the money for HTTP.
The surface area for HTTP is quite large, and your little API, which never needed range-requests, basic-auth, multipart form upload, etc suddenly gets owned because of a vulnerability in one of those things you not only never used, you also never knew existed!
"Surface area" is a problem, reducing it is one way to mitigate.
the "roll your own crypto vibe", which I disagree with
Again, you run into the attack surface area here. Think about the Heartbleed vulnerability. It was a vulnerability in the DTLS implementation of OpenSSL, but it affected every single user, including the 99% that weren't using DTLS.
Experienced developers can, and should, be able to elide things like side-channel attacks and the other gotchas that scare folks off of rolling their own crypto. The right solution here is better-defined, well understood acceptance criteria and test cases, not blindly trusting something you downloaded from the internet.
1. It's really really hard to verify that you have not left a vulnerability in (for a good time, try figuring out all the different "standards" needed in x509), but, more importantly,
2. You already have options for a reduced attack surface; You don't need to use OpenSSL just for TLS, you can use WolfSSL (I'm very happy with it, actually). You don't need WolfSSL just for public/private keys signing+encryption, use libsodium. You don't need libsodium just for bcrypt password hashing, there's already a single function to do that.
With crypto, you have some options to reduce your attack surface. With HTTP you have few to none; all the HTTP libs take great care to implement as much of the specification as possible.
That's actually not really crypto, though - that's writing a parser (for a container that includes a lot of crypto-related data). And again... if you import a 3rd-party x.509 parser and you only need DER but not BER, you've got unnecessary bloat yet again.
There are several issues with "Batteries Included" ecosystems (like Python, C#/.NET, and Java):
1. They are not going to include everything. This includes things like new file formats.
2. They are going to be out of date whenever a standard changes (HTML, etc.), application changes (e.g. SQLite/PostgreSQL/etc. for SQL/ORM bindings), or API changes (DirectX, Vulcan, etc.).
3. Things like data structures, graphics APIs, etc. will have performance characteristics that may be different to your use case.
4. They can't cover all nice use cases such as the different libraries and frameworks for creating games of different genres.
For example, Python's XML DOM implementation only implements a subset of XPath and doesn't support parsing HTML.
The fact that Python, Java, and .NET have large library ecosystems proves that even if you have a "Batteries Included" approach there will always be other things to add.
"Batteries included" means "ossification is guaranteed", yah. "stdlib is where code goes to die" is a fairly common phrase for a reason.
There's clearly merit to both sides, but personally I think a major underlying cause is that libraries are trusted. Obviously that doesn't match reality. We desperately need a permission system for libraries, it's far harder to sneak stuff in when doing so requires an "adds dangerous permission" change approval.
But also everyone sane avoids the built-in http client in any production setting because it has rather severe footguns and complicated (and limited) ability to control it. It can't be fixed in-place due to its API design... and there is no replacement at this point. The closest we got was adding some support for using a Context, with a rather obtuse API (which is now part of the footgunnery).
There's also a v2 of the json package because v1 is similarly full of footguns and lack of reasonable control. The list of quirks to maintain in v2's backport of v1's API in https://github.com/golang/go/issues/71497 (or a smaller overview here: https://go.dev/blog/jsonv2-exp) is quite large and generally very surprising to people. The good news here is that it actually is possible to upgrade v1 "in place" and share the code.
There's a rather large list of such things. And that's in a language that has been doing a relatively good job. In some languages you end up with Perl/Raku or Python 2/3 "it's nearly a different language and the ecosystem is split for many years" outcomes, but Go is nowhere near that.
Because this stuff is in the stdlib, it has taken several years to even discuss a concrete upgrade. For stuff that isn't, ecosystems generally shift rather quickly when a clearly-better library appears, in part because it's a (relatively) level playing field.
This looks like an ad for batteries included to me.
Libraries also don't get it right the first time so they increment minor and major versions.
Then why is it not okay for built-in standard libraries to version their functionality also? Just like Go did with JSON?
The benefits are worth it judging by how ubiquitous Go, Java and .NET are.
I'd rather leverage billions of support paid by the likes of Google, Oracle and Microsoft to build libraries for me than some random low bus factor person, prone to be hacked at anytime due to bad security practices.
Setting up a large JavaScript or Rust project is like giving 300 random people on the internet permission to execute code on my machine. Unless I audit every library update (spoiler: no one does it because it's expensive).
Third party libraries have been avoiding those json footguns (and significantly improving performance) for well over a decade before stdlib got it. Same with logging. And it's looking like it will be over two decades for an even slightly reasonable http client.
Stuff outside stdlib can, and almost always does, improve at an incomparably faster rate.
And I think the Go people seem to do a fairly good job of picking out the best and most universal ideas from these outside efforts and folding them in.
.NET's JSON and their Kestrel HTTP server beg to differ.
Their JSON even does cross-platform SIMD and their Kestrel stack was top 10/20 on techempower benchmarks for a while without the ugly hacks other frameworks/libs use to get there.
stdlib is the science of good enough and sometimes it's far above good enough.
Libraries don't get it right the first time, but there are often multiple competing libraries which allows more experimentation and finding the right abstraction faster.
Another downside of a large stdlib, is that it can be very confusing. Took my a while how unicode is supposed to work in go, as you have to track down throughout the APIs what are the right things to use. Which is even more annoying because the support is strictly binary and buried everywhere without being super explicit or discoverable.
I'm not sure I understand. Why would a standard library, a collection of what would otherwise be a bunch of independent libraries, bundled together, be more confusing than the same (or probably more) independent libraries published on their own?
For me, the v2 re-writes, as well as the "x" semi-official repo are a major strength. They tell me there is a trustworthy team working on this stuff, but obviously not everything will always be as great as you might want, but the floor is rising.
yea, I like the /x/ repos a fair bit. "first-party but unstable" is an extremely useful area to have, and many languages miss it by only having "first-party stable forever" and "third party". you need an experimentation ground to get good ideas and seek feedback, and keeping it as a completely normal library allows people/the ecosystem to choose versions the same way as any other library.
100% to libraries having permissions. If I'm using some code to say compute a hash of a byte array, it should not have access to say the filesystem nor network.
"Batteries included" means "ossification is guaranteed", yah. "stdlib is where code goes to die" is a fairly common phrase for a reason.
Except I rather have ossified batteries that solve my problem, even if not as convinient as more modern alternatives, than not having them at all on a given platform.
please! nobody uses Xpath (coz json killed XML), it RDF (semantic web never happened, and one ever 10years is not fast), schema.org (again, nobody cares), PNG: no change in the last 26 years, not fast. the HTML "living standard" :D completely optional and hence not a standard but definition.
XPath 1.0 is a pain to write queries for. XPath 2.0 adds features that make it easier to write queries. XPath 3.1 adds support for maps, arrays, and JSON.
And the default Python XPath support is severely limited, not even a full 1.0 implementation. You can't use the Python XPath support to do things like `element[contains(@attribute, 'value')]` so you need to include an external library to implement XPath.
XPath is used in processing XML (JATS and other publishing/standards XML files) and can be used to proces HTML content.
RDF and the related standards are still used in some areas. If the "Batteries Included" standard library ignores these then those standards will need an external library to support them.
Schema.org is used by Google and other search engines to describe content on the page such as breadcrumbs, publications, paywalled content, cinema screenings, etc. If you are generating websites then you need to produce schema.org metadata to improve the SEO.
Did you notice that a new PNG standard was released in 2025 (last year, with a working draft in 2022) adding support for APNG, HDR, and Exif metadata? Yes, it hasn't changed frequently, but it does change. So if you have PNG support in the standard library you need to update it to support those changes.
And if HTML support is optional then you will need an external library to support it. Hence a "Batteries Included" standard library being incomplete.
comparing to Node, .NET is batteries included: built-in Linq vs needing lodash external package, built-in Decimal vs decimal.js package, built-in model validation vs class-validator & class-transformer packages, built-in CSRF/XSRF protection vs csrf-csrf package, I can go on for a while...
That's my point. You can have a large standard library like those languages I mentioned, but that isn't going to include everything nor cover every use case, so you'll have external libraries (via PyPi for Python, NuGet for .NET, and Maven for Java/JVM).
depends, JavaScript in the Browser has many useful things available, which I miss with python, e.g., fetch, which in Python you need a separate package like requests to avoid a clunky API. Java had this issue for long time as well, since Java 11 there is the HttpClient with a convenient API.
Not at all. We simply need M-of-N auditors to sign off on major releases of things. And the package managers need to check this (the set of auditors can be changed, same as browser PKI for https) before pulling things down.
That's the system we have in our Safebox ecosystem
While it's true that the packages are first party, .NET still relies on packages to distribute code that's not directly inside the framework. You still probably transiently depend on `Microsoft.Extensions.Hosting.Abstractions ` for example - if the process for publishing this package was compromised, you'd still get owned.
This just moves the trust from one group to another. Now the standard library/language maintainers need to develop/maintain more high quality software. So either they get overworked and burn out, don't address issues, fail to update things or they recruit more people who need to be trusted. Then they are responsible for doing the validation that you should have done. Are they better equipped to do that? Maybe they go, oh hey, Axios is popular and widely trusted, let's make it an official library and bring the maintainers into the fold... wait isn't this exactly where we started?
What process did you trust the standard library/language maintainers in the first place? How do they differ from any other major library vendor?
I agree with you and follow the same principles myself, but JavaScript already has HTTP, and yet everyone still uses Axios. So the problem isn't that JS doesn't have batteries, it's that people don't want to use them for some reason.
I'm guessing it's similar to the tragedy of the commons phenomenon. When things are freely available people tend to overuse or carelessly use them. NPM is just too easy to use. If a package offers a 1% ergonomics increase over a builtin function, many folks will just go for it because it costs them nothing (well, it seems to cost them nothing).
NPM should learn from Linux distribution package managers.
Have a branch called testing, and packages stay in testing for few weeks, after which they go to stable. That is how many Linux distributions handle packages. It would have prevented many of these.
Advising every user of npm/pnpm to change their settings and set their own cooldown periods is not a real choice.
NPM is one big AUR, where anyone can submit arbitrary unverified code. The difference is that AUR is intentionally harder to use to prevent catastrophic one-line installs.
Is a "AUR" now just how we name unaudited software repositories?
Just to note, if we're talking about Linux Distributions. There's also COPR in Fedora, OBS for OpenSUSE (and a bunch of other stuff, OBS is awesome), Ubuntu has PPAs. And I am sure there's many more similar solutions.
Not all distributions work with a staging repository, and it's not really intended for this purpose either.
Besides there's always a way to immediately push a new version to stable repositories. You have to in order to deal with regressions and security fixes.
I know not all, but Debian/Ubuntu/Fedora does, and while the intended purpose of multi-stage releases is not necessarily security but stability, it still does help up with security too. Because third parties can look and scan the dependencies while they are still not in stable.
Most of the supply chain vulnerabilities that ended up in the NPM would have been mitigated with having mandatory testing / stable branches, of course there needs to be some sort of way to skip the testing but that would be rather rare and cumbersome and audited, like it is in Linux distributions too.
There's a package manager discussion, but the bit that stands out to me is that this started with a credential compromise. At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar. I wish that _important keys live in hardware_ becomes more standard given the stakes.
Dealing with dependencies is another question; if it's stupid stuff like leftpad then it should be either vendored in or promoted to be a language feature anyway (as it has been).
Or those people can (fund) separate repackaging and redistribution with more stringent and formalized review process.
Maybe not all users should pull all packages straight from what devs are pushing.
There's no reason we can't have "node package distributions" like we have Linux distributions. Maybe we should stop expecting devs and maintainers and Microsoft to take responsibility for our supply-chain.
Also, considering how prevalent TPM/Secure Enclaves are on modern devices, I would guess most package maintainers already have hardware capable of generating/using signing keys that never leave hardware.
I think it is mostly a devex/workflow question.
Considering the recent ci/cd-pipeline compromises, I think it would make sense to make a two phase commit process required for popular packages. Build and upload to the registry from a pipeline, but require a signature from a hardware resident key before making the package available.
Most of axios' functionality has effectively been promoted to a language feature as `fetch`, but the problem is people don't bother to migrate. I've migrated our direct usage of it but it's still pulled in transitively in several parts of our codebase.
Even left-pad is still getting 1.6 million weekly downloads.
Annoyingly, the times I reach for axios and similar is when I need to keep track of upload progress, which I could only do with XMLHttpRequest, not fetch, unless I've missed some recent browser changes, and the API of XMLHttpRequest remains as poor as the first times I had to use it. Download progress been supported by fetch since you can track chunks yourself, but somehow they didn't think to do that for requests for some reason, only responses.
At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar
I kind of feel like the authors here should want that for themselves, before the community would even realize it's needed. I can't say I've worked on packages that are as popular as axios, but once some packages we were publishing hit 10K downloads or so, we all agreed that we needed to up our security posture, and we all got hardware keys for 2FA and spent 1-2 weeks on making sure it was as bullet-proof we could make it.
To be fair, most FOSS is developed by volunteers so I understand not wanting to spend any money on something you provide for free, but on the other hand, I personally wouldn't feel comfortable being responsible for something that popular without hardening my own setup as much as I could, even if it means stopping everything for a week.
maybe the community could chip in to buy the authors a couple of YubiHSM
There's no community, the users of axios are devs that looked at stackoverflow for "how to download a file in javascript", they barely know or care what axios is.
Now the users of axios are devs that ask Claude Code or Codex to scrape a website or make a dashboard, they don't even know about the word axios.
I personally had to delete axios a couple of time from my codebase when working with junior devs.
I am not saying this is the reason for this compromise but the sudden explosion of coding assistant like claude code, and tools like openclaw is teaching entire crop of developers (and users) that it is ok to have sensitive credentials .env files.
I know there is a cooldown period for npm packages, but I’m beginning to want a cooldown for domains too. According to socket, the C2 server is sfrclak[.]com, which was registered in the last 24 hours.
This creates a secondary deception layer. After infection, running npm list in the project directory will report plain-crypto-js@4.2.0 — because npm list reads the version field from the installed package.json, which now says 4.2.0. An incident responder checking installed packages would see a version number that does not match the malicious 4.2.1 version they were told to look for, potentially leading them to conclude the system was not compromised.
WTF!!!! gaslighting your victims into believing they are not victims. the ingenuity of this is truly mindblowing. I am shocked at such thing is even allowed. like packages should not be able to modify their contents while they are being instaleld.
Guix saves you from this. You can import NPM packages in a container (not even touching $HOME) and giving you a shell on the spot with just the dependencies and nothing more.
Learn about 'guix import'.
Oh, and you can install Guix on any GNU/Linux distro.
With all the recent supply chain attacks, I'm starting to think it's only a matter of time before all of us are victims. I think this is a sign to manually check all package diffs or postinstall scripts.
Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
This needs to be done (as we've seen from these recent attacks) in your devenv, ci/cd and prod environments. Not one, or two, but all of these environments.
The easiest way is via using something like kubernetes network policies + a squid proxy to allow limited trusted domains through, and those domains must not be publicly controllable by attackers. ie. github.com is not safe to allow, but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it.
Network firewalls that perform SSL interception and restrict DNS queries are an option also, though more complicated or expensive than the above.
This stops both DNS exfil and HTTP exfil. For your devenv, software like Little Snitch may protect your from these (I'm not 100% on DNS exfil here though). Otherwise run your devenv (ie vscode) as a web server, or containerised + vnc, a VM, etc, with the same restrictions.
Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
Getting zero day patches 7 days later if no proper monitoring about important patches or if this specific patch is not in the important list. Always a tradeoff.
Thats true. Setting to 7 days saves you from a supply chain attack, but opens you to zero days. Another example why network filtering is a better solution.
but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it
But raw.githubusercontent.com still contains code and now the attacker can publish the code he wants no!?
Don't get me wrong: I love the idea to secure as much as possible. I'm running VMs and containerizing and I eat firewalling rules for breakfast, my own unbound DNS with hundreds of thousands (if not millions) of domains blocked, etc. I'm not the "YOLO" kind of guy.
But I don't understand what's that different between raw.githubusercontent.com and github.com? Is it for exploits that are not directly in the source code? Can you explain a bit more?
In the case of compromised code, the attacker has already loaded what he wants, so loading extra code from raw.githubusercontent.com is not the issue, or our threat model. We are already compromised!
The issue is that code then extracting secrets and data from your organisation, ie. data exfil.
raw.githubusercontent.com can not be used to submit data to, it's read only, but github.com obviously can.
Note, if you really needed github.com access in your application or environment, then you need to use SSL interception (using squid or a firewall) and allow certain URLs and methods ie. GET requests only from your organisations path, to make it safe.
Just a reminder that you can run most node things with deno run and have opt in permissions, audit trail and even external permission system integration now. The gotcha is that "deno task <<some package.json script>>" will NOT execute with this model which I find extremely unintuitive and had me thinking deno abandoned its sandbox for nodejs compatibility completely.
Can we get a non-AI-generated article for this? I think the aikido one might be fine, but if there’s a more official source let’s use that in lieu of this AI nonsense.
There’s a recurrent pattern with these package compromises: the attacker exfiltrates credentials during an initial phase, then pivots to the next round of packages using those credentials. That’s how we saw them make the Trivy to LiteLLM leap (with a 5 day gap), and it’ll almost certainly be similar in this case.
The solution to this is twofold, and is already implemented in the primary ecosystems being targeted (Python and JS): packagers should use Trusted Publishing to eliminate the need for long lived release credentials, and downstreams should use cooldowns to give security researchers time to identify and quarantine attacks.
(Security is a moving target, and neither of these techniques is going to work indefinitely without new techniques added to the mix. But they would be effective against the current problems we’re seeing.)
I don’t know what this means. Discipline is good, but I think you need to have good tools/primitives in place to help people exercise discipline.
(The classic example being passwords: we wouldn’t need MFA is everybody just “got good” and used strong/unique passwords everywhere. But that’s manifestly unrealistic, so instead we use our discipline budget on getting people to use password managers and phishing-resistant MFA.)
In this case, the author's NPM account was taken over, email address changed to one the attacker controls, and the package was manually published.
Since the attacker had full control of the NPM account, it is game over - the attacker can login to NPM and could, if they wanted, configure Trusted Publishing on any repo they control.
Axios IS using trusted publishing, but that didn't do anything to prevent the attack since the entire NPM account was taken over and config can be modified to allow publishing using a token.
Well, that sucks! It’ll be interesting to learn how they obtained a valid second factor or 2FA bypass; that will inform the next round of defenses here.
Yeah, NPM should be enforcing 2FA and likely phishing resistant 2FA for some packages/ this should be a real control, issuing public audit events for email address changes, and publish events should include information how it was published (trusted publishing, manual publish, etc).
You don't want "project had X users so it's less safe" to suddenly transition into "now this software has X*10 users so it has to change things", it's disruptive.
It's not much better than nothing. It basically solves "I reused my password across sites" exclusively, that's it. If you're going to go through the effort of TOTP, it seems odd that you wouldn't just use a unique password.
If you use a unique password it's questionable if it adds any value at all. Perhaps in very niche situations like "password authentication is itself vulnerable due to a timing attack/ bug" or some such thing... but we've rarely seen that in the wild.
I use a password manager and systemically use long random passwords. An attacker would need to compromise my password manager, phish me, wrench me, or compromise the site the credential is associated with to get that.
Using local only TOTP (no cloud storage or portability for me, by choice) they would have to additionally phish me, wrench me, compromise my phone, or compromise my physical security to get the code.
None of these are easy except the wrench which is high risk. My password manager had standard features which make me more phishing resistant, and together they are more challenging than either apart. For example the fact that my password manager will not fill in the password on a non associated site means I am much less likely to fill in a TOTP code on an inappropriate site. Though there are vulnerable scenarios they aren't statistically relevant in the wild and the bar is higher regardless.
Now I happen to have a FIDO key which I use for my higher security contexts but I'm a fairly low value target and npm isn't one of my high security contexts. TOTP improves my security stance generally and removing it from npmjs.org weakened my security stance there.
Important: Publishing to npm requires either:
Two-factor authentication (2FA) enabled on your account, OR
A granular access token with bypass 2FA enabled
One wonders if Microsoft/npm.js should allow new packages to be published immediately following an account email address change? I mean changes to email address are already recognized as potential attack vectors, so emails are sent to the old address warning of potential account take over. But this seems to have been done at night, so the warning email would not be seen yet. Even so a new package could be published and served to the world immediately. Unless I misunderstand something about the facts this would indicate an extreme lack of imagination in the people at Microsoft who already went through several cycles of hardening the service against supply chain poisoning attacks.
There's another element to the solution here: runtime behavioral analysis. No matter how completely the maintainer's credentials are compromised, no matter how well the malware is concealed, it still has to act like malware (in the case of LiteLLM, credential harvesting, in this case a remote-access Trojan). It's possible to detect the behavior, rather than relying on supply-chain integrity.
Very detailed and props to the security researchers, but the blog post has several indicators that it was written by AI, to which point I suspect their malware analysis was also done by a LLM.
I just wish it had more human interaction rather than have a GenAI spit out the blog post. It's very repetitive and includes several EM dashes.
It is harder and harder to trust any blog post anymore, the more AI there is. I used to read blog posts because of the personality and the precision level. Now both have been taken away.
It was in part written by a LLM, but I believe some of the analysis was done by humans. You can just check where there's proper punctuation and em-dashes in the commented code
I used axios in the distant past but haven't used it whenever I had my say in the past five years. You don't need it, and for special things like retries I could roll my own just fine. Now ai will roll it for you
This is exactly to avoid this kind of issue that I decided to work on StableBuild.
StableBuild pins and hosts a copy of your dependencies at a specific freeze date, so that your supply chain is never contaminated.
This way, a compromised version published after your freeze date (even with the same version number!) would never reach your build.
Is this Jia Tan 5.0? I've lost count. You really should stop trusting packages (implicitly). Or don't. It's your funeral, not mine. See you at Jia Tan 6.0 April?
Not at all, it was a regular maintainer account that was hijacked (probably through phishing) and used to push a malicious payload, not a threat actor posing as a contributor and adding a backdoor like in the Jia Tan case.
I'd argue this has not much in common with Jia Tan apart from both being supply chain attacks, there is no malicious maintainer here, a trusted maintainer had their account taken over.
I guess the end result is the same, a malicious package pushed by an account that was thought to be trusted, but I think the Jia Tan case is worth being looked at differently than just simple account takeover.
It's just a longer backstory. All the same in the end. Hackers targeted a popular package. The lead maintainer was compromised. The pattern fits. There will be more of these.
Of course the "wait 7 days" are not a silver bullet, but it gives automated scanners plenty of time to do their work. Those automated scanners surely catch this `eval(base64.decode("..."))` stuff that some of those attacks used so in my book this dependency cooldown is a net win. I guess the skilled malicious actors will then up their game but I think it's okay to kick off an arms race between them and the security scanners in the dependency world.
That's a good point. In some level I'd prefer the delay to happen on publication of the package itself. Do any of these scanners have cryptographic attestations or similar?
What a great time to be alive! Now, that's exactly why I enjoy writing software with minimal dependencies for myself (and sometimes for my family and friends) in my spare time - first, it's fun, and second, turns out it's more secure.
This only limits the possibility of compromise, it doesn't remove it. Python itself could be compromised, or the package that your linux distro provides could be.
With AI agents the volume and frequency of supply chain attacks is going to explode. I think our entire notion of how to develop and distribute software safely needs to change. I don't have answers; "reflections on trusting trust" explains the difficulties we now face.
How is it we've made it this far and we still don't have any kind of independent auditing of basic publish security on NPM? You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
Because all mainstream packages are published via CI/CD pipeline not by an MFA'd individual uploading a GZIP to npm.com
Requiring a human-in-the-loop for final, non-prerelease publication doesn't seem like that onerous of a burden. Even if you're publishing multiple releases a day on the regular (in which case ... I have questions, but anyway) there are all sorts of automations that stay secure while reducing the burden of having to manually download an artifact from CI, enter MFA, and upload it by hand.
You can still have a step that requires a certain user/group to sign off, and you can still enforce that those users have MFA set up. Almost any serious shop that expects to pass audits already does this in some form or fashion before pushing code to prod.
All sorts of reasons, but this isn't a left-pad situation. Axios's functionality is something provided by a library in a lot of languages (C/C++ with libcurl and friends, Python with requests, Rust with reqwest, and so on).
That's not to say it's inherently necessary for it to be a third-party package (Go, Ruby, and Java are counterexamples). But this isn't a proliferation/anemic stdlib issue.
We're going to see this more and more and more. And it's not going to stop. Because nobody in the industry will use the simplest, industry-standard security practices. Because they don't feel like it. A software building code is the only thing that'll fix it.
1/5 of your CLI and 1/3 of your gui apps are npm based. Each one has 400+ dependencies , none notable enough to go viral when they are breached. And who knows what other packages are currently compromised. We all have 30+ node_modules on our disks, and 2/3 of them were shipped by outside vendors , packaged in an archive.
“I’m smart I use fetch instead of axios”. “I pin my versions” – sure but certainly one of your npx or Electron apps uses axios or another less notably compromised package.
Has anyone else noticed there was a recent sudden flurry of 3000 deleted issues on axios/axios? The jump happened on March 23. Was this a first sign of compromise? Or just coincidence of an AI agent going rogue.
Maybe just a coincidence but they have cubic-dev-ai edit every single PR with a summary. And that bot edits PR descriptions even for outside contributors.
nope this was just someone bombing the repo throught the API it seemd
i then just closed and deleted them with a script.. seems it is happening with a couple repos, blocked the users who were doing this
I think it could well have been the attacker trying to hiding any notifications of suspicious activity (email address changed, suspicious login) in the flurry of issue related emails.
Do we have a way yet to tell if something on our system is compromised? There's plenty of end user software built on node, like Gemini CLI and LM Studio.
I've been advocating to minimize the number of dependencies for some time now. Once you've maintained an open source project for several years, you start to understand the true cost of dependencies and you actually start to pay attention to the people behind the libraries. Popularity doesn't necessarily mean reliable or trustworthy or secure.
Agree. This is one of the major takeaways I've had from writing Go over the years -- which is even a Go proverb [0], "a little copying is better than a little dependency." Fortunately, LLMs make writing your own implementations of little dependencies super easy too.
I'll agree with that, and you would think it's common sense for any competent engineer, but for many people it's just an afterthought. Including senior and lead engineers. General matters like security are a political liaison, how can you raise the alarm/advocate for critical security improvements as an IC without making people around and above you look bad? How can you justify time invested into these things without raising the alarm? At the same time, you can't just ignore glaring security holes. It's a fine line to walk, and actually being realistic about the possibilities has found me nothing but enmity from peers and superiors due to it seeming like I'm throwing them under the bus.
In general, management was to see progress. I've come to find that technical details like these are an afterthought for most engineers, so far as the deadlines are being met.
It's one of these things that are under the water, tech side jobs. Everyone has to be on board, if your peers don't give a fuck you're just an annoyance and will be swimming counter-current.
Can relate. It's like; if you're less rigorous than the CTO, they would think you're incompetent. If you're more rigorous than the CTO, they would think you're overly pedantic; not pragmatic enough.
Ok it's bad, but our npm projects are pinned in the package-lock.json, which I imagine most would be? So who would pull this besides security scanners?
I don't think that's right if it's in your package-lock it wouldn't pull it unless you npm update axios, or delete the package-lock.json and then npm install.
This was not opportunistic. It was precision. The malicious dependency was staged 18 hours in advance.
Another obvious ChatGPT-ism. The fact that people are using AI to write these security posts doesn't surprise me, but the fact they use it to write a verbose article with spicy little snippets that LLMs seem to prefer does make it really hard to appreciate anything other than the simple facts in the article.
I'm assuming you are talking about agents like claude-code and open-code which rely on GPT functions (AKA Large Language Models).
The reason they don't detect these risks is primarily because these risks are emergent, and happen overnight (literally in the case of axios - compromised at night). Axios has a good reputation. It is by definition impossible for a pre-trained LLM to keep up with time-sensitive changes.
I mean that agents can scan the code to find anything "suspicious". After all, security vendors that claim to "detect" malware in packages are relying on LLMs for detection.
An LLM is not a suitable substitute for purpose-built SAST software in my opinion. In my experience, they are great at looking at logs, error messages, sifting through test output, and that sort of thing. But I don't think they're going to be too reliable at detecting malware via static analysis. They just aren't built for that.
At this point picking Node for a backend is a foot gun. Large companies have the funds for private, security vetted npm repositories, but what about small companies, startups, personal projects? Pnpm makes things more secure without install scripts, mininum package time, but it's still the same activity, does an extra parachute make skydiving any less inherently dangerous?
I'm not dogmatic about the whole "JS for the backend is sin" from backend folks, but it seems like it was the right call. You should stick to large org backed packages, or languages with good enough standard libraries, like Go, Java, Python, C#.
Many of the suggestions in this thread (min-release, ignore script) are defenses for the consumers.
I've been working on Proof of Resilience, a set of 4 metrics for OSS, and using that as a scoring oracle for what to fund.
Popularity metrics like downloads, stars, etc are easy to fake today with ai agents. An interesting property is that gaming these metrics produces better code, not worse.
These are the 4 metrics:
1. Build determinism - does the published artifact match a reproducible build from source?
2. Fuzzing survival - does the package survive fuzz testing?
3. Downstream stability - does it break any repos dependent on this project when pushing a release?
4. Patch velocity - how fast are fixes merged?
Here's a link to the post, still early but would appreciate any feedback.
Carl, with all due respect, have you used AI for making this hackmd post?
"it's not just a waste of money — it's a security problem"
I am really passionate about these things, but I am not going to read something which you haven't written. Even sharing a prompt/rough-sketches/raw-writing might be beneficial but I recommend writing it by-hand man, we are all burnt out reading AI slop, I can't read more AI
You're right, I used an LLM to help write it from sketches. Gonna rewrite it properly because I think the ideas are worth exploring.
Thanks for taking the time to read and reply.
In my opinion, its okay to use LLM to help find some sources and then validating them (but I also recommend using hand researching too as you might find some good things that you maybe weren't even looking for!)
but, please don't use LLM to help write it from sketches. Even show the sketch :)
Much of my writing is very sketch-y. Some people don't like it, but its mine and I am proud of it and I hope that even if you write sketches/refine them, you can be comfortable sharing your ideas in your words in the way you wish to write them carl!
My thinking is that, I improve my writing by well... practice itself. So I write publically and there are some thoughts which occur in my head during the writing process itself (PG has a good article about it recently)
In a world of AI, to me, Human writing is a breath of fresh air. Please don't fall into the rabbit-hole that you might need LLM to help write you.
These are just my 2 cents though, but I feel like I am definitely not alone in thinking so.
Have a nice day and I am looking forward for you to write the article yourself. Feel free to share me when you do with my mail as I would love to read it, as I am also passionate about the funding of open source :)
This is the part that's tough — we push everyone to keep dependencies updated and automate it with Renovate or Dependabot, but that's exactly the pipeline that would have pulled this in before anyone noticed. Lockfiles and pinning help slow it down, but most teams pair those with automated update PRs which kind of negates the point. You can reduce your dependency surface area to lower the odds but one compromised maintainer on a top-10 package and none of that matters.
Rejecting any packages newer than X days is one nice control, but ultimately it'd be way better to maintain an allowlist of which packages are allowed to run scripts.
Unfortunately npm is friggen awful at this...
You can use --ignore-scripts=true to disable all scripts, but inevitably, some packages will absolutely need to run scripts. There's no way to allowlist specific scripts to run, while blocking all others.
There are third-party npm packages that you can install, like @lavamoat/allow-scripts, but to use these you need to use an entirely different command like `npm setup` instead of the `npm install` everyone is familiar with.
This is just awful in so many ways, and it'd be so easy for npm to fix.
# If you have a projects folder containing multiple projects on macOS, you can run this script to recursively scan all subfolders for vulnerable axios versions and the presence of plain-crypto-js, helping you quickly identify potentially affected projects:
This is the second major npm supply chain attack this year and the playbook is identical every time: hijack a maintainer account, publish via CLI to bypass CI/CD, inject a dependency nobody's heard of.
The fix isn't better scanning (though Socket catching it in 6 minutes is impressive). The fix is npm making Trusted Publishers mandatory for packages above a download threshold. If axios can only be published through GitHub Actions OIDC, a stolen password is useless.
We run a fleet of AI agents that depend on npm packages. First thing we did tonight was audit every lockfile. Clean — but only because we aggressively minimise dependencies. The real victims here are the thousands of teams who npm install with ^ ranges and never check what changed.
My main question here is mostly why so many people still rely on axios for their fetch implementation.
Native fetch has been a thing in the JavaScript world for so long, and the DX gains to using axios over it are miniscule. The only thing I can think of is axios instances, but you can easily write a tiny wrapper for fetch that would do the same.
This is a genuine question - if you still use axios, why exactly?
FWIW I vibe coded https://github.com/astrostl/surplies to detect evidence of the Axios and LiteLLM malware, using StepSecurity's writeups as a data source.
The postinstall script vector is getting all the attention, but IMO the scarier part is how the attacker chain works: compromise one package's credentials, use that access to pivot to the next target. Trivy -> LiteLLM -> now potentially axios. Each compromised package becomes a credential harvester for the next round.\n\nThe min-release-age configs (now in npm, pnpm, bun, uv) are a good start, but they only work as herd immunity — you need enough early adopters installing fresh releases to trigger detection before the 7-day window expires for everyone else. It's basically a bet that security researchers will catch it faster than your cooldown period.\n\nFor Node specifically: if you're still using axios for new projects, it's worth asking why. Native fetch has been stable in Node since v21. One less dependency in your tree is one less attack surface.
Three hours between the malicious publish and npm pulling the versions. If your CI ran an install during that window, this went straight to prod. Most teams I've worked with still have loose version ranges somewhere in their dependency tree even if they think they've locked everything down.
1. Only the registry itself can build packages (only source provided)
2. Builds must be reproducable (no network or external files during build / publish)
3. New versions are hidden by default
4. Releases can only be published by an account, using a hardware 2fa token + password (no persistent login, no long lasting token)
5. All commits must be signed (maybe block web commits or add a cooldown of a few days?)
6. builtin scanners (using ai, virustotal, existing services)
7. if a security violation is found the version is instantly removed
8. Atleast 1 - 3 Days delay for releases
9. Hard no on binaries / post install scripts and binary data
10. blockchain like public record to see who published, updated, owns what
Supply chain woes continue
One more reason to use Fetch
until Node is compromised
Harder to do. Also node is not updated at the rate of npm deps.
Stop trying to make Fetch happen.
No, I will not stop trying to create a more standardized and secure software ecosystem.
The comment you replied to is a quote from the movie Mean Girls.
https://knowyourmeme.com/memes/stop-trying-to-make-fetch-hap...
Thats a coincidence
It’s a quote from the movie mean girls ;)
LMAO take my upvote
is there even a reason to use axios?
Called it yesterday.
Doesn’t npm mandate 2FA as of some time last year? How was that bypassed?
Apparently it's possible to create access tokens that bypass 2FA. Might've been this.
https://docs.npmjs.com/creating-and-viewing-access-tokens
Correct, for CI/CD systems that want to push releases.
If GitHub, gitlab, or circleci, trusted publishing is available. No access token whatsoever.
Incident tracking:
https://github.com/axios/axios/issues/10604
It's reasons like this why I refuse to download Node or use anything NPM. Thankfully other languages are better anyways.
Because no other language has ever had supply chain attacks ever, in history. Nope.
https://blog.rust-lang.org/2022/05/10/malicious-crate-rustde...
https://en.wikipedia.org/wiki/Log4Shell
https://blog.pypi.org/posts/2024-12-11-ultralytics-attack-an...
https://about.gitlab.com/blog/gitlab-catches-mongodb-go-modu...
https://www.reversinglabs.com/blog/packagist-php-repo-supply...
C++ ftw
Other languages have package managers (perl) and there are package managers in existence that are not so vulnerable to this issue. IMO, it stems from one place: Transitive dependencies and general opaqueness of the issue.
In package managers like pacman, apt, apk,... it's easier to catch such issue. They do have postinstall scripts, but it's part of the submission to the repo, not part of the project. Whatever comes from the project is hashed, and that hash is also visible as part of the submission. That makes it a bit difficult to sneak something. You don't push a change, they pull yours.
Log4Shell was hardly a supply-chain attack - just a latent bug in a widely-used library. That can happen anywhere.
Maven to this day represents my ideal of package distribution. Immutable versions save so much trouble and I really don't understand why, in the age of left-pad, other people looked at that and said, "nah, I'm good with this."
Completely agree. NPM has the only registry where massive supply chain attacks happen several times a year. Mainly the fault lies with NPM itself, but much of it is just a terrible opsec culture in the community.
Most package.jsons I see have semver operators on every dependency, so patches spread incredibly quickly. Package namespacing is not enforced, so there is no way of knowing who the maintainer is without looking it up on the registry first; for this reason many of the most popular packages are basically side projects maintained by a single developer*. Post-install scripts are enabled by default unless you use pnpm or bun.
When you combine all these factors, you get the absolute disaster of an ecosystem that NPM is.
*Not really the case for Axios as they are at least somewhat organized and financed via sponsors.
The semantics are irrelevant. The effect is what's important: Hijacking widely used software to exploit systems. The OC is somehow under the illusion that avoiding JS altogether is a silver bullet for avoiding this.
Forest > Trees
The semantics are very relevant, since you presented it as a supply-chain attack. If you call a library vulnerability a supply-chain attack, then your argument has lost coherence.
Avoiding package managers with shitty policies is the silver bullet for this attack vector. I get that it can be useful in the moment to retract published artifacts, or update them in-place, or run some code after your artifact is downloaded, but all of these are false economies in our hostile environment.
Log4Shell was not a supply chain attack.
Come on dude. The issue is the frequency and magnitude of these attacks. Log4Shell was also not a supply chain attack.
I looked at the Rust one for example, which is literally just a malicious crate someone uploaded with a similar name as a popular one:
Compared to Axios, which gets 83 million downloads and was directly compromised.
What an extremely disingenuous argument lol
What exactly do you think the argument is?
The issues have everything to do with npm as a platform and nothing with JS as a language. You can use JS without npm. Saying you'll escape supply chain attacks by not using JS is like saying you'll be saved from an car crash with a parachute.
Well, this particular case could be wholly avoided if it didn't take 2 decades to get competent HTTP(S) client into core language
JS as a language is part of the problem because the standard library is so minimal that people need to use a lot more 3rd party libraries than they would in most popular languages.
Should increase the delay to dependency updates.
Slow Russian roulette is still a losing strategy
It’s only a losing strategy if you assume everyone universally adopts the slow strategy, and no research teams spot it in the interim. For things with large splash radius, that’s unrealistic, so defenders have an information advantage.
Makes actual security patches tougher to roll out though - you need to be vigilant to bypass the slowdown when you’re actually fixing a critical flaw. But nobody said this would be easy!
Yeah. 7 days in 2026 is a LONG TIME for security patches, especially for anything public facing.
Stuck between a rock (dependency compromise) and a hard place (legitimate security vulnerabilities).
Doesn't seem like a viable long-term solution.
but wouldn't it work in this case? sure if a package was compromised for months/years it wouldn't save you
but tell dependabot to delay a week, you'd sleep easy from this nonesense
slowly walking through a minefield isn’t any safer than running.
So unless you’re saying the extra time will be spent inspecting every package, whenever you do update, you will be getting an insecure package.
You’re not safe by dodging axios. There are currently thousands of breached packages ready to install that aren’t notable.
“I’ll run npm install after checking twitter” won’t help
Most packages don't become unsafe just because they were released a week ago.
How much do you want to bet me that the credential was stolen during the previous LiteLLM incident? At what point are we going to have to stop using these package managers because it's not secure? I've got to admit, it's got me nervous to use Python or Node.js these days, but it's really a universal problem.
My feelings precisely. Min package age (supported in uv and all JS package managers) is nice but I still feel extremely hesitant to upgrade my deps or start a new project at the moment.
I don’t think this is going to stabilize any time soon, so figuring out how to handle potentially compromised deps is something we will all need to think about.
NPM only gained minimum package age in February of this year, and still doesn't support package exclusions for internal packages.
https://github.com/npm/cli/pull/8965
https://github.com/npm/cli/issues/8994
Its good that that they finally got there but....
I would be avoiding npm itself on principle in the JS ecosystem. Use a package manager that has a history of actually caring about these issues in a timely manner.
It almost doesn't matter, because you can get pwned by a transitive dependency. If someone doesn't have the same scruples as you have, you're still at risk.
minimumReleaseAge and lockfiles also pin down transitive dependencies.
PNPM makes you approve postinstall scripts instead of running them by default, which helps a lot. Whenever I see a prompt to run a postinstall script, unless I know the package normally has one & what it does, I go look it up before approving it.
(Of course I could still get bitten if one of the packages I trust has its postinstall script replaced.)
How does this stance work with your CICD?
I suppose you would have to commit your node_modules, or otherwise cache your setup so that all prerequesite modules are built and ready to install without running post-install scripts?
More like the Trivy incident (which led to the compromise of LiteLLM).
I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
Also from the report:
Good news for pnpm/bun users who have to manually approve postinstall scripts.
Fetch wasn't added to Node.js as a core package until version 18, and wasn't considered stable until version 21. Axios has been around much longer and was made part of popular frameworks and tutorials, which helps continue to propagate it's usage.
Also it has interceptors, which allow you to build easily reusable pieces of code - loggers, oauth, retriers, execution time trackers etc.
These are so much better than the interface fetch offers you, unfortunately.
You can do all of that in fetch really easily with the init object.
})
What does an interceptor in the RequestInit look like?
A wrapper function around fetch… that’s what interceptors are…
There are pretty much two usage patterns that come up all the time:
1- automatically add bearer tokens to requests rather than manually specifying them every single time
2- automatically dispatch some event or function when a 401 response is returned to clear the stale user session and return them to a login page.
There's no reason to repeat this logic in every single place you make an API call.
Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
Finally, there's some nice mocking utilities for axios for unit testing different responses and error codes.
You're either going to copy/paste code everywhere, or you will write your own helper functions and never touch fetch directly. Axios... just works. No need to reinvent anything, and there's a ton of other handy features the GP mentioned as well you may or may not find yourself needing.
Helper functions seem trivial and not like you’re reimplementing much.
Don't be silly, this is the JS ecosystem. Why use your brain for a minute and come up with a 50 byte helper function, if you can instead import a library with 3912726 dependencies and let the compiler spend 90 seconds on every build to tree shake 3912723 out again and give you a highly optimized bundle that's only 3 megabytes small?
IMO interceptors are bad. they hide what might get transformed with the API call at the place it is being used.
This is not true unless you are not interfacing with your own backends. even then why not just make a helper that unwraps as json by default but can be passed an arg to parse as something else
Interceptors are just wrappers in disguise.
Convenience is a thing, but it doesn't require a massive library.
but it does for massive DDoS :p
That fetch requires so many users to rewrite the same code - that was already handled well by every existing node HTTP client- says something about the standards process.
It could also be trivially written for XMLHttpRequest or any node client if needed. Would be nice if they had always been the same, but oh well - having a server and client version isn't that bad.
Because it is so few lines it is much more sensible to have everyone duplicate that little snippet manually than import a library and write interceptors for that...
(Not only because the integration with the library would likely be more lines of code, but also because a library is a significantly liability on several levels that must be justified by significant, not minor, recurring savings.)
fetch responses have a .json() method. It's literally the first example in MDN: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/U...
It's literally easier than not using JSON because I have to think about if I want `repsponse.text()` or `response.body()`.
that's such a weak argument. you can write about 20 lines of code to do exactly this without requiring a third party library.
One more use case for Axios is it automatically follows redirects, forwarding headers, and more importantly, omiting or rewriting the headers that shouldn't be forwarded for security reasons.
fetch automatically follows redirects, fetch will forward your headers, omitting or rewriting headers is how security breaks… now a scraper got through because it’s masquerading as Chrome.
It also supports proxies which is important to some corporate back-end scenarios
fetch supports proxies
Before that we had node-fetch. If you already use a dependency why not one that's pretty much what will come natively to every JS runtime soon.
The fetch API is designed for browsers. It's not designed for servers. Fetch may work for a particular use case on the server, it may not. Servers have needs over and above what a browser allows the client to do.
Now I'm curious, because we have a big server side code base using fetch(). What are you using that doesn't work with fetch? Especially since axios nowadays has a fetch adapter.
Right. Though I would've used the built in xhr then. Not going to install a dep just to make http calls.
Would they not have approved it for earlier versions? But also wouldn't the chance of addition automatic approval be high (for such a widely used project)?
The prompt would be to approve the new malicious package (plain-crypto-js)'s scripts, too, which could tip users off that something was fishy. If they were used to approving one for axios and the attackers had just overwrote axios's own instead of making a new package, it would probably catch people out.
Can't speak for other devs but I like to read postinstall scripts or at least put them through an LLM if they're too hard to grok.
It's also a little context dependent, for example if I was using Axios and I see a prompt to run the plain-crypto-js postinstall script, alarm bells would instantly ring, which would at least make me look up the changelog to see why this is happening.
In most cases I don't even let them run unless something breaks/doesn't work as expected.
Assuming axios didn't have a postinstall script before, it wouldn't have been approved for a previous version. If you ignore it, you ignore it, but postinstall scripts are relatively rare in npm deps, so it would seem a bit out of place when the warning pops up.
You can remember this answer for every time you ask same question again:
"Coz whatever else/builtin was before was annoying enough for common use cases"
Because axios existed before the builtin fetch, and so there's a lot of stackoverflow answers explaining how to use fetch, and the llm models are trained on that, so they will write axios requests instead of fetch
I spent two years trying to get it out of a project that began long after Axios had become redundant but it's very hard to go back and challenge decisions like this because every business priority is aligned against this kind of work.
I expect libraries built on top of fetch will be the next to be compromised, because why would you use fetch without an arbitrary layer of syntactic sugar...
There was never the business value. But now remember this axios case and use it as ammo for the next issue. Just don’t abuse it
Has anyone tested general purpose malware detection on supply chains ? Like clamscan . I tried to test the LiteLLM hack but the affected packages had been pulled. Windows Defender AV has an inference based detector that may work when signatures have not yet been published
You could use Trivy! /s
I second this question. I usually scan our containers with snyk and guarddog, and have wondered about guarddog in particular because it adds so much build time.
Hey, I have been part of the archival effect/Litellm issue thread. I think I have stored them in archive.org for preservation purposes
https://web.archive.org/web/20260325073027/https://files.pyt...
(I have also made an archive of the github issue with all the comments manually till a certain point at https://web.archive.org/web/20260325054202/https://serjaimel...)
thanks for highlighting that i will take a look and see if there's similar archive for the other vulnerabilities as well .
If i can make it work with clamscan & MS Defender i'll run a scan and try to report back
Glad to see that Clamscan experiment worked. Keep me updated on the continued testing and I am glad that my archival efforts are appreciated :)
absolutely massive help. i love HN community and thanks. If you do see an archive of axios or other compromised artifacts, please send those my way for continued testing. i'm going to test automation and see if this actually has utility.
Hey tony, sorry I had read your comment, I was a little busy. I will try to do something about axios tomorrow to help you out!
And I love HN community too and your welcome!
no worries. I ended up finding a compromised axios archive online and the test worked, with the unofficial sigs
https://github.com/extremeshok/clamav-unofficial-sigs/tree/m...
the primitive clamscan experiment worked! it detected Txt.Trojan.TeamPCP-10059839-1 from the .tar.gz archive. I'll continue testing to see if it's viable
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default.
Here's how to set global configs to set min release age to 7 days:
(Side note, it's wild that npm, bun, and pnpm have all decided to use different time units for this configuration.)
If you're developing with LLM agents, you should also update your AGENTS.md/CLAUDE.md file with some guidance on how to handle failures stemming from this config as they will cause the agent to unproductively spin its wheels.
and for yarn berry
If everyone avoids using packages released within the last 7 days, malicious code is more likely to remain dormant for 7 days.
They’re usually picked up by scanners by then.
that's why people are telling others to use 7 days but using 8 days themselves :)
Genius
brb, switching everything to 9 days
That is 3D chess level type shit. xD
You don't have to be faster than the bear, you just have to be faster than the other guy.
What do you base that on? Threat researchers (and their automated agents) will still keep analyzing new releases as soon as they’re published.
Their analysis was triggered by open source projects upgrading en-masse and revealing a new anomalous endpoint, so, it does require some pioneers to take the arrows. They didn't spot the problem entirely via static analysis, although with hindsight they could have done (missing GitHub attestation).
A security company could set up a honeypot machine that installs new releases of everything automatically and have a separate machine scan its network traffic for suspicious outbound connections.
The problem is what counts as suspicious. StepSecurity are quite clear in their post that they decide what counts as anomalous by comparing lots of open source runs against prior data, so they can't figure it out on their own.
The entire history of malware lol
Can you elaborate? Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
Attackers going "low and slow" when they know they're being monitored is just standard practice.
I'm sure they will, but attackers will adapt. And I'm really unconvinced that these delays are really going to help in the real world. Imagine you rely on `popular-dependency` and it gets compromised. You have a cooldown, but I, the attacker, issue "CVE-1234" for `popular-dependency`. If you're at a company you now likely have a compliance obligation to patch that CVE within a strict timeline. I can very, very easily pressure you into this sort of thing.
I'm just unconvinced by the whole idea. It's fine, more time is nice, but it's not a good solution imo.
What, in your view, is a better solution?
There are many options. Here's a post just briefly listing a few of the ones that would be handled by package managers and registries, but there are also many things that would be best done in CI pipelines as well.
https://news.ycombinator.com/item?id=47586241
The fact threat researchers and especially their automated agents are not all that good at their jobs
Those threat researchers and their autonomous agents caught this axios release.
I suspect most packages will keep a mix of people at 7 days and those with no limit. That being said, adding jitter by default would be good to these features.
This became evident, what, perhaps a few years ago? Probably since childhood for some users here but just wondering what the holdup is. Lots of bad press could be avoided, or at least a little.
Which will never even come close to happening, unless npm decides to make it the default, which they won't.
Most people won’t.
7 days gives ample time for security scanning, too.
This highly depends on the detection mechanism.
About the use of different units: next time you choose a property name in a config file, include the unit in the name. So not “timeout” but “timeoutMinutes”.
timeoutMs is shorter ;)
You guys can't appreciate a bad joke
not timeout at all is even shorter.
Megaseconds are about the right timescale anyway
What megaseconds? They clearly meant the Microsoft-defined timeout.
Well megaseconds has the nice property that it's about about equal to a Scaramucci so it can be used across domains.
timoutμs is even better. People will learn how to type great symbols.
Yes timout indeed!
They wouldn't have to, if the file format accepted floats in proper exponential format.
Or require the value to specify a unit.
At that point, you're making all your configuration fields strings and adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either; either you write a bunch of parsing code (not terribly difficult but not something I wanna do when I can just not), or you use some time library to parse a duration string, in which case the programming language and time library you happen to use suddenly becomes part of your config file specification and you have to exactly re-implement your old time handling library's duration parser if you ever want to switch to a new one or re-implement the tool in another language.
I don't think there are great solutions here. Arguably, units should be supported by the config file format, but existing config file formats don't do that.
TOML has a datetime type (both with or without tz), as well as plain date and plain time:
We should extend it with durations:
And like for datetimes, we should have a readable variant:
Edit: discussed in detail here: https://github.com/toml-lang/toml/issues/514
great, now attackers can also target all the libraries to enable all that complexity in npm too.
I'd argue that it is ideal, in the sense that it's the sweet spot for a general config file format to limit itself to simple, widely reusable building blocks. Supporting more advanced types can get in the way of this.
Programs need their own validation and/or parsing anyway, since correctness depends on program-specific semantics and usually only a subset of the values of a more simply expressed type is valid. That same logic applies across inputs: config may come from files, CLI args, legacy formats, or databases, often in different shapes. A single normalization and validation path simplifies this.
General formats must also work across many languages with different type systems. More complex types introduce more possible representations and therefore trade-offs. Even if a file parser implements them correctly (and consistently with other such parsers), it must choose an internal form that may not match what a program needs, forcing extra, less standard transformation and adding complexity on both sides for little gain.
Because acceptable values are defined by the program, not the file, a general format cannot fully specify them and shouldn’t try. Its role is to be a medium and provide simple, human-usable (for textual formats), widely supported types, avoid forcing unnecessary choices, and get out of the way.
All in all, I think it can be more appropriate for a program to pick a parsing library for a more complex type, than to add one consistently to all parsers of a given file format.
Another parsing step is the common case. Few parameters represent untyped strings where all characters and values are valid. For numbers as well, you often have a limited admissible range that you have to validate for. In the present case, you wouldn’t allow negative numbers, and maybe wouldn’t allow fractional numbers. Checking for a valid number isn’t inherently different from checking for a regex match. A number plus unit suffix is a straightforward regex.
Yes!! This goes for any time you declare a time interval variable. The number of times I've seen code changes with a comment like "Turns out the delay arg to function foo is in milliseconds, not seconds".
First day with javascript?
You mean first 86,400 seconds?
You have to admire the person who designed the flexibility to have 87239 seconds not be old enough, but 87240 to be fine.
This is the difference between thinking about the user experience and thinking just about the technical aspect
I actually think it is not too bad a design, because seconds are the SI base unit for time. Putting something like "x days" requires additional parsing steps and therefore complexity in the implementation. Either knowing or calculating how many seconds there are in a day can be expected of anyone touching a project or configuration at this level of detail.
Seconds are also unambiguous. Depending on your chosen definition, "X days" may or may not be influenced by leap seconds and DST changes.
I doubt anyone cares about an hour more or less in this context. But if you want multiple implementations to agree talking about seconds on a monotonic timer is a lot simpler
I came here to argue the opposite. Expressing it in seconds takes away questions about time zones and DST.
I think you're incorrect to say that second are also ambiguous. Maybe what you mean is that days are more practical, but that seems very much a personal preference.
I think you misread the comment you're replying to.
I understand the [flawed] reasoning behind "x seconds from now is going to be roughly now() + x on this particular system", but how does defining the cooldown from an external timestamp save you from dealing with DST and other time shenanigans? In the end you are comparing two timestamps and that comparison is erroneous without considering time shenanigans
Could you explain what you mean re: ambiguity? I understand why “calendar units” like months are ambiguous, but minutes, hours, days, and weeks all have fixed durations (which is why APIs like Python’s `timedelta` allows them).
In the UK last Sunday was 23 hours long because we switched to BST, and occasionally leap seconds will result in a minute being something other 60 seconds.
The minute between December 31, 2016 23:59 and January 1st 2017 is 61 seconds, not 60 seconds. The hour that contains that minute is 3601 seconds, the day that contains that hour is 43201 seconds, etc. If you assume a fixed duration and simply multiply by 43200, your math will be wrong compared to the rest of the world.
Daylight savings time makes a day take 23 hours or 25 hours. That makes a week take 7254000 seconds or 7261200 seconds. Etc.
exploiting the ambiguity in date formats by releasing a package during a leap second
True. But seconds are not the base unit for package compromises coming to light. The appropriate unit for that is almost certainly days.
that kind of complexity is always worth it. Every single time. It's user time that you're saving and it also makes config clearer for readers and cuts out on "too many/little zeroes on accident" errors
It's just library for handling time that 98% of the time your app will be using for something else.
I find it best when I need a calculator to understand security settings. 604800 here we come
Probably went with the simplest implementation, if starting from the current “seconds since epoch” value. Let the user do any calculations needed to translate three days into that measurement.
It also efficiently annoys the most people at once: those what want hours will complain if they set it to days, thought that want days will complain if hours are used. By using minutes or seconds you can wind up both segments while not offend those who rightly don't care because they can cope with a little arithmetic :)
Though doing what sleep(1) does would be my preference: default to seconds but allow m/h/d to be added to change that.
I'm old enough to remember computers being pitched as devices that can do tedious math for us. Now we have to do tedious math for them apparently.
Hence the way I would do it (and have for other purposes), as stated in my final sentence. Have the human state the intent and convert to your own internally preferred units as needed.
I'm sure you would like to memorize all kinds of API instead of having something idiot proof and straightforward
As if `minimumReleaseAge` in `[install]` section of `.bunfig.toml` doesn't require the same kind of memorization.
Hey that's a great joke, you made me spill my morning home-brewed kombucha.
I'm going to steal that one for my JavaScript monthly developers meetup.
Is it ok if I attribute it to "Xirdus on Hacker News"?
Lol sure.
No no no, see now we just say "computer! do tedious math!", and it will do some slightly different math for us and compliment us on having asked it to do so.
The one true unit of time is hexadecimal encoded nanoseconds since the unix epoch. (I'm only half joking because I actually have authored code that used that before.)
Well, you have 1000000 microseconds in between. That's a big threshold.
wait what if we start on a day DST starts or ends????
OP should be glad a new time unit wasn't invented
N multiplications of dozen-second
Workdays! Think about it, if you set the delay in regular days/seconds the updated dependency can get pulled in on a weekend with only someone maybe on-call.
(Hope your timezones and tzdata correctly identifies Easter bank holiday as non-workdays)
And we also need localization. Each country can have their own holidays
And we need groups of locales for teams that are split across multiple locations; e.g.:
Might be better to calculate them separately for each locale and then tie-break with your own approach (min/max/avg/median/etc.)
Hopefully "es" will have Siesta support too.
Don't forget about regional holidays, which might follow arbitrary borders that don't match any of the official subdivisions of the country. Or may even depend on the chosen faith of the worker
Pulaski day in Illinois. Or Reds Opening Day in Cincinnati.
This is javascript, not Java.
In JavaScript something entirely new would be invented, to solve a problem that has long been solved and is documented in 20+ year old books on common design patterns. So we can all copy-paste `{ or: [{ days: 42, months: 2, hours: "DEFAULT", minutes: "IGNORE", seconds: null, timezone: "defer-by-ip" }, { timestamp: 17749453211*1000, unit: "ms"}]` without any clue as to what we are defining.
In Java, a 6000LoC+ ecosystem of classes, abstractions, dependency-injectables and probably a new DSL would be invented so we can all say "over 4 Malaysian workdays"
But you know that Java solution will continue working even after we no longer use the Gregorian Calendar, the collapse and annexation of Malaysia to some foreign power, and then us finally switching to a 4-day work week; so it'd be worth it.
It probably won’t work correctly from the get go. But it can be debugged everywhere so that’s good.
... and since it was architectured to allow runtime injection-patching of events before they hit the enterprise-service-bus, everyone using this library must first set fourteen ENV vars in their profile, and provide a /etc/java/springtime/enterprise-workday-handling/parse-event-mismatch.jar.patch. Which should fix the bug for you.
You can find the patch files for your OSs by registering at Oracle with a J3EE8.4-PatchLibID (note, the older J3EE16-PatchLib-ids aren't compatible), attainable from your regional Oracle account-manager.
JavaScript Temporal. Not sure knowing what a "workday" is in each timezone is in it's scope but it's the much needed and improved JS, date API (granted with limited support to date)
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
In before someone thinks it's a joke, the most commonly used logging library in Java had LDAP support in format scripts enabled by default" (which resulted, of course in CVE)
There's an extra digit in your timestamp.
Ah but you don't know how far in the future I intended it to be :)
Why would it get pulled in over the weekend? What automatic deployments are you running if there also isn't a human working to get it out?
Do you run automatic dependency updates over the weekend? Wouldn't you rather do that during fully-staffed hours?
When I worked in Finance our internal Date extension did actually have Workdays that took into account Stock Market and Bank Holidays.
…now imagine a list of instruments, some of which have durations specified in days/weeks/months (problems already with the latter) and some in workdays, and the user just told your app to display it sorted by duration.
I tried to write this function in Power Query (Excel hell). Gave up after an hour or so.
Me too, it was just a constant filled with bank holidays for the next 6 years
Nah, working hours and make global assumptions of 0900-1230/1330-1730, M-F, and have an overly convoluted way to specify what working ours actually are in the relevant location(s).
If we're taking suggestions, I'd like to propose "parsec" (not to be confused with the unit of distance of the same name)
That way Han Solo can make sense in the infamous quote.
EDIT: even Gemini gets this wrong:
They explained it in the Solo movie.
https://www.reddit.com/r/MovieDetails/comments/ah3ptm/solo_a...
Making a whole movie just to retcon the parsec misuse in Ep IV was a choice
They made a movie to make money. I doubt anyone holding the purse strings cared one iota if that bit were corrected or not. It’s not really a retcon either because they didn’t change anything.
That had more or less been the explanation in the books for decades, and even in George Lucas' notes from 1977:
It was already fine, because it’s a metric defined on a submanifold of relativistic spacetime.
Parallax arc-second -> distance.
For Star Wars, they retconned it to mean he found the shortest possible route through dangerous space, so even for Han Solo's quote, it's still distance.
To me it sounds safer to have different big infra providers with different delays, otherwise you still hit everyone at the same time when something does inevitably go undetected.
And the chances of staying undetected are higher if nobody is installing until the delay time ellapses.
It's the same as not scheduling all cronjobs to midnight.
Props to uv for actually using the correct config path jfc what is “bunfig”
Silly portmanteau of "bun" and "config"
A trendy sandwich
Pnpm did this first but I’m glad to see all the others follow suit
For anyone wondering, you need to be on npm >= 11.10.0 in order to use it. It just became available Feb 11 2026
https://github.com/npm/cli/releases/tag/v11.10.0
The config for uv won't work. uv only supports a full timestamp for this config, and no rolling window day option afaik. Am I crazy or is this llm slop?
https://docs.astral.sh/uv/concepts/resolution/#dependency-co...
My bad. This works for per project configuration, but not for global user configuration.
I think it should work at the user config level too:
https://docs.astral.sh/uv/concepts/configuration-files/
It should work for global configuration too, please file an issue if you’re observing otherwise.
(Make sure you’re on a version that actually supports relative times, please!)
This is what tripped me up. I added that config and then got this error:
error: Failed to parse: `.config/uv/uv.toml` Caused by: TOML parse error at line 1, column 17 | 1 | exclude-newer = "7 days" | ^^^^^^^^ failed to parse year in date "7 days": failed to parse "7 da" as year (a four digit integer): invalid digit, expected 0-9 but got
I was on version 0.7.20, so I removed that line, ran "uv self update" and upgraded to 0.11.2 and then re-added the config and it works fine now.
Yeah, that error message isn’t ideal on older versions, but unfortunately there’s no way to really address that. But I’m glad it’s working for you on newer versions.
For what it's worth the error made sense enough to me that I figured I needed to upgrade. :-)
Run npm/pnpm/bun/uv inside a sandbox.
There is no reason to let random packages have full access to your machine
Sandboxing by to default world be really nice. One of the things I really appreciate about Claude Code is its permissions model
I think the npm doesn't support end of line comments, so
actually doesn't set it at all, please edit your comment.
EDIT: Actually maybe it does? But it's weird because
`npm config list -l` shows: `min-release-age = null` with, and without the comment. so who knows ¯\_(ツ)_/¯
ok, it works, only the list function shows it as null...
Good luck with any `npm audit` in a pipeline. Sometimes you have to pull the latest release because the previous one had a critical vulnerability.
min release age to 7 days about patch releases exposes you to the other side of the coin, you have an open 7 days window on zero-day exploits that might be fixed in a security release
Out of the frying pan and into the frier.....
Exactly what I thought too when I read this...
Urgent fix, patch released, invisible to dev team cause they put in a 7 day wait. Now our app is vulnerable for up to 7 days longer than needed (assuming daily deploys. If less often, pad accordingly). Not a great excuse as to why the company shipped an "updated" version of the app with a standing CVE in it. "Sorry we were blinded to the critical fix because set an arbitrary local setting to ignore updates until they are 7 days old". I wouldn't fire people over that, but we'd definitely be doing some internal training.
Not really an issue though right because virtually none of these have lasted more than 1-2 days before being discovered?
At least with pnpm, you can specify minimumReleaseAgeExclude, temporarily until the time passes. I imagine the other package managers have similar options.
[1]: https://pnpm.io/settings#minimumreleaseageexclude
I haven't checked, but it would be surprising that the min-release-age applies to npm audit and equivalent commands
The packages that are actually compromised are yanked, but I assume you're talking about a scenario more like log4shell. In that case, you can just disable the config to install the update, then re-enable in 7 days. Given that compromised packages are uploaded all the time and zero-day vulnerabilities are comparatively less common, I'd say it's the right call.
`uv` has per-package overrides, I imagine there may be similar in other managers
Is there a way to do that per repo for these tools ? We all know how user sided configuration works for users (they usually clean it whenever it goes against what they want to do instead of wondering why it blocks their changes :))
At least with npm, you can have a .npmrc per-repo
pnpm does global + per-repo
Fairly sure every single one has a repo level config that you can add these settings to. Others have pointed out the pnpm and npm, and I bunfig can also be repo level.
It's wild that none of these are set by default.
I know 90% of people I've worked with will never know these options exist.
If everyone or a majority of people sets these options, then I think issues will simply be discovered later. So if other people run into them first, better for us, because then the issues have a chance of being fixed once our acceptable package/version age is reached.
That would likely mean same amount of people get the vulnerability, just 7 days later.
The compromised packages were removed from the registry within hours.
Because everyone got updates immediately. If the default was 7 days, almost no one would get updates immediately but after 7 days, and now someone only finds about after 7 days. Unless there is a poor soul checking packages as they are published that can alert the registry before 7 days pass, though I imagine very few do that and hence a dedicated attacker could influence them to not look too hard.
If I remember correctly, in all the recent cases it was picked up by automated scanning tools in a few hours, not because someone updated the dependency, checked the code and found the issue.
So it looks like even if no one actually updates, the vast majority of the cases will be caught by automated tools. You just need to give them a bit of time.
npm is claiming this doesn’t exist
Make sure you're on version 11.10 or later?
lol with mise I used a fourth time unit: https://mise.jdx.dev/configuration/settings.html#install_bef...
Where in the pnpm documentation does it say that it ignores scripts by default?
From https://pnpm.io/cli/install#--ignore-scripts:
Weird. The config also appears to default to `false`
https://pnpm.io/settings#ignorescripts
This page describes the behavior, "disables the automatic execution of postinstall scripts in dependencies":
https://pnpm.io/supply-chain-security
While this explicitly calls out "postinstall", I'm pretty sure it affects other such lifecycle scripts like preinstall in dependencies.
The --ignore-scripts option will ignore lifecycle scripts in the project itself, not just dependencies. And it will ignore scripts that you have previously allowed (using the "allowBuilds" feature).
And when you actually need a super hot fix for a 0-day, you will need to revert this and keep it that way for some time to then go back to minimum age.
While this works, we stillneed a permanent solution which requires a sort of vetting process, rather than blindly letting everything through.
Who will do the vetting process?
I think my vetting would settle for a repo diff against the previous version, confirming the only difference was the security fix (though that doesn't cover all the bases).
Jia Tan
pnpm since v10.19.0 allows excluding specific dependencies from minReleaseAge by version.
Everyone has forgotten standard ISO 8601 durations and invented their own syntax.
uv supports it, https://docs.astral.sh/uv/reference/settings/#exclude-newer
Perfect
mise has an option as well (note the caveats though):
https://mise.jdx.dev/configuration/settings.html#install_bef...
And homebrew has discussed it, kinda sorta:
https://github.com/Homebrew/brew/issues/21129
The solution is not moar toolz. That's the problem—this crazy mindset that the problems endemic to bad tooling have a solution in the form of complementing them with another layer, rather than fewer.
Git and every sane SCM already allow you to manage your source tree without jumping through a bunch of hoops to go along with wacky overlay version control systems like the one that the npmjs.com crew designed, centering around package.json as a way to do an end-run around Git. You don't need to install and deploy anything containing never-before-seen updates just because the NodeJS influencer–developers say that lockfiles are the Right Way to do things. (It's not.)
Opting in to being vulnerable to supply chain attacks is a choice.
<https://news.ycombinator.com/item?id=46006471>
<https://news.ycombinator.com/item?id=46360308>
`~/Library/Preferences/pnpm/rc` reads like is MacOS.. I'm using Linux...?
What version of npm has this? This isn't working for me on npm 11.6.2 (Node v24):
Pin your dependencies folks! Audit and don't upgrade to every brand new version.
But also have a regular review of your dependencies to update them when necessary, because as bad as compromised packages may be things do have vulnerabilities occasionally, and upgrading things that are a long way out-of-date can be quite hard.
I recommend everyone to use bwrap if you're on linux and alias all package managers / anything that has post build logic with it.
I have bwrap configured to override: npm, pip, cargo, mvn, gradle, everything you can think of and I only give it the access it needs, strip anything that is useless to it anyway, deny dbus, sockets, everything. SSH is forwarded via socket (ssh-add).
This limits the blast radius to your CWD and package manager caches and often won't even work since the malware usually expects some things to be available which are not in a permissionless sandbox.
You can think of it as running a docker container, but without the requirement of having to have an image. It is the same thing flatpak is based on.
As for server deployments, container hardening is your friend. Most supply chain attacks target build scripts so as long as you treat your CI/CD as an untrusted environment you should be good - there's quite a few resources on this so won't go into detail.
Bonus points: use the same sandbox for AI.
Stay safe out there.
AFAIK maven doesn’t support post install logic like npm does. You have to explicitly optin with build plugins. It doesn’t let any arbitrary dependency run code on your machine.
some post processors have chains to execution (ex: lombok)
You explicitly opt in by using a compiler plugin. Merely having it as a dependency, like in npm, doesn’t mean it can run code at build time.
Do you have a recommendation for something like bwrap but for macos? I've been trying to use bwrap more on my servers when I remember.
unfortunately not, but there is work being done to support overlays properly I think?
Maybe I misunderstood this point. But the ssh socket also gives access to your private keys, so I see no security gain in that point. Better to have a password protected key.
It's so your private key is not stolen, but you're right passphrase protected keys win anyway. I use hardware keys so this isn't a problem for me to begin with.
This only works for post-install script attacks. When the package is compromised, just running require somewhere in your code will be enough, and that runs with node/java/python and no bwrap.
node is also sandboxed within bwrap I have sandbox -p node if I have to give node access to other folders, I also have sandbox -m to define custom mountpoints if necessary and UNSAFE=1 as a last resort which just runs unsandboxed.
Check also https://github.com/wrr/drop which is a higher-level tool than bwrap. It allows you to make such isolated sandboxes with minimal configuration.
This looks nice but I wouldn't trust a very fresh tool to do security correctly.
As a higher-level alternative to bwrap, I sometimes use `flatpak run --filesystem=$PWD --command=bash org.freedesktop.Platform`. This is kind of an abuse of flatpaks but works just fine to make a sandbox. And unlike bwrap, it has sane defaults (no extra permissions, not even network, though it does allow xdg-desktop-portal).
Shame it's not a bit more mature, it does look like more the sort of thing I want. I use firejail a bit, but it's a bit awkward really.
To be honest - and I can't really believe I'm saying it - what I really want is something more like Android permissions. (Except more granular file permissions, which Android doesn't do at all well.) Like: start with nothing, app is requesting x access, allow it this time; oh alright fine always allow it. Central place to manage it later. Etc.
I wrote a Docker-based sandbox [1] for myself last year to control the blast radius of such malicious packages.
https://github.com/ashishb/amazing-sandbox
^ Vibe-coded slop spam ^
What makes you think that? Your can see the commit history <10% of code is written by agents.
Rest was all written by me.
Unlike other criticisms of the project, this one feels personal as it is objectively incorrect.
Oops, sorry. Only had a look at recent commits.
Very useful, thanks
I think firejail is a much more flexible security sandbox than bwrap. It also comes with pre-defined profiles
bwrap is as secure as you want it to be which I think is the primary advantage over anything else.
I like the idea of bubblewrap, but my pain point is that it is work to set it up correctly with bind mounts and forwarding necessary environment variables to make the program actually work usefully. Could you share your pip bwrap configuration? It sounds useful.
can't really share a file here, feel free to email me
This may not be popular, but is there a place for required human actions or just timed actions to slow down things like this? For instance, maybe a GH action to deploy requires a final human click and to change that to cli has a 3 day cooling period with mandatory security emails sent out. Similarly, you switch to read only for 6 hrs after an email change. There are holes in these ideas but the basic concept is to treat security more like physical security, your goal isn't always to 100% block but instead to slow an attacker for xxx minutes to give the rest of the team time to figure out what is going on.
Hi, security here. We've tried, but the amount of people you need for this vs the amount of people you have trying to review and click the big button always means that this step will be a bottleneck. Thus this step will be eliminated.
A much better approach would be to pin the versions used and do intentional updates some time after release, say a sprint after.
Yeah, I am looking at that on the use end. It sounds like on the python side this type of thing will be more standard (uv now and soon pip supported with version date requirements). I think time is a big missing element in many security in depth decisions. It can be time until you adopt like use no package newer than xx days or time it takes to deploy etc etc. Unfortunately the ecosystem is getting really diverse and that means ever more sophisticated attacks so we may need to do things that are annoying just to survive.
Yes, that's why I recommend intentional updates. Planning at least a sprint later gives you a week or two, hoping the community catches such issues.
Why not just release escrow? If I try to push a new release version another developer or developers have to agree to that release. In larger projects you would expect the release to be coordinated or scheduled anyways. Effectively we're just moving "version pinning" or "version delay" one layer up the release chain.
A lot of libraries are maintained by a single person.
Are those the ones typically involved in supply chain attacks?
There are no perfect solutions; but, let's be reasonable.
Actually, yes, they are the prime targets: https://en.wikipedia.org/wiki/Npm_left-pad_incident or seemingly https://en.wikipedia.org/wiki/XZ_Utils_backdoor as well.
xz has dozens of contributors and two active maintainers. It was the actual example I was thinking of. The code was submitted by a third party and not a result of a developer machine compromise.
left pad wasn't a security incident. It was a capitalism incident.
Pinning, escrowing, and trailing all help, but I'm not sure "this step will be eliminated" is inevitable.
Package manager ecosystems are highly centralized. npm.org could require MFA (or rate limit, or email verification, or whatever) and most packagers would gripe but go along with this. A minority would look for npm competitors that didn't have this requirement, and another minority would hack/automate MFA and remove the added security, but the majority of folks would benefit from a centralized requirement of this sort.
Let me rephrase - manual security verification is a velocity blocker. People won't do manual security verification of changes.
I agree that npm.org requiring MFA is a good idea in general and in this case.
Yup. As someone who's been on both the eng and security side, you cannot improve security by blocking the product bus. You're just going to get run over. Your job is to find ways of managing risk that work with the realities of software development.
And before anyone gets upset about that, every engineering discipline has these kind of risk tradeoffs. You can't build a bridge that'll last 5,000 years and costs half of our GDP, even though that's "safer". You build a bridge that balances usage, the environment, and good stewardship of taxpayer money.
lgtm (didn't read)
Min release age sucks, but we’ve been here before. Email attachments used to just run wild too, then everyone added quarantine delays and file blocking and other frictions... and it eventually kinda/sorta worked. This does feel worse, though, with fewer chokepoints and execution as a natural part of the expectation.
Edit: bottom line is installs are gonna get SOOO much more complicated. You can already see the solution surface... Cooling periods, maintainer profiling, sandbox detonation, lockfile diffing, weird publish path checks. All adds up to one giant PITA for fast easy dev.
Min release age might just postpone vulnerability to be applied few days later in non trivial cases like this. More I think about it, Odin lang approach of no package manager makes senses. But, for that approach won't work for Javascript as it needs npm package even for trivial things. Even vendoring approach like golang won't work with Javascript with the amount of churn and dependencies.
It does not _need_ it, that’s the thing. It has become a custom to import a dependency for a lot of things. Especially for JavaScript.
Coded has zero nom dependencies. Neat!
174025 dependents.
Please can we just have a 2FA step on publishing? Do we really need a release to be entirely and fully automated?
It won't stop all attacks but definitely would stop some of these
Genuinely how are you supposed to make sure that none of the software you have on your system pulls this in?
It’s things like this that make me want to swap to Qubes permanently, simply as to not have my password manager in the same context as compiling software ever.
This sounds like satire but isn't - I just make sure the nodejs/npm packages don't exist on my system. I've yet to find a crucial piece of software that requires it. As much as I love that cute utility that turns maps into ascii art, it's not exactly sqlite in terms of usefulness.
Bit ridiculous to dismiss the most popular programming languages packaging repo as silly toys.
I don't deny that node/npm is useful for building servers, devtools for JS development itself, etc. but as an end user I haven't encountered anything useful which requires having it on my machine.
Ok? So you don't code in that language?
You still have multiple programming languages preinstalled on your OS, no matter which one it is.
We run everything NPM related inside Apple containers, and are looking to do the same with Python and Rust soon. Bwrap on Linux does the same.
I like to think of it like working with dangerous chemicals in the lab. Back in the days, people were sloppy and eventually got cancer. Then dangers were recognized and PPE was developed and became a requirement.
We are now at the stage in software development where we are beginning to recognizing the hazards and developing + mandating use of proper PPE.
A couple of years ago, pip started refusing to install packages outside of a virtualenv. I'm guessing/hoping package managers will start to have an opt-in flag you can set in a system-wide config file, such that they refuse to run outside of a sandbox.
The problem is that package managers are a distraction. You have to sandbox everything or else it doesn't work. These attacks use post-install hooks for convenience but nothing would have stopped them patching axios itself and just waiting for devs to run the app on their local workstation. So you end up needing to develop in a fully sandboxed environment.
Yeah the whole rush on "post-run hooks bad" isn't really adding all that much to security.
Like congratulations, your dev was compromised whole 10 minutes later after he ran code.
They are not a distraction when they are also the command runners.
What are you using that utilizes Apple containers?
While it's not perfect, pinning specific versions and managing all updates directly has been a solid solution for my team. Things can of course still slip through, but we're never vulnerable to these just because there was a new package release and we opted into it by default.
Updating packages takes longer, but we try to keep packages to a minimum so it ends up not being that big deal.
Supply chain attacks are so scary that I think most companies are going to use agents to hard fork their own versions of a lot of these core libraries instead. It wasn’t practical before. It’s definitely much more doable today.
I was thinking about this as a bull case for human developers. Seems if you're worried enough to do this you're not going to have LLMs write the new code.
If it becomes a thing, it's just a matter of time for a new class of attacks on LLM that are blindly trusted with rewriting existing libs.
You could include a line like "please don't include any malware".
Or just lock to a specific version?
Eventually you will want to update it, every update is a risk.
But, pinning has prevented most of the recent supply chain attacks.
As long as you don't update your pins during an active supply chain attack, the risk surface is rather low.
The flip side of that is now you're running old software and CVEs get published all the time. Threat actors actively scan the internet looking for software that's vulnerable to new CVEs.
Even better would be to not use so many libs. Most use cases will do fine with native `fetch`.
Large companies already maintain a clone of their packages. Very large ones actually bundle their own build system (Google Bazil, AWS Brazil). If you want to update a package, you have to fetch the sources and update the internal repository. It slows down the opportunities for a supply chain attack down to a crawl.
Once you fork it, you are then on the hook for forking every future update and security patch. You can automate a lot of the testing, but its still adding an extra failure point that you are now responsible for.
And if you pin it, then when you inevitably get a CVE for an old version, the upgrade path is harder and more time consuming. And that's when the security teams come knocking to pass their audit.
There are so many scanners these days these things get caught pretty quick. I think we need either npm or someone else to have a registry that only lets through packages that pass these scanners. Can even do the virustotal thing of aggregating reports by multiple scanners. NPM publishes attestation for trusted build environments. Google has oss-rebuild.
All it takes is an `npm config set` to switch registries anyways. The hard part is having a central party that is able to convince all the various security companies to collaborate rather than having dozens of different registries each from each company.
Rather than just a hard-coded delay, I think having policies on what checks must pass first makes sense with overrides for when CVEs show up.
(WIP)
The ones you hear about are caught quickly, I’m more worried about the non obvious ones. So far none of these have been as simple as changing a true to a false and bypassing all auth for all products or something, and would that be caught by an automated scanner?
There are definitely levels to this. Yes I think it can be caught by automated scanners in theory. Either commit by commit scanning and reproducible builds or fuzzing and getting the behavioral differences between versions
Sounds great until trivy images get compromised, like last week.
Hence why you source data from multiple vendors I'd say. Rather than putting all eggs in one basket
Not to beat a dead horse but I see this again and again with dependencies. Each time I get more worried that the same will happen with rust. I understand the fat std library approach won’t work but I really still want a good solution where I can trust packages to be safe and high quality.
Hosting curated dependencies is a commercially valuable service. Eventually an economy arises where people pay vendors to vet packages.
It already exists; cloudsmith
It's what linux distributions do.
Queue appimage or other packed binary and there go your finetuned packages.
what?
Yes, that why those need to be 100% sandboxed by default (ideally a VM), unless they are provided by distro
Linux distros and BSD ports did that since the 90's. When Linux distros had barely a PM or just tarballs, Infomagic sold 4 CD full of libre software. When I had no internet at home, back in the day I bought 3 DVD's of Debian Sarge for 20 euros, about $20. A bargain, it was the price of a hard-cover best seller book.
GB's of libre software, graphical install, 2.6 kernel, KDE3 desktop, very light on my Athlon 2000 with 256MB of RAM. It was incredible compared to what you got with Windows XP and 120 Euro per seat. Nonfree software and almost empty.
And, well, if for instance I could get read only, ~16TB durable USB drive with tons of Guix packages offline (in a two yearly basis with stable releases) for $200 I would buy them in the spot.
You would say that $200 for a distro it's expensive, but for what it provides, if you are only interested in libre gaming and tools, they amount you save can be huge. I've seen people spend $400 in Steam games because of the Holyday sales...
If the fat std library is not viable you can only increase security requirements.
Axios has like 100M downloads per week. A couple of people with MFA should have to approve changes before it gets published.
This is the actual answer: stupid cost saving creating an operational risk.
At least then they will have to pay off a dev or something, changes their economic calculus and is additionally illegal
An alternative:
- copy the dependencies' tests into your own tests
- copy the code in to your codebase as a library using the same review process you would for code from your own team
- treat updates to the library in the same way you would for updates to your own code
Apparently, this extra work will now not be a problem, because we have AI making us 10x more efficient. To be honest, even without AI, we should've been doing this from the start, even if I understand why we haven't. The excuses are starting to wear thin though.
I don't know where you've worked but a hostile and intelligent actor or internal red team would succeed under each of those cases at every job I've worked at.
Good to know. Where were the places you worked at?
Defending against a targeted attack is difficult, yes. But these recent campaigns were all directed at everyone. Auditing and inspecting your dependencies does absolutely help thwart that because there will always be people who don't.
They succeeded in poisoning the whole supply chain and making everyone distrust package management to a degree never seen before, and people who aren't reviewing their dependencies are already getting hit. You seem to suggest that we all accept that.
That attitude might be the reason why the places you've worked would be under threat. The places I've worked would also be under threat, because several of my colleagues had that attitude, and this is why red teaming works.
Just going to put features on hold for a month while I review the latest changes to ffmpeg.
As you should. Also, the constant complaint from devs on these very boards is that quality and security are relegated behind new features that are often described as useless but pushed by management.
Are you in management?
Why wouldn't the "fat std" thing work? Yes it's hard to design properly, both in scope and actual design (especially for an unstandardized language still moving fast), but throwing the towel and punting the problem to the "free market" of uncurated public repos is even worse.
It's what we call in France "la fête du slip".
PS: that's one reason I try to use git submodules in my Common Lisp projects instead of QuickLisp, because I really see the size of my deptree this way.
Because fat std is rigid, impractical, and annoying.
In practice (e.g. Go) it’s actually pretty good and infinitely preferable to third party everything.
Works just fine in Go.
Yeah, it's annoying to have good support for dates in Java since 2014, instead of only getting it now like in JS.
I think we found the constituency that led to the present sorry situation.
That's rather rude.
If you're referring to my packages on npm, I joined way late to that game. This was also ~15 years ago.
Fat std library mistakes/warts would likely result in third party packages being used anyway.
So it provides another option, and in worst case it doesn't make situation worse than it is right now?
Yeah, pretty bad idea.
Not necessarily, but let's agree that some design faults would happen: you still get the option to use the solid, boring and slightly rusty std instead of another 100 dependencies from the supply chain supermarket.
At work, we're happy with Python's included batteries when we need to make scripts instead of large programs.
NPM should have a curation mechanism, via staff review or crowdsourcing, where versions of popular packages are promoted to a stable set, like linux distros do. I would only use curated versions if they had such a thing.
This is why Node.js is completely unsuitable as backend. Until recently, there wasn’t even a standard Promise-based HTTP client. Why should we need to download a library just to make a simple HTTP request? It’s because Node.js’s standard library is too limited, leading to an explosive growth in third-party libraries. As a result, it’s vulnerable to security attacks, and maintaining it in an enterprise environment becomes a major challenge. Let’s use .NET or Go. Why use JavaScript outside of the browser when there are excellent backend environments out there?
JS package managers (pnpm, bun) now will ignore postinstall scripts by default. Except for npm, it still runs them for legacy reasons.
You should probably set your default to not run those scripts. They are mostly unnecessary.
83M weekly downloads!
All these supply chain attacks make me nervous about the apps I use. It would be valuable info if an app used such dependencies, but on the other hand, programmers would cut their sales if they gave you this info.
This is just going to get worse and worse as agentic coding gets better. I think having a big dependency tree may be a thing of the past in the coming years. Seems like eventually new malware will be coming out so fast it will basically be impossible to stop.
PSA: Make sure to set a minimum release age and pin versions where possible.
The amount of people still using this instead of fetch. Nonetheless when wasn't axios, it would be something else.
This is why corporations doing it right don't allow installing the Internet into dev machines.
Yet everyone gets to throw their joke about PC virus, while having learnt nothing from it.
People are lazy. And sometimes they find old stuff via a google search and use that.
Axios has a long history, and is included in a lot of code, also in indirect dependencies. Just check its npm page: it has 174025 dependents as of this moment, including a lot of new packages (I see openclaw and mcp related packages in the list).
And with LLMs generating more and more code, the risk of copying old setups increases.
Package managers are a failed experiment.
We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
The issue is not dependencies themselves, it's transitive ones. Nobody installs left-pad or is-even-number directly, and "libraries" like these are the vast majority of the attack surface. If you get rid of transitive dependencies, you get rid of the need of a package manager, as installing a package becomes unzipping a few files into a vendor/ folder.
There's so many C libraries like this. Off the top of my head, SQLite, FreeType, OpenSSL, libcurl, libpng/jpeg, stb everything, zlib, lua, SDL, GLFW... I do game development so I'm most familiar with the ones commonly used in game engines, but I'm sure other fields have similarly high quality C libraries.
They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust, and C#/Java/JS/Python libraries are never used outside their respective language (aside form Java ones in other JVM langs).
If you're developing for the web your attack surface is quite a bit bigger. Your proposed solution of copying a few files might work but how do you keep track of updates? You might be vulnerable to a published exploit fixed a few months ago. A package manager might tell you a new version is available. I don't know how that would work in your scenario.
I'd really like to see package managers organized around rings where a very small core of incredibly important stuff is kept in ring 0, ring 1 gets a slightly wider amount of stuff and can only depend on ring 0 dependencies and then ring 2+ is the crapware libraries that infect most ecosystems.
But maybe that's not the right fit either. The world where package managers are just open to whatever needs to die. It's no longer a safe model.
Malicious actor KPI: affect a Ring 0 package.
In practice, "ring 0" is whatever gets merged into your language's standard library. Node and python both have pretty expansive standard libraries at this point, stepping outside of those is a choice
The OS distro model is actually the right one here. Upstream authors hate it, but having a layer that's responsible for picking versions out of the ecosystem and compiling an internally consistent grouping of known mutually-compatible versions that you can subscribe to means that a lot of the random churn just falls away. Once you've got that layer, you only need to be aware of security problems in the specific versions you care about, you can specifically patch only them, and you've got a distribution channel for the fixes where it's far more feasible to say "just auto-apply anything that comes via this route".
That model effectively becomes your ring 1. Ring 0 is the stdlib and the package manager itself, and - because you would always need to be able to step outside the distribution for either freshness or "that's not been picked up by the distro yet" reasons - the ecosystem package repositories are the wild west ring 2.
In the language ecosystems I'm only aware of Quicklisp/Ultralisp and Haskell's Stackage that work like this. Everything else is effectively a rolling distro that hasn't realised that's what it is yet.
Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
What is a problem is library quality. Which is downstream of nobody getting paid for it, combined with an optimistic but unrealistic "all packages are equal" philosophy.
OpenSSL is one of the ones where there's a ground up rewrite happening because the code quality is so terrible while being security critical.
On the other end, javascript is uniquely bad because of the deployment model and difficulty of adding things to the standard library, so everything is littered with polyfills.
Absolute nonsense. What does automated world even mean? Even if one could infer reasonably, it's no justification. Appealing to "the real world" in lieu of any further consideration is exactly the kind of mindlessness that has led to the present state of affairs.
Automation of dependency versions was never something we needed it was always a convenience, and even that's a stretch given that dependency hell is abundant in all of these systems, and now we have supply chain attacks. While everyone is welcome to do as they please, I'm going to stick to vendoring my dependencies, statically compiling, and not blindly trusting code I haven't seen before.
How do you handle updating dependencies then?
Relax, while mentioning the real world without any criticism for the soundness of the solution is absolute nonsense, some would say idiotic, thinking only in the absolute best solution given your narrow world view is not any better.
While I agree that my view is narrow, the "best solution" in question is what we used to do, and it was fine. There are still many places that manually manage dependencies. Fundamentally automatic software versioning is an under-developed area in need of attention, and technologies like semantic versioning which are ubiquitous are closer to suggestions, and not true indicators of breaking changes. My personal view is that fully automatic dependency version management is an ongoing experiment and should be treated as such.
People are trying to automate the act of programming itself, with AI, let alone all the bits and pieces of build processes and maintenance.
For some reason, NPM is the only ecosystem with substantial issues with supply-chain attacks.
apart from that python one the other day
Popularity
The number of issues is disproportionately larger than the one for Debian.
Debian is slower so npm is more attractive
The culture within the npm/js community has mainly been one of using the package manager rather than "re-inventing the wheel", as such the blast radius of a compromised package is much greater
It’s not just NPM, though. Every Rails project and every Rust project I’ve seen ended up with massive numbers of dependencies vs what an equivalent project in Go or C# would have needed.
CPAN too, just try Hailo under Perl to test an old-fashioned chatbot based on Markov chains where very small LLM's and Hailo converge if used with the advanced training options for it. Yes, it will pull tons of dependencies, (less with cpanminus if run with 'cpanm -n Hailo'), but contrary to NPM, Pip and the like CPAN's repos are highly curated and before PHP and ubiquitoous Python Perl was used everywhere, from a sysadmin language (better than Bash/Sh for sure) to CGI, IRC bots and whatnot. How many issues did we have? Zero or near zero.
It's more to do with the standard library being so barren of common application needs, and looking for a solution that the community has gotten behind. Axios has been a common dependency in many codebases, because it is a solid solution that many have already used. Every developer could try building all the libraries that they would reach for themselves, but then each company has now taken on the task of ensuring their own (much larger) codebase is free from security issues, on top of taking care of their own issues and bugs.
It is because it has the lowest barrier to entry with no quality control. Ever.
This is what happens when there is no barrier to entry and it includes everyone who has no idea what they are doing in charge of the NPM community.
When you see a single package having +25 dependencies, that is a bad practice and increases the risk of supply chain attacks.
Most of them don't even pin their dependencies and I called this out just yesterday on OneCLI. [0]
It just happens that NPM is the worst out of all of the rest of the ecosystems due to the above.
[0] https://news.ycombinator.com/item?id=47577183
no no, please we don't want to get back to dragging files to your project to make them work.
And manual FTP uploads, while we're at it.
You don’t need to. Llm will do it for you and include the security bug
Rust libraries are infrequently used outside of Rust because if you have the option, you'd just use Rust, not the ancient featureless language intrinsically responsible for 70% of all security issues. C libraries are infrequently used in Rust outside of system libc, for the same reason; I go and toggle the reqwest switch to use rustls every time, because OpenSSL is horrendous. This is also why you say 'rarely' instead of 'never', when a few years ago it was 'never'; a few years from now you'll say 'uncommonly', and so on. The reason C libraries are used is because you don't feel like reimplementing it yourself, and they are there; but that doesn't apply more to C libraries than Rust libraries, and the vast majority of crates.io wouldn't be usefully represented in C anyway, or would take longer to bind to than to rewrite. (No, nobody uses libcurl.) Finally, this only happens in NPM, and the Rust libraries you pull in are all high-quality. So this sounds like a bunch of handwaving about nonsense.
Rust is terrible for pulling in hundreds of dependencies though. Add tokio as a dependency and you'll get well over 100 packages added to your project.
pin-project-lite is the only base dependency, which itself has no dependencies. If you enable the "full" feature, ie all optional doodads turned on (which you likely don't need), it's 17: bytes, cfg-if, errno, libc, mio, parking_lot+parking_lot_core+lock_api, pin-project-lite, proc_macro2+quote+syn+unicode-ident, scopeguard, signal-hook-registry, smallvec, and socket2. You let me know which ones you think are bloat that it should reimplement or bind to a C library about, and without the blatant fabrication this time.
Even side stepping that tokio no longer pulls multiple packages, it used to be split into multiple packages, in the same way that KDE in Rust would be hundreds of packages.
Rust projects tend to take their project and split it into many smaller packages, for ease of development, faster compiles through parallelization, ensuring proper splitting of concerns, and allowing code reuse by others. But the packages are equivalent to a single big package. The people that write it are the same. They get developed in tandem and published at the same time. You can take a look at the del tree for ripgrep, and the split of different parts of that app allows me to reuse the regex engine without dealing with APIs that only make sense in the context of a CLI app or pulling in code I won't ever use (which might be hiding an exploit too).
Counting 100 100 line long crates all by the same authors as inherently more dangerous than 1 10000 line long crate makes no sense to me.
It's worth noting that Rust packages (crates) are all single compilation units, and every compilation unit is a package. It's the equivalent of complaining that OpenSSL pulls in hundreds of `.c` files.
I don't think this community of professionals is going to come around to a solution which requires marginally more effort.
If no one checks their dependencies, the solution is to centralize this responsibility at the package repository. Something like left-pad should simply not be admitted to npm. Enforce a set of stricter rules which only allow non-trivial packages maintained by someone who is clearly accountable.
Another change one could make is develop bigger standard libraries with all the utilities which are useful. For example in Rust there are a few de facto standard packages one needs very often, which then also force you to pull in a bunch of transitive dependencies. Those could also be part of the standard library.
This all amounts to increasing the minimal scope of useful functionality a package has to have to be admitted and increasing accountability of the people maintaining them. This obviously comes with more effort on the maintainers part, but hey maybe we could even pay them for their labor.
You are just swapping a package manager with security by obscurity by copy pasting code into your project. It is arguably a much worse way of handling supply chain security, as now there is no way to audit your dependencies.
This argument makes no sense. Obviously reducing the amount of transitive dependencies is almost always a good thing, but it doesn't change the fundamental benefits of a package manager.
The language with the most fundamental and dangerous ways of handling memory, the language that is constantly in the news for numerous security problems even in massively popular libraries such as OpenSSL? Yes, definitely copy-paste that code in, surely nothing can go wrong.
This is a WILD assumption, doing C-style bindings is actually quite common. YOu will of course then also be exposing a memory unsafe interface, as that is what you get with C.
What exactly is your argument here? It feels like what you are trying to say is that we should just stop doing JS and instead all make C programs that copy paste massive libraries because that is somhow 'high quality'.
This seems like a massively uninformed, one-sided and frankly ridiculous take.
You should try writing code, and not relying on libraries for everything, it may change how you look at programming and actually ground your opinions in reality. I'm staring at company's vendor/ folder. It has ~15 libraries, all but one of which operate on trusted input (game assets).
I literally told you why they don't matter if you write code in a sane way.
I know bindings for Rust libraries exist. Read the literal words you quoted. "Rust libraries are very rarely used outside of Rust". Got some counterexamples?
https://github.com/memflow/memflow
https://github.com/PyO3/pyo3
https://slint.dev/
https://github.com/dora-rs/dora
It is VERY common in existing codebases that are migrating from C++/C to make heave use of FFI/ existing C
Gamedev is its own weird thing, and isn't a model you want to generalize to other industries. It has to optimize for things a lot of software does not, and that skews development.
Vendoring libraries is almost always a terrible idea because it immediately starts to bitrot and become a footgun.
Sometimes it's necessary, but it's not desirable, and you almost always just want to pin your dependencies instead.
I think you can do copy paste in most languages. But it will be a pain to update when there are improvements / security fixes.
You got a project with 1-2 depencies? Sure. But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs) good luck.
So don’t?
With manual deps management, everyone soon gravitates to a core set of deps. And libraries developer tends to reduce their deps needs, That’s why you see most C libraries deals with file formats, protocols, and broad concerns. Smaller algorithms can be shared with gists and blog articles.
You just invented a worse Stack Overflow.
Using libraries is good, actually.
Like is-even or leftpad?
Like requests and pytest and ruff and so on, yes.
Rewriting the world to protect against a specific kind of threat is insane.
It's not there isn't good libraries everywhere. It just the practice around NPM that people are appalled with. It's all about lowering the barrier for developers, even in spite of security and quality.
I'm not sure why you believe this is more secure than a package manager. At least with a package manager there is an opportunity for vetting. It's also trivial that it did not increase your executable's size. If your executable depends on it, it increases its effective size.
Package managers are older than some users here. From CPAN/CTAN to ports under BSD's.
Some pm's are badly maintained (Pip/NPM), while others are curated enough.
Again, if you have GNU/Linux installed, install Guix, read the Info manual on 'guix import' and just create a shell/container with 'guix shell --container' (and a manifest package created from guix import) and use any crap you need for NPM in a reproducible and isolated way. You $HOME will be safe, for sure.
They're not a failed experiment. No one has ever "experimented" by making a safe package manager for their new language. And it is not that insane to do so. Very basic things will get you very far:
1. Packages should carry a manifest that declares what they do at build time, just like Chrome extensions do. This manifest would then be used to configure its build environment.
2. Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
3. Every package registry should produce a detailed audit log that contains a "who, what, when". Every build/ command should be producing audit logs that can be collected by endpoint agents too.
4. Every package registry should support TUF.
5. Typosquatting defenses should be standard.
etc etc etc. Some of this is hard, some of this is not hard. All of this is possible. No one has done it, so it's way too early to say "package managers can't be made safe" when no one has tried.
Love these ideas!
How is this enforced when it's pushed via a pipeline?
Your account is separate from your publishing. That is, in order to go to my account to change configuration values, 2FA must be required.
Publishing should be handled via something like Trusted Publishing, which would leverage short lived tokens and can integrate with cryptographic logs for publish information (ie: "Published from the main branch of this repo at this time").
I don't understand commercial aspect of large OSS like package managers but i was wondering for years why this was missing from npm. I think typosquatting was handled by npm last year but only after some popular miss typed packages started stealing developer creds.
The people building package managers are unaware of these problems going into it and it becomes extremely disruptive to start adding these things later on since your entire ecosystem is built on the assumption that they can do these things.
It's also shockingly controversial to suggest typosquatting suggestions. I made this suggestion ages ago for cargo, demonstrated that basic distance checks would have impacted <1% of crates over all time, and people still didn't want it.
For those who didn't know what TUF means (like me), I think they're referring to The Update Framework (https://theupdateframework.io).
Sorry, I should have clarified that - you're correct. `cosign` is an example of a tool that makes this quite straightforward and proves that this sort of system can work today.
Then you've got ecosystems like Clojure where many projects are just considered done and used by many. You can pin these (and be warned if a new version still comes out, say for an actual security fix). There are Clojure projects so stable, without any know exploit (we're certainly not talking about daily npm exploits here), that haven't been updated in years because they are... Done. Simply done. Perfection.
Something to reflect upon too.
SQLite had 6 CVE's last year. FreeType had an RCE bug published last year. libcurl's last CVE reported was 2 weeks ago. Libpng had 4 vulnerabilities published this year....
Thought experiment: a language that enforces dependencies to be 2 levels deep or less.
I wonder if dependencies would flatten out with this limitation in mind.
NPM gets worse than russian roulette. Perhaps we have to rename russian roulette to node roulette: noulette.
Running almost anything via npx will trigger this
npx is just a bad command to use.
And yet so many "Installation" pages will recommend it!
Just sanity checking - if I only ever install axios in a container that has no secrets mounted in to its env, is there any real way I can get pwned by this kind of thing?
Yes. Docker breakout is a class of vulnerabilities into itself.
Essential steps to minimise your exposure to NPM supply chain attacks:
— Run Yarn in zero-installs mode (or equivalent for your package manager). Every new or changed dependency gets checked in.
— Disable post-install scripts. If you don’t, at least make sure your package manager prompts for scripts during install, in which case you stop and look at what it’s going to run.
— If third-party code runs in development, including post-install scripts, try your best to make sure it happens in a VM/container.
— Vet every package you add. Popularity is a plus, recent commit time is a minus: if you have this but not that, keep your eyes peeled. Skim through the code on NPM (they will probably never stop labelling it as “beta”), commit history and changelog.
— Vet its dependency tree. Dependencies is a vector for attack on you and your users, and any new developer in the tree is another person you’re trusting to not be malicious and to take all of the above measures, too.
Idk, lockfiles provide almost as good protection without putting the binaries in git. At least with `--frozen-lockfile` option.
Exactly. Yarn uses a yarn.lock file with the sha256 hashes of each npm package it downloads from the repo (they are .tgz files). If the hash won't match, install fails. No need to commit the dependencies into your git.
Zero-installs mode does not replace the lockfile. Your lockfile is still the source of truth regarding integrity hashes.
However, it’s an extra line of defence against
1) your registry being down (preventing you from pushing a security hotfix when you find out another package compromised your product),
2) package unpublishing attacks (your install step fails or asks you to pick a replacement version, what do you do at 5pm on a Friday?), and
3) possibly (but haven’t looked in depth) lockfile poisoning attacks, by making them more complicated.
Also, it makes the size of your dependency graph (or changes therein) much more tangible and obvious, compared to some lines in a lockfile.
Number 1 would only be a win for zero-installs if it happened that registry was up when you made the security hotfix, since you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy. Seems like a highly unlikely scenario to me. Also, cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Most npm CVEs are stuff like DDoS vulnerabilities, and you should have mitigations for those in place for at the infra-level anyway (e.g. request timeouts, rate limits, etc), or you are pretty much guaranteed to be cooked sooner or later anyway. The really dangerous stuff like arbitrary command execution from a library that takes end user input is much much more rare. The most recent big one I remember is React2shell.
Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
I can see how zero-installs are useful under some specific constraints where you want to minimize dependencies to external services, e.g. when your CI runs under strict firewalls. But for most, nah, not worth it.
Which dependency? It sounds like you are assuming some specific scenario, whereas the fix can take many forms. In immediate term, the quickest step could be to simply disable some feature. A later step may be vendoring in a safe implementation.
The registry doesn’t need to be actually down for you, either; the necessary condition is that your CI infrastructure can’t reach it.
Not sure what you mean by “npm CVEs”. The registry? The CLI tool?
As I wrote, if you are running compromised software in production, you want to fix it ASAP. In first moments you may not even know whether bad things will happen or not, just that you are shipping malicious code to your users. Even if you are lucky enough to determine with 100% confidence (putting your job on the line) that the compromise is inconsequential, you don’t want to keep shipping that code for another hour because your install step fails due to a random CI infra hiccup making registry inaccessible (as happened in my experience at least half dozen times in years prior, though luckily not in a circumstance where someone attempted to push an urgent security fix). Now imagine it’s not a random hiccup but part of a coordinated targeted attack, and somehow it becomes something anticipated.
Those rare conditions exist. Also, you are making it sound as if the registry is infallible, and no humans and/or LLMs there accept untrusted input from their environment.
The key aspect of modern package managers, when used correctly, is that even when the registry is compromised you are fine as long as integrity check crypto holds up and you hold on to your pre-compromise dependency tree. The latter is not a technical problem but a human problem, because conditions can be engineered in which something may slip past your eyes. If this slip-up can be avoided at little to no cost—in fact, with benefits, since zero-installs shortens CI times, and therefore time-to-fix, due to dramatically shorter or fully eliminated install step—it should be a complete no-brainer.
As I wrote, I suspect it’d complicate such attacks or make them easier to spot, not make them impossible.
This is why Artifactory and similar exist and they do this better. You ~never want to vendor libraries.
Are you saying it replaces my package manager, or that I should add another tool to my stack, vet yet another vulnerable dependency for critical use, to do something my package manager already does just as well?
I just explained why you should, and you are yet to provide a counter-argument.
Good points. But what do you mean with 3: "lockfile poisoning attacks, by making them more complicated" — making the lockfiles more complicated?
Also, 4) Simpler to `git diff` the changes, when you have the source locally already :- )
Poor phrasing; I meant the attacks. Now you don’t just have a lockfile you need to sneakily modify, and the diff grows.
As to your second point, yes. It’s really a different feeling when you add one more package and suddenly have 215 new files to check in!
How big is your repo, if I may ask?
Personally I store vendored dependencies in a submodule, where I can squash history, if it grows too large.
It’s a subjective question, but in one of the zero-installs projects I definitely remember that when I added a couple of particular GUI libraries there suddenly a very, very long list of new files to track, since those maintainers to keep things decoupled. I wouldn’t stop using that library at that point (there were deadlines), but I would definitely try to find something lighter or more batteries-included next time.
There can be a tiny project with just one dependency that happens to have an overgrown, massive graph of further transitive dependencies (a very unpleasant scenario which I would recommend to avoid).
With zero installs turned on, such a codebase could indeed qualify as “big repo”, which I think would reflect well its true nature.
Without zero installs it could be tiny but with a long long lockfile that nobody really checks when committing changes.
I don’t like the added mental overhead of submodules, and so prefer to avoid them when possible, which I guess is a subjective preference.
Since this is, coneptually speaking, the matter of package management more so than it is the matter of version control in genetal, I prefer to rely on package manager layer to handle this. I can see how your approach could make sense, but honestly I would be more anxious about forgetting something when keeping vendored dependencies up-to-date in that scenario.
Your approach could be better in a sense that you can spot-check not just the list of changed packages, but also the actual code (since you presumably vendor them as is, while Yarn checks in compressed .tgz files). Not sure whether that justifies the added friction.
In light of these nonstop supply chain attacks: Tonight I created /supply-chain-audit -- A simple claude code skill that fetches info on the latest major package vulnerability, then scans your entire ~/ and gives you a report on all your projects.
https://github.com/IsaacGemal/claude-skills
It's a bit janky right now but I'd be interested to hear what people think about it.
Skills are great attack vector as well.
That sounds terrifying. Stay out of my ~/ thank you very much.
One paragraph is written two times??
The NPM ecosystem is a joke. I don't even want anything to do with it, because my stack is fully Elixir. But, just because of this one dependency that is used in some interfaces within my codebase, I need to go back to all my apps and fix it. Sigh.
JavaScript, its entire ecosystem is just a pack of cards, I swear. What a fucking joke.
The attacker went through the hassle to compromise a very widely used package, but use a non standard port (8000) on their C2... If you plan to do something like that, use 443 at least, many corporate network do not filter this one ;)
I wonder if this has any connection with the recent string of attacks including the FBI director getting hacked. The attack surface is large, executed extremely cleanly - almost as if done by a high profile state sponsored actor, just like in Hollywood movies.
I'm working on a multi signature solution that helps to detect unauthorized releases in the case of an account hijack. It is open source, self hostable, accountless and I am looking for feedback!
Website: https://asfaload.com/
GitHub:https://github.com/asfaload/asfaload
Spec: https://github.com/asfaload/spec
I have a few projects which rely on npm (and react) and every few months I have to revisit them to do an update and make sure they still build, and I am basically done with npm and the entire ecosystem at this point.
Sure, its convenient to have so much code to use for basic functionality - but the technical debt of having to maintain these projects is just too damn high.
At this point I think that, if I am forced to use javascript or node for a project, I reconsider involvement in that project. Its ecosystem is just so bonkers I can't justify the effort much longer.
There has to be some kind of "code-review-as-a-service" that can be turned on here to catch these things. Its just so unproductive, every single time.
I am glad I don't need to touch JS or web dev at all.
Now, I tend to use Python, Rust and Julia. With Python I am constantly using few same packages like numpy and matplotlib. With Rust and Julia, I try as much as possible to not use any packages at all, because it always scares me when something that should be pretty simple downloads half of the Internet to my PC.
Julia is even worse than Rust in that regard - for even rudimentary stuff like static arrays or properly namespaced enums people download 3rd party packages.
It's mind boggling when a simple Rust app pulls in Serde and with it half a black hole worth of packages to serialize some mundane JSON.
Isn't Rust just as susceptible to this issue? For example, how do you deal with Rust's lack of support for HTTP in the standard library? Importing hyper pulls in a couple dozen transitive libraries which exposes you to the exact same kind of threats that compromised axios.
Given how HTTP is now what TCP was during the 90s and almost all modern networked applications needing to communicate in it one way or another, most rust projects come with an inherent security risk.
These days, I score the usability of programming languages by how complete their standard library is. By that measure, Rust and Javascript get an automatic F.
It is, therefore I have stated I avoid any dependencies while writing Rust, unless they are self-contained. And I said I am glad I don't do web, so I don't have need for HTTP implementations.
I believe compartmentalized operating systems like Qubes are the future for defending against these kinds of attacks.
Storing your sensitive data on a single bare-metal OS that constantly downloads and runs packages from unknown maintainers is like handing your house key out to a million people and hoping none of them misuse it.
i am rolling back a huge number of 'features' in my personal pc and going back to extremely miminal setups
the security solution i have is where it needs to become more simple, getting rid of attack surface that is coming out of these bloated releases
The frustrating thing here is that axios versions display on npmjs with verified provenance. But they don’t use trusted publishing: https://github.com/axios/axios/issues/7055 - meaning the publish token can be stolen.
I wrongly thought that the verified provenance UI showed a package has a trusted publishing pipeline, but seems it’s orthogonal.
NPM really needs to move away from these secrets that can be stolen.
I am now migrating all my unencrypted secrets on my machines to encrypted ones. If a tool supports scripted credential providers (e.g. aws-cli or Ansible), I use that feature. Otherwise, I wrap the executable with a script that runs gpg --decrypt and injects an environment variable.
That way, I can at least limit the blast radius when (not if) I catch an infostealer.
In case it helps, we open-sourced a tool to audit dependencies for this kind of supply-chain issue. The motivation was that there is a real gap between classic “known vulnerability” scanning and packages whose behavior has simply turned suspicious or malicious. We also use AI to analyze code and dependency changes for more novel or generic malicious behavior that traditional scanners often miss.
Project: https://point-wild.github.io/who-touched-my-packages/
Hopefully desktop Linux users will start to understand that malware actually does exist for Linux and that their operating system is doing nothing to protect them from getting RATed.
What do you mean?
Linux has the most powerful native process isolation arsenal at the user disposal.
And some distros use even more isolation mechanisms on top of the ones provided by the kernel like snap and flatpak.
And then you can recreate the entire thing like a spellbook with nix.
Docker works natively in it. Do I need to say more?
Linux is a decade ahead here with regards for security options available to the user.
Yet npm isn't using them allowing this RAT to work. It is not secure by default. It requires every app to manually opt in to being secure. This opt in approach to security puts desktop Linux decades behind in regards to security. Not ahead.
Linux is not making anything less secure than other OSs.
In fact it even gives the user more security tools.
So I fail to reason on you singling out Linux here.
Take for example iOS and Android. All apps are sandboxed by default. You can't make a program that just steals all of your credentials like you can on desktop Linux. Having security tools means nothing if they aren't being used.
No one is running npm in Android or iOS.
A more apt comparison is vs Windows and macOS.
And Linux offer more than these two with regards to security.
They aren't because Node haven't developed a Node app for them. Desktop Linux does not offer more security than macOS. macOS has proper security around stuff like apps using the mic and camera.
Android is running Linux...
And they added a lot of code to make it safe for users to install and use apps in general.
Is this an issue for those only using axios on the frontend side like in a VueJS app?
Absolutely. If you ever did a npm install on a project using one of the affected axios versions, your entire system may be compromised.
I strongly recommend you read the entire article.
Good morning, or as they say in the NPM world, which package got compromised today?
I lost respect for Axios when they made a breaking change in a patch release. Digging into the root cause, I found the maintainer had approved an outside PR with an obvious AI slop PR description: https://github.com/axios/axios/issues/7059
Looks like the maintainer wasn't just careless when reviewing PRs.
That maintainer (also the one whose creds got stolen) also has an obvious chatgpt slop profile picture on github.
True, but think of how many new successfully closed PRs will be created as a result of this ~~incident~~ opportunity. Exponential KPI growth!
A command to recursively check for the compromised axios package version:
Or more simply:
Let’s not encourage people to respond to security incidents by… copy/pasting random commands they don’t understand.
What’s with all those escapes codes?
script kiddies love their ANSI color codes and fancy ASCII art
So the root cause was again a developer's opsec. For improving things, I haven't seen many new initiatives on that side (beyond 2FA, but even that seems unenforced in these repositories, I reckon).
To have an initial smoke test, why not run a diff between version upgrades, and potentially let an llm summarise the changes? It’s a baffling practice that a lot of developers are just blindly trusting code repos to keep the security standards. Last time I installed some npm package (in a container) it loaded 521 dependencies and my heart rate jumped a bit
Is this the first time you have ever thought about the idea of supply chain attacks? This is the first thought 90% of people have and it doesn't work. Too much work to manually verify diffs and LLMs aren't good enough at this yet.
No, I think about it all the time. It’s just baffling that this kind of attack is still a thing, after a decade+ of this happening over and over again.
Why is it baffling? It's like saying "why do we still have outages". Well, yes.
Default setting latest should be caught in every static code scanner. How many times has this issue been raised.
npm really needs to provide a options to set individual packages to only be publishable via trusted publishing.
Glad to be using native fetch.
"Batteries included" ecosystems are the only persistent solution to the package manager problem.
If your first party tooling contains all the functionality you typically need, it's possible you can be productive with zero 3rd party dependencies. In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography. These are beacons for attackers. Everything depends on this stuff so the motivation for attacking these common surfaces is high.
I can literally count on one hand the number of 3rd party dependencies I've used in the last year. Dapper is the only regular thing I can come up with. Sometimes ScottPlot. Both of my SQL providers (MSSQL and SQLite) are first party as well. This is a major reason why they're the only sql providers I use.
Maybe I am just so traumatized from compliance and auditing in regulated software business, but this feels like a happier way to build software too. My tools tend to stay right where I left them the previous day. I don't have to worry about my hammer or screw drivers stealing all my bitcoin in the middle of the night.
This is a rather superlative and tunnel vision, "everything is a nail because I'm a hammer" approach. The truth is this is an exceedingly difficult problem nobody has adequately solved yet.
I think the AI tooling is, if not completely solving sandboxing, at least making the default much better by asking you every time they want to do something and providing files to auto-approve certain actions.
Package managers should do the same thing
Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
Tell me about it. Using AI Chatbots (not even agents), I got a MVP of a packaging system[1] to my liking (to create packages for a proprietary ERP system) and an endpoint-API-testing tool, neither of which require a venv or similar to run.
------------------------------
[1] Okay, all it does now is create, sign, verify and unpack packages. There's a roadmap file for package distribution, which is a different problem.
Really? I thought 'asking you every time they want to do something' was called 'security fatigue' and generally considered to be a bad thing. Yes you can concatenate files in the current project, Claude.
Yes it has to be combined with a robust way to allowlist actions you trust
Oddly, since I wrote that Claude 'auto' mode just landed and I built something with it (instead of 'dangeously skip') and it's working.
Fully agree with this! I think today .NET is probably the most batteries included platform you can get. This means that even if you use third-party libraries, these typically depend only on first-party dependencies, making it much less likely for something shady to sneak in.
To me, I really like Golang's batteries included platform. I am not sure about .NET though
C#'s LINQ (code as data, like LISP) wins over golang for any type of data access. Strongly-typed, language-native queries. Go has its own advantages though.
EF is amazing
And now with NativeAOT, you can use C# like go - you don't need to ship the CLR.
With the notable exception of cross-platform audio.
Not really notable, aiui the only mainstream language with anything like that is JS in the browser
And for good reason. There are enough platform differences that you have to write your own code on top anyway.
and cross-platform UI
Kinda.
With Bun I use less dependencies from NPM than I used from Nuget with .NET to build minimal apis. For example the pg driver.
Why is .NET more "batteries included" than Java?
The other thing that keeps coming up is the github-code-is-fine-but-the-release-artifact-is-a-trojan issue. It really makes me question if "packages" should even exist in JavaScript, or if we could just be importing standard plain source code from a git repo.
I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
Why wouldn't that work well with legacy projects? In fact, the projects I was a part of that I'd call legacy nowadays, was in fact built by copy-and-pasting .js libraries into a "vendor/" directory, and that's how we shipped it as well, this was in the days before Bower (which was the npm of frontend development back in the day), vendoring JS libs was standard practice, before package managers became used in frontend development too.
Not sure why it wouldn't work, JavaScript is a very moldable language, you can make most things work one way or another :)(
`vendor/` folders give me the worst developer PTSD :p
6 conflicting versions of jquery, and you know every single one of them was monkey patched, cemented into the codebase forever.
This might make things worse not better.
Yes - the postinstall hook attack vector goes away. You can do SHA pinning since Git's content addressing means that SHA is the hash of the content. But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
This is basically what Deno's import maps tried to solve, and what they ended up with looked a lot like a package registry again.
At least npm packages have checksums and a registry that can yank things.
The irony is that this is actually the current best practice to defend against supply chain attacks in the github actions layer. Pin all actions versions to a hash. There's an entire secondary set of dev tools for converting GHA version numbers to hashes
You can just git submodule in the dependencies. Super easy. Also makes it straightforward to develop patches to send upstream from within your project. Or to replace a dependency with a private fork.
In my experience, this works great for libraries internal to an organization (UI components, custom file formats, API type definitions, etc.). I don't see why it wouldn't also work for managing public dependencies.
Plus it's ecosystem-agnostic. Git submodules work just as well for JS as they do for Go, sample data/binary assets, or whatever other dependencies you need to manage.
This is where attestation/sigstore comes into play. Github has a first-party action for it and I wish more projects would use it. Regarding javascript specifically, I believe npm has builtin support for sigstore.
* https://docs.github.com/en/actions/concepts/security/artifac...
* https://www.sigstore.dev/
* https://github.com/actions/attest
What kind of apps do you build / industry etc?
Honestly, you can get pretty far with just Bun and a very small number of dependencies. It’s what I love most about Bun. But, I do agree with you generally. .NET is about as good as I’ve ever seen for being batteries included. I just hate the enterprisey culture that always seems to pervade .NET shops.
I agree about the culture. If I take my eye off the dev team for too long, I'll come back and we'll be using entity framework and a 20 page document about configuring code cleanup rules in visual studio.
Entity framework is pretty good.
Unless you are Python, where the standard library includes multiple HTTP libraries and everyone installs the requests package anyways.
Few languages have good models for evolving their standard library, so you end up with lots of bad designs sticking around forever. Libraries are much easier to evolve, giving them the advantage in terms of developer UX and performance.
I'm pretty sure it's really one HTTP library: urllib.request is built on top of http.client. But the very Java-inspired API for the former is awful.
What type of developer chooses UX and performance over security? So reckless.
I removed the locks from all the doors, now entering/exiting is 87% faster! After removing all the safety equipment, our vehicles have significantly improved in mileage, acceleration and top speed!
Better developer UX can directly lead to better safety. "You are holding it wrong" is a frequent source of security bugs, and better UX reduces the ways you can hold it wrong, or at least makes you more likely to hold it the right way
Friction is helpful. Putting seatbelts on takes more time than just driving, but it’s way safer for the driver. Current dev practices increase speed, not safety.
Depends. If you had to add to a Makefile for your dependencies, you sure as hell aren't going to add 5k dependencies manually just to get a function that does $FOO; you'd write it yourself.
Now, with AI in the mix, there's fewer and fewer reasons to use so many dependencies.
Initially I assumed this is sarcastic, but apparently not. UX and performance is what programmers are paid to do! Making sure UX is good is one of the most important things in programmer job.
While security is a moving target, a goal, something that can never be perfect, just "good enough" (if NSA wants to hack you, they will). You make it sound like installing third party packages is basically equivalent to a security hole, while in practice the risk is low, especially if you don't overdo it.
Wild to read extreme security views like that, while at the same time there are people here that run unconstrained AI agents with --dangerous-skip-confirm flags and see nothing wrong with it.
Installing 3rd party packages the way Node and Python devs do regularly _is_ a security hole.
We definitely agree on that. Fortunately some of the 600+ comments here include suggestions of what to do about it.
Even more wild to read that sarcasm about "removing locks from doors for 87% speedup" is considered extreme...
And yes, we agree that running unconstrained AI agents with --dangerous-skip-confirm flags and seeing nothing wrong with it is insane. Kind of like just advertising for burglars to come open your doors for you before you get home - yeah, it's lots faster to get in (and to move about the house with all your stuff gone).
"Security" is often more about corporate CYA than improving my actual security as a user, and sometimes in opposition, and there is often blatant disregard for any UX concession at all. The most secure system is fully encrypted with all copies of the encryption key erased.
The amount of time defining same data structures over and over again vs `pip install requests` with well defined data structures.
requests should be in the Python standard library. Hard choices need to be made.
Can you name some examples?
Personally I've heard Odin [1] to do a decent job with this, at least from what I've superficially learned about its stdlib and included modules as an "outsider" (not a regular user). It appears to have things like support for e.g. image file formats built-in, and new things are somewhat liberally getting added to core if they prove practically useful, since there isn't a package manager in the traditional sense. Here's a blog post by the language author literally named "Package Managers are Evil" [2]
(Please do correct me if this is wrong, again, I don't have the experience myself.)
[1] https://pkg.odin-lang.org/
[2] https://www.gingerbill.org/article/2025/09/08/package-manage...
Scala could be one example? When I upgraded to a newer version of the standard library (the Scala 2.13 or Scala 3 collections library), there was a tool, Scalafix [1], that could update my source code to work with the new library. Don't think it was perfect (don't remember), but helpful.
[1] https://scalacenter.github.io/scalafix/
For a lot of code, I switched to generating code rather than using 3rd party libraries. Things like PEG parsers, path finding algorithms, string sanitizers, data type conversion, etc are very conveniently generated by LLMs. It's fast, reduces dependencies, and feels safer to me.
Ah, so you've traded the possibility of bad dependencies for certainty.
Remember, our objective function here is “feels safe.”
How can you come to that conclusion, given the specific examples I have given, which are tedious to write, but easy to proof-read and test?
Because AI threatens the identity of many programmers.
Lol. My most recent comment before this one is here: https://news.ycombinator.com/item?id=47583593. You judge if AI threatens my identity. But hey, don't let the facts get in the way of a slick narrative.
Or find the best third party library and copy the code from a widely used version that has been out long enough to have been well tested into your source tree.
The problem is not third party libraries. It is updating third party libraries when the version you have still works fine for your needs.
Don't do this. Use a package manager that let's you specify a specific version to pin against. Vendoring side steps most automated tooling that can warn you about vulnerabilities. Vendoring is a signal that your tooling is insufficient, 99% of the time.
Vendoring means you don't have to fetch the internet for every build, that you can work offline, that you're not at the mercy of the oh-so-close-99.999 availability, that it will keep on working in 10 years, and probably other advantages.
If your tooling can pull a dependency from the internet, it could certainly check if more recent version from a vendored one is available.
Is there any package manager incapable of working offline?
I think you've identified the problem here: package management and package distribution are two different problems. Both tools have possibilities for exploits, but if they are separate tools then the surface area is smaller.
I'm thinking that the package distribution tool maintains a local system cache of packages, using keys/webrings/whatever to verify provenance, while the package management tool allows pinning, minver/maxver, etc.
This is only true if you aren’t internally mirroring those packages.
Most places I’ve worked have Artifactory or something like it sitting between you and actual PyPI/npm/etc. As long as someone has pulled that version at some point before the internet goes out, it’ll continue to work after.
And this is exactly why we see noise on HN/Reddit when a supply-chain cyberattack breaks out, but no breach is ever reported. Enterprises are protected by internal mirroring.
What are some examples of batteries-included languages that folk around here really feel productive in and/or love? What makes them so great, in your opinion?
(Leaving aside thoughts on language syntax, compile times, tooling etc - just interested in people's experiences with / thoughts on healthy stdlibs)
These are the big ones I use, specifically because of the standard libraries:
Python (decent standard library) - It's pretty much everywhere. There's so many hidden gems in that standard library (difflib, argparse, shlex, subprocess, cmd)
C#/F# (.NET)
C# feels so productive because of how much is available in .NET Core, and F# gets to tag along and get it all for free too. With C# you can compile executables down to bundle the runtime and strip it down so your executables are in the 15 MiB range. If you have dotnet installed, you can run F# as scripts.
These are definitely some good thoughts, thanks!
Do you worry at all about the future of F#? I've been told it's feeling more and more like a second-class citizen on .NET, but I don't have much personal experience.
I used to, but the knowledge of .NET seems mostly transferrable to C#. It's super useful to do `dotnet fsi` and then work out the appropriate .NET calls in the F# repl.
That's a really good point, thank you. I'm running Elixir in prod right now, so F# would be up my alley.
As someone with all the memory of a fruit fly, I love a good REPL when I'm trying to recall the various magic incantations that make the computer go.
Go is well known for its large and high quality std lib
Go didn't even have versioning for dependencies for ages, so CVE reporting was a disaster.
And there's plenty of libraries you'll have to pull to get a viable product.
I work in a NIS2 compliance sector, and we basically use Go and Python for everything. Go is awesome, Python isn't as such. Go didn't always come with the awesome stllib that it does today, which is likely partly why a lot of people still use things like Gin for web frameworks rather than simply using the standard library. Having worked with a lot of web frameworks, the one Go comes with is nice and easy enough to extend. Python is terrible, but on the plus side it's relatively easy to write your own libraries with Python, and use C/Zig to do so if you need it. The biggest challenges for us is that we aren't going to write a better MSSQL driver than Microsoft, so we use quite a bit of dependencies from them since we are married with Azure. These live in a little more isolation than what you might expect, so they aren't updated quite as often as many places might. Still, it's a relatively low risk factor that we can accept.
Our React projects are the contrast. They live in total and complete isolation, both in development and in production. You're not going to work on React on a computer that will be connected to any sort of internal resources. We've also had to write a novel's worth of legal bullshit explaining how we can't realistically review every line of code from React dependencies for compliance.
Anyway, I don't think JS/TS is that bad. It has a lot of issues, but then, you could always have written your own wrapper ontop of Node's fetch instead of using Axios. Which I guess is where working in the NIS2 compliance sector makes things a little bit different, because we'd always chose to write the wrapper instead of using one others made. With the few exceptions for Microsoft products that I mentioned earlier.
This is really interesting, thanks for sharing. Great food for thought.
Being tightly coupled with MS already, did you ever explore .NET?
We used to have some C# but we moved away from it to have fewer languages and because it was a worse fit for us than Go and Python. I'm not sure .NET would really give us any advantages though. Microsoft treats most major languages as first class citizens in Azure, and since we build everything to be sort of platform agnostic, we wouldn't have the tie-ins that you could have with .NET. I'm not saying it would be fun to switch cloud, but all our services are build so that there is a decoupled "adapter" between our core logic and Azure. We use a lot of Azure functions as an example, but they run in container apps on a managed k8s, so the Azure function part is really just an ingress that could be swapped for anything else.
It's been a while since I worked with an "actual" function app in Azure. We did have a few .NET ones that weren't using containers. At the time they were pretty good, but today I'm not sure what the benefit over a managed container envrionment with container apps would be. Similarily with sqlserver. We use it because of governance and how it ties into data factory and I guess fabric, but we don't use ORM's so something like Entity Framework wouldn't really be something we'd benefit from with .NET.
I think the only thing we couldn't realistically replace and get something similar is the governance, but that's more to do with how Management Groups, Policies, Subscriptions and EntraID works than anything else.
Eventuallyt everything will probably be Python and then C/Zig for compute heavy parts. Not because Python is great, it's terrible, but it's what everyone uses. We're an energy company and with the internal AI tools we've made widely available we now have non-SWE employees writing code. It's Business Intelligence, it's Risk analysys, it's powerplant engineers, it's accountants. They're all working with AI code in their sandboxed environments and it's all Python. Since some of it actually turns out to generate great value, it's better for us (and the business) if our SWE teams can easily take over when "amateur hour" needs to meet operational compliance for the more "serious" production envrionments. I put things in "'s because I'm still not entirely sure how to express this. A lot of what gets build is great, and would have never been build without AI because we don't have the man power, but it's usually some pretty bad software. Which is fine, until it isn't.
Thanks for taking the time, I really appreciate the insights.
The irony in this case is that axios is not really needed now given that fetch is part of the JS std lib.
Irony is that Node has no need for Axios, native fetch support has been there for years, so in terms of network requests it is batteries included.
People use axios or ky because with fetch you inevitably end up writing a small wrapper on top of it anyway.
Some might say the tradeoff of writing a small wrapper is worth it given what’s been demonstrated here.
Yeah but what about other deps like db drivers?
In my experience people feel the need to wrap axios too.
These are the kind of people I hope AI replaces
AI was trained on Axios wrappers, so it's just going to be wrappers all the way down. Look inside any company "API Client" and it's just a branded wrapper around Axios.
Speak for yourself, Claude works fine with fetch on my system.
I have never consciously wrapped Axios or fetch, but a cursory search suggests that there was a time when it was impossible for either to force TLS1.3. It's easy to imagine alternate implementations exist for frivolous reasons, but sometimes there are hard security or performance requirements that force you into them.
Fetch has also lacked support for features that xhr has had for over a decade now. For example upload progress. It's slowly catching up though, upload progress is the only thing I'd choose xhr for.
You can pipe through a TransformStream that counts how many bytes you've uploaded, right?
That is a way to approximate it, though I'd be curious to know the semantics compared to xhr - would they both show the same value at the same network lifecycle of a given byte?
That would show how quickly the data is passing into the native fetch call but doesn’t account for kind of internal buffer it might have, network latency etc
I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
Hm, I don't think axios would do much better here. `fetch` is the official replacement for axios. If both are flawed that's another topic
No. Axios is still maintained. They have not deprecated the project in favor of fetch.
I'm not saying that axios is unmaintained, I'm saying that if you want something like axios from the standard lib, fetch is the closest thing you get to official
Sure but Axios determine what the official replacement for Axios is.
It's not deprecated, it's obsoleted.
Axios has maxContentLength and maxBodyLength options. I would probably go with undici nowadays though (it also has maxResponseSize).
Nor is fetch a good client-side API either; you want progress indicators, on both upload and download. Fetch is a poor API all-round.
You can pass to `fetch` an `AbortSignal` like `AbortSignal.timeout(5000)` as a simple and easy guard.
If you also want to guard on size, iterating the `response.body` stream with for/await/of and adding a counter that can `abort()` a manual `AbortSignal` is relatively straightforward, though sounds complicated. You can even do that as a custom `ReadableStream` implementation so that you can wrap it back into `Response` and still use the `response.json()` shortcut. I'm surprised I'm not seeing a standard implementation of that, but it also looks straightforward from MDN documentation [1].
[1] https://developer.mozilla.org/en-US/docs/Web/API/Streams_API...
Browser fetch can lean on the fact that the runtime environment has hard limits per tab and the user will just close the tab if things get weird. on the server you're right
It doesn't have a need _now_. Axios is more than 10 years old now, and even before axios other libraries did the same utility of making requests easier
Browsers too.
It’s not needed anymore.
Node fetch is relatively new. Wasn't marked stable until 2023, though I've used it since like 2018.
It doesn't matter. We pulled axios out of our codebase, but it still ends up in there as a child or peer from 40 other dependencies. Many from major vendors like datadog, slack, twilio, nx (in the gcs-cache extension), etc...
Yep, got stung because a bunch of things still pull in axios even though we don't use it.
Every time I'm reminded of this article about package managers. https://www.gingerbill.org/article/2025/09/08/package-manage...
Serverless framework up until late yesterday also.
Because native fetch lack retries, error handling is verbose, search and body serialization create ton of boilerplate. I use KY http client, small lib on top of fetch with great UX and trusted maintainer.
But javascript is batteries included in this case, you can use xmlhttprequest or fetch
I agree. Got downvoted a lot the other day for proposing Node should solve fundamental needs.
Different programmers have very different ideas about what is "all the functionality you typically need."
So, youre on Microsoft then, judging by ScottPlot you write .NET desktop apps. If you use Dapper, you probably use Microsoft.Data.SqlClient, which is... distributed over NuGet and vulnerable to supply chain attack. You may not need many deps as a desktop dev. Modern day line of business apps require a lot more deps. CSVHelper, ClosedXML, AutoMapper, WebOptimizer, NetEscapades.AspNetCore.SecurityHeaders.
Yes less deps people need the better but it doesn't fix trhe core problem. Sharing and distrib uting code is a key tenant of being able to write modern code.
I agree that dependencies are a liability, but, sadly, "batteries included" didn't work out for Python in practice (i. e. how do I even live without numpy? No, array aren't enough).
To the extend that Python is indeed "batteries included," that seems true. But just how "batteries included" is it? I'd argue that its batteries are pretty limited. Exhibit A: everybody uses the third-party requests instead of the stdlib urllib. Exhibit B: http.server isn't a production-ready webserver, so people use Flask or something beefier.
I'd contrast Python with Go, which has an amazing stdlib for the domains that Go targets. This last part is key--Go has a more focused scope than Python, and that makes it easier for its stdlib to succeed.
We could have different Python package bundles: Python base. Python webdev. Python desktop.
Nit, but relevant nit: Flask is also not a production-grade webserver. You could say it is also missing batteries ... and those batteries are often missing batteries too. Which is why you don't deploy flask, you deploy flask on top of gunicorn on top of nginx. It's missing batteries all the way down (or at least 3 levels down).
Appreciate the nit. Had no idea that Flask wasn't production-grade. Yeesh.
I really don't miss this part of the Python world. When I started on backend stuff ~10 years ago, the morass of runtime stuff for Python webservers felt bewildering. uWSGI? FastCGI? Gunicorn? Twisted? Like you say, missing batteries all the way down, presumably due to async/GIL related pains.
Then you step into the Go world and it's just the stdlib http package.
Anyway, ranting aside, batteries included is a real thing, and it's great. Python just doesn't have it.
Language churn makes this problem worse.
Frankly inventing a new language is irresponsible these days unless you build on-top of an existing ecosystem because you need to solve all these problems.
or you don't use a package manager where anyone can just publish a package (i.e. use your system package manager). There is still some risk, but it is much smaller. Like, if xz were distributed by PyPI or NPM, everyone would have been pwned, but instead it was (barely) found.
It's true that system repos doesn't include everything, but you can create your own repositories if you really need to for a few things. In practice Fedora/EPEL are basically sufficient for my needs. Right now I'm deploying something with yocto, which is a bit more limited in slection, but it's pretty easy to add my own packages and it at least has hashes so things don't get replaced without me noticing (to be fair, I don't know if the security practices of open-embedded recipes are as strong as Fedora...).
it's muddying what a package is. A package, or a distro, is the people who slave and labor over packaging, reviewing, deciding on versions to ship, having policies in place, security mailing lists, release schedules, etc.
just shipping from npm crap is essentially the equivelant of running your production code base against Arch AUR pkgbuilds.
Batteries included systems are still susceptible to supply chain attacks, they just move slower so it’s not as attractive of a target.
I think packages of a certain size need to be held to higher standards by the repositories. Multiple users should have to approve changes. Maybe enforced scans (though with trivy’s recent compromise that wont be likely any time soon)
Basically anything besides lone developer can decide to send something out on a whim that will run on millions of machines.
While technically true, it's so much slower that it's essentially a different thing. Third party packages being attacked is a near daily occurrence. First party attacks happens on the timescale and frequency of decades.
It's like the difference in protecting your home from burglars and foreign nation soldiers. Both are technically invaders to your home, but the scope is different, and the solutions are different.
Well, there’s other things. Maven doesn’t allow you to declare “version >= x.y.z” and doesn’t run arbitrary scripts upon pulling dependencies, for one thing. The Java classpath doesn’t make it possible to have multiple versions of the same library at the same time. That helps a lot too.
NPM and the way node does dependency management just isn’t great. Never has been.
yep!
This is exactly the world I'm working towards with packaging tooling with a virtual machine i.e. electron but with virtual machines instead so the isolation aspect comes by default.
Or write your own stuff. Yes, that's right, I said it. Even HTTP. Even cryptography. Just because somebody else messed it up once doesn't mean nobody should ever do it. Professional quality software _should_ be customized. Professional developers absolutely can and should do this and get it right. When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
Good luck
This post is modded down (I think because of the "roll your own crypto vibe", which I disagree with), but this is actually spot on the money for HTTP.
The surface area for HTTP is quite large, and your little API, which never needed range-requests, basic-auth, multipart form upload, etc suddenly gets owned because of a vulnerability in one of those things you not only never used, you also never knew existed!
"Surface area" is a problem, reducing it is one way to mitigate.
Again, you run into the attack surface area here. Think about the Heartbleed vulnerability. It was a vulnerability in the DTLS implementation of OpenSSL, but it affected every single user, including the 99% that weren't using DTLS.
Experienced developers can, and should, be able to elide things like side-channel attacks and the other gotchas that scare folks off of rolling their own crypto. The right solution here is better-defined, well understood acceptance criteria and test cases, not blindly trusting something you downloaded from the internet.
The reason I disagree about crypto is because:
1. It's really really hard to verify that you have not left a vulnerability in (for a good time, try figuring out all the different "standards" needed in x509), but, more importantly,
2. You already have options for a reduced attack surface; You don't need to use OpenSSL just for TLS, you can use WolfSSL (I'm very happy with it, actually). You don't need WolfSSL just for public/private keys signing+encryption, use libsodium. You don't need libsodium just for bcrypt password hashing, there's already a single function to do that.
With crypto, you have some options to reduce your attack surface. With HTTP you have few to none; all the HTTP libs take great care to implement as much of the specification as possible.
That's actually not really crypto, though - that's writing a parser (for a container that includes a lot of crypto-related data). And again... if you import a 3rd-party x.509 parser and you only need DER but not BER, you've got unnecessary bloat yet again.
There are several issues with "Batteries Included" ecosystems (like Python, C#/.NET, and Java):
1. They are not going to include everything. This includes things like new file formats.
2. They are going to be out of date whenever a standard changes (HTML, etc.), application changes (e.g. SQLite/PostgreSQL/etc. for SQL/ORM bindings), or API changes (DirectX, Vulcan, etc.).
3. Things like data structures, graphics APIs, etc. will have performance characteristics that may be different to your use case.
4. They can't cover all nice use cases such as the different libraries and frameworks for creating games of different genres.
For example, Python's XML DOM implementation only implements a subset of XPath and doesn't support parsing HTML.
The fact that Python, Java, and .NET have large library ecosystems proves that even if you have a "Batteries Included" approach there will always be other things to add.
Why would they be out of date? The ecosystems themselves (for example .NET) receives regular updates.
Yes, they cannot include everything, but enough that you do not _need_ third party packages.
"Batteries included" means "ossification is guaranteed", yah. "stdlib is where code goes to die" is a fairly common phrase for a reason.
There's clearly merit to both sides, but personally I think a major underlying cause is that libraries are trusted. Obviously that doesn't match reality. We desperately need a permission system for libraries, it's far harder to sneak stuff in when doing so requires an "adds dangerous permission" change approval.
Golang seems to do a good job of keeping the standard library up to date and clean
Largely, yes.
But also everyone sane avoids the built-in http client in any production setting because it has rather severe footguns and complicated (and limited) ability to control it. It can't be fixed in-place due to its API design... and there is no replacement at this point. The closest we got was adding some support for using a Context, with a rather obtuse API (which is now part of the footgunnery).
There's also a v2 of the json package because v1 is similarly full of footguns and lack of reasonable control. The list of quirks to maintain in v2's backport of v1's API in https://github.com/golang/go/issues/71497 (or a smaller overview here: https://go.dev/blog/jsonv2-exp) is quite large and generally very surprising to people. The good news here is that it actually is possible to upgrade v1 "in place" and share the code.
There's a rather large list of such things. And that's in a language that has been doing a relatively good job. In some languages you end up with Perl/Raku or Python 2/3 "it's nearly a different language and the ecosystem is split for many years" outcomes, but Go is nowhere near that.
Because this stuff is in the stdlib, it has taken several years to even discuss a concrete upgrade. For stuff that isn't, ecosystems generally shift rather quickly when a clearly-better library appears, in part because it's a (relatively) level playing field.
This looks like an ad for batteries included to me.
Libraries also don't get it right the first time so they increment minor and major versions.
Then why is it not okay for built-in standard libraries to version their functionality also? Just like Go did with JSON?
The benefits are worth it judging by how ubiquitous Go, Java and .NET are.
I'd rather leverage billions of support paid by the likes of Google, Oracle and Microsoft to build libraries for me than some random low bus factor person, prone to be hacked at anytime due to bad security practices.
Setting up a large JavaScript or Rust project is like giving 300 random people on the internet permission to execute code on my machine. Unless I audit every library update (spoiler: no one does it because it's expensive).
Third party libraries have been avoiding those json footguns (and significantly improving performance) for well over a decade before stdlib got it. Same with logging. And it's looking like it will be over two decades for an even slightly reasonable http client.
Stuff outside stdlib can, and almost always does, improve at an incomparably faster rate.
And I think the Go people seem to do a fairly good job of picking out the best and most universal ideas from these outside efforts and folding them in.
.NET's JSON and their Kestrel HTTP server beg to differ.
Their JSON even does cross-platform SIMD and their Kestrel stack was top 10/20 on techempower benchmarks for a while without the ugly hacks other frameworks/libs use to get there.
stdlib is the science of good enough and sometimes it's far above good enough.
Libraries don't get it right the first time, but there are often multiple competing libraries which allows more experimentation and finding the right abstraction faster.
Rust is especially vulnerable witb Serde included in everything and maintained by 1 perso
Another downside of a large stdlib, is that it can be very confusing. Took my a while how unicode is supposed to work in go, as you have to track down throughout the APIs what are the right things to use. Which is even more annoying because the support is strictly binary and buried everywhere without being super explicit or discoverable.
I'm not sure I understand. Why would a standard library, a collection of what would otherwise be a bunch of independent libraries, bundled together, be more confusing than the same (or probably more) independent libraries published on their own?
For me, the v2 re-writes, as well as the "x" semi-official repo are a major strength. They tell me there is a trustworthy team working on this stuff, but obviously not everything will always be as great as you might want, but the floor is rising.
yea, I like the /x/ repos a fair bit. "first-party but unstable" is an extremely useful area to have, and many languages miss it by only having "first-party stable forever" and "third party". you need an experimentation ground to get good ideas and seek feedback, and keeping it as a completely normal library allows people/the ecosystem to choose versions the same way as any other library.
100% to libraries having permissions. If I'm using some code to say compute a hash of a byte array, it should not have access to say the filesystem nor network.
Except I rather have ossified batteries that solve my problem, even if not as convinient as more modern alternatives, than not having them at all on a given platform.
The goal is not to cover everything, the goal is to cover 90% of the use cases.
For C#, I think they achieved that.
You might want to elaborate on the "etc.", since HTML updates are glacial.
glaciers change faster than HTML
Oof, I honestly hadn't considered that.
because there is more human effort
The HTML "Living Standard" is constantly updated [1-6].
The PNG spec [7] has been updated several times in 1996, 1998, 1999, and 2025.
The XPath spec [8] has multiple versions: 1.0 (1999), 2.0 (2007), 3.0 (2014), and 3.1 (2017), with 4.0 in development.
The RDF spec [9] has multiple versions: 1.0 (2004), and 1.1 (2014). Plus the related specs and their associated versions.
The schema.org metadata standard [10] is under active development and is currently on version 30.
[1] https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... (New)
[2] https://web.dev/baseline/2025 -- popover API, plain text content editable, etc.
[3] https://web.dev/baseline/2024 -- exclusive accordions, declarative shadow root DOM
[4] https://web.dev/baseline/2023 -- inert attribute, lazy loading iframes
[5] https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... (Baseline 2023)
[6] https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... (2020)
[7] https://en.wikipedia.org/wiki/PNG
[8] https://en.wikipedia.org/wiki/XPath
[9] https://en.wikipedia.org/wiki/Resource_Description_Framework
[10] https://schema.org/
please! nobody uses Xpath (coz json killed XML), it RDF (semantic web never happened, and one ever 10years is not fast), schema.org (again, nobody cares), PNG: no change in the last 26 years, not fast. the HTML "living standard" :D completely optional and hence not a standard but definition.
Xpath is still used for e2e tests and things like scraping. Especially when there aren't better selectors available.
The point is that you don't need the very latest version. The 20 years old version is enough.
XPath 1.0 is a pain to write queries for. XPath 2.0 adds features that make it easier to write queries. XPath 3.1 adds support for maps, arrays, and JSON.
And the default Python XPath support is severely limited, not even a full 1.0 implementation. You can't use the Python XPath support to do things like `element[contains(@attribute, 'value')]` so you need to include an external library to implement XPath.
XPath is used in processing XML (JATS and other publishing/standards XML files) and can be used to proces HTML content.
RDF and the related standards are still used in some areas. If the "Batteries Included" standard library ignores these then those standards will need an external library to support them.
Schema.org is used by Google and other search engines to describe content on the page such as breadcrumbs, publications, paywalled content, cinema screenings, etc. If you are generating websites then you need to produce schema.org metadata to improve the SEO.
Did you notice that a new PNG standard was released in 2025 (last year, with a working draft in 2022) adding support for APNG, HDR, and Exif metadata? Yes, it hasn't changed frequently, but it does change. So if you have PNG support in the standard library you need to update it to support those changes.
And if HTML support is optional then you will need an external library to support it. Hence a "Batteries Included" standard library being incomplete.
Its used plenty in legacy systems that are still around.
Python, .NET, and Java are not examples of batteries included.
Django and Spring
comparing to Node, .NET is batteries included: built-in Linq vs needing lodash external package, built-in Decimal vs decimal.js package, built-in model validation vs class-validator & class-transformer packages, built-in CSRF/XSRF protection vs csrf-csrf package, I can go on for a while...
And in fact wasn't a popular Python library just compromised very recently? See https://news.ycombinator.com/item?id=47501426.
So Python's clearly not "batteries included" enough to avoid this kind of risk.
That's my point. You can have a large standard library like those languages I mentioned, but that isn't going to include everything nor cover every use case, so you'll have external libraries (via PyPi for Python, NuGet for .NET, and Maven for Java/JVM).
Python's standard library is definitely much more batteries-included than JavaScript's.
depends, JavaScript in the Browser has many useful things available, which I miss with python, e.g., fetch, which in Python you need a separate package like requests to avoid a clunky API. Java had this issue for long time as well, since Java 11 there is the HttpClient with a convenient API.
Not at all. We simply need M-of-N auditors to sign off on major releases of things. And the package managers need to check this (the set of auditors can be changed, same as browser PKI for https) before pulling things down.
That's the system we have in our Safebox ecosystem
I think stay vigilant is better than trust anything.
While it's true that the packages are first party, .NET still relies on packages to distribute code that's not directly inside the framework. You still probably transiently depend on `Microsoft.Extensions.Hosting.Abstractions ` for example - if the process for publishing this package was compromised, you'd still get owned.
This just moves the trust from one group to another. Now the standard library/language maintainers need to develop/maintain more high quality software. So either they get overworked and burn out, don't address issues, fail to update things or they recruit more people who need to be trusted. Then they are responsible for doing the validation that you should have done. Are they better equipped to do that? Maybe they go, oh hey, Axios is popular and widely trusted, let's make it an official library and bring the maintainers into the fold... wait isn't this exactly where we started?
What process did you trust the standard library/language maintainers in the first place? How do they differ from any other major library vendor?
I agree with you and follow the same principles myself, but JavaScript already has HTTP, and yet everyone still uses Axios. So the problem isn't that JS doesn't have batteries, it's that people don't want to use them for some reason.
I'm guessing it's similar to the tragedy of the commons phenomenon. When things are freely available people tend to overuse or carelessly use them. NPM is just too easy to use. If a package offers a 1% ergonomics increase over a builtin function, many folks will just go for it because it costs them nothing (well, it seems to cost them nothing).
My first thought was does VS Code Insiders use it (or anything it relies on, or do any extensions etc). Made me think.
If someone from github is reading this, https://github.com/axios/axios/issues/10604#issuecomment-416...
I think that jason might like if someone from github team can contact them as soon as possible.
(8 minutes ago at the time of writing)
compiled JS solves a problem that no longer exists. IE6 is dead RIP.
Now we have a 20MB main.min.js problem
Some great tips in this thread and I've been collecting them all at https://github.com/bodadotsh/npm-security-best-practices
NPM should learn from Linux distribution package managers.
Have a branch called testing, and packages stay in testing for few weeks, after which they go to stable. That is how many Linux distributions handle packages. It would have prevented many of these.
Advising every user of npm/pnpm to change their settings and set their own cooldown periods is not a real choice.
NPM is one big AUR, where anyone can submit arbitrary unverified code. The difference is that AUR is intentionally harder to use to prevent catastrophic one-line installs.
Is a "AUR" now just how we name unaudited software repositories?
Just to note, if we're talking about Linux Distributions. There's also COPR in Fedora, OBS for OpenSUSE (and a bunch of other stuff, OBS is awesome), Ubuntu has PPAs. And I am sure there's many more similar solutions.
Not all distributions work with a staging repository, and it's not really intended for this purpose either.
Besides there's always a way to immediately push a new version to stable repositories. You have to in order to deal with regressions and security fixes.
I know not all, but Debian/Ubuntu/Fedora does, and while the intended purpose of multi-stage releases is not necessarily security but stability, it still does help up with security too. Because third parties can look and scan the dependencies while they are still not in stable.
Most of the supply chain vulnerabilities that ended up in the NPM would have been mitigated with having mandatory testing / stable branches, of course there needs to be some sort of way to skip the testing but that would be rather rare and cumbersome and audited, like it is in Linux distributions too.
There's a package manager discussion, but the bit that stands out to me is that this started with a credential compromise. At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar. I wish that _important keys live in hardware_ becomes more standard given the stakes.
Dealing with dependencies is another question; if it's stupid stuff like leftpad then it should be either vendored in or promoted to be a language feature anyway (as it has been).
Or those people can (fund) separate repackaging and redistribution with more stringent and formalized review process.
Maybe not all users should pull all packages straight from what devs are pushing.
There's no reason we can't have "node package distributions" like we have Linux distributions. Maybe we should stop expecting devs and maintainers and Microsoft to take responsibility for our supply-chain.
Totally agree.
Also, considering how prevalent TPM/Secure Enclaves are on modern devices, I would guess most package maintainers already have hardware capable of generating/using signing keys that never leave hardware.
I think it is mostly a devex/workflow question.
Considering the recent ci/cd-pipeline compromises, I think it would make sense to make a two phase commit process required for popular packages. Build and upload to the registry from a pipeline, but require a signature from a hardware resident key before making the package available.
Most of axios' functionality has effectively been promoted to a language feature as `fetch`, but the problem is people don't bother to migrate. I've migrated our direct usage of it but it's still pulled in transitively in several parts of our codebase.
Even left-pad is still getting 1.6 million weekly downloads.
Annoyingly, the times I reach for axios and similar is when I need to keep track of upload progress, which I could only do with XMLHttpRequest, not fetch, unless I've missed some recent browser changes, and the API of XMLHttpRequest remains as poor as the first times I had to use it. Download progress been supported by fetch since you can track chunks yourself, but somehow they didn't think to do that for requests for some reason, only responses.
I kind of feel like the authors here should want that for themselves, before the community would even realize it's needed. I can't say I've worked on packages that are as popular as axios, but once some packages we were publishing hit 10K downloads or so, we all agreed that we needed to up our security posture, and we all got hardware keys for 2FA and spent 1-2 weeks on making sure it was as bullet-proof we could make it.
To be fair, most FOSS is developed by volunteers so I understand not wanting to spend any money on something you provide for free, but on the other hand, I personally wouldn't feel comfortable being responsible for something that popular without hardening my own setup as much as I could, even if it means stopping everything for a week.
I thought npm started requiring hardware keys for publish, or may have been new accounts only
There's no community, the users of axios are devs that looked at stackoverflow for "how to download a file in javascript", they barely know or care what axios is.
Now the users of axios are devs that ask Claude Code or Codex to scrape a website or make a dashboard, they don't even know about the word axios.
I personally had to delete axios a couple of time from my codebase when working with junior devs.
I am not saying this is the reason for this compromise but the sudden explosion of coding assistant like claude code, and tools like openclaw is teaching entire crop of developers (and users) that it is ok to have sensitive credentials .env files.
Where would you suggest putting the sensitive credentials?
Not in .env files next to your code that is exposed to supply chain risks.
infisical is a great solution
I know there is a cooldown period for npm packages, but I’m beginning to want a cooldown for domains too. According to socket, the C2 server is sfrclak[.]com, which was registered in the last 24 hours.
NextDNS has a setting to block newly registered (<30d) domains.
Woah! TIL! Thank you!
Reset the clock
first day at hacker news and this is the first post i saw
WTF!!!! gaslighting your victims into believing they are not victims. the ingenuity of this is truly mindblowing. I am shocked at such thing is even allowed. like packages should not be able to modify their contents while they are being instaleld.
I'm impressed how this was caught as a network anomaly in a GitHub actions monitoring tool.
This might have taken a lot longer to discover, otherwise.
Guix saves you from this. You can import NPM packages in a container (not even touching $HOME) and giving you a shell on the spot with just the dependencies and nothing more.
Learn about 'guix import'.
Oh, and you can install Guix on any GNU/Linux distro.
A new day, a new npm incident.
I don't see how a system that relies on trust can scale safely
With all the recent supply chain attacks, I'm starting to think it's only a matter of time before all of us are victims. I think this is a sign to manually check all package diffs or postinstall scripts.
Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
This needs to be done (as we've seen from these recent attacks) in your devenv, ci/cd and prod environments. Not one, or two, but all of these environments.
The easiest way is via using something like kubernetes network policies + a squid proxy to allow limited trusted domains through, and those domains must not be publicly controllable by attackers. ie. github.com is not safe to allow, but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it.
Network firewalls that perform SSL interception and restrict DNS queries are an option also, though more complicated or expensive than the above.
This stops both DNS exfil and HTTP exfil. For your devenv, software like Little Snitch may protect your from these (I'm not 100% on DNS exfil here though). Otherwise run your devenv (ie vscode) as a web server, or containerised + vnc, a VM, etc, with the same restrictions.
Getting zero day patches 7 days later if no proper monitoring about important patches or if this specific patch is not in the important list. Always a tradeoff.
Thats true. Setting to 7 days saves you from a supply chain attack, but opens you to zero days. Another example why network filtering is a better solution.
But raw.githubusercontent.com still contains code and now the attacker can publish the code he wants no!?
Don't get me wrong: I love the idea to secure as much as possible. I'm running VMs and containerizing and I eat firewalling rules for breakfast, my own unbound DNS with hundreds of thousands (if not millions) of domains blocked, etc. I'm not the "YOLO" kind of guy.
But I don't understand what's that different between raw.githubusercontent.com and github.com? Is it for exploits that are not directly in the source code? Can you explain a bit more?
In the case of compromised code, the attacker has already loaded what he wants, so loading extra code from raw.githubusercontent.com is not the issue, or our threat model. We are already compromised!
The issue is that code then extracting secrets and data from your organisation, ie. data exfil.
raw.githubusercontent.com can not be used to submit data to, it's read only, but github.com obviously can.
Note, if you really needed github.com access in your application or environment, then you need to use SSL interception (using squid or a firewall) and allow certain URLs and methods ie. GET requests only from your organisations path, to make it safe.
Absolute wave of supply chain attacks recently. Hopefully this causes everyone to tighten up their dependencies and update policies.
I've been saying for ages, use xmlhttprequest, or hell, even fetch().
Stop downloading code from the internet unless it's a major strategic decision.
Just a reminder that you can run most node things with deno run and have opt in permissions, audit trail and even external permission system integration now. The gotcha is that "deno task <<some package.json script>>" will NOT execute with this model which I find extremely unintuitive and had me thinking deno abandoned its sandbox for nodejs compatibility completely.
Can we get a non-AI-generated article for this? I think the aikido one might be fine, but if there’s a more official source let’s use that in lieu of this AI nonsense.
There’s a recurrent pattern with these package compromises: the attacker exfiltrates credentials during an initial phase, then pivots to the next round of packages using those credentials. That’s how we saw them make the Trivy to LiteLLM leap (with a 5 day gap), and it’ll almost certainly be similar in this case.
The solution to this is twofold, and is already implemented in the primary ecosystems being targeted (Python and JS): packagers should use Trusted Publishing to eliminate the need for long lived release credentials, and downstreams should use cooldowns to give security researchers time to identify and quarantine attacks.
(Security is a moving target, and neither of these techniques is going to work indefinitely without new techniques added to the mix. But they would be effective against the current problems we’re seeing.)
There are solutions, the problem is almost always discipline.
I don’t know what this means. Discipline is good, but I think you need to have good tools/primitives in place to help people exercise discipline.
(The classic example being passwords: we wouldn’t need MFA is everybody just “got good” and used strong/unique passwords everywhere. But that’s manifestly unrealistic, so instead we use our discipline budget on getting people to use password managers and phishing-resistant MFA.)
Really? You don't know the difference between having a door lock, and using it?
MFA is typically enforced by organizations, forcing discipline. Individual usage of MFA is dramatically lower
In this case, the author's NPM account was taken over, email address changed to one the attacker controls, and the package was manually published.
Since the attacker had full control of the NPM account, it is game over - the attacker can login to NPM and could, if they wanted, configure Trusted Publishing on any repo they control.
Axios IS using trusted publishing, but that didn't do anything to prevent the attack since the entire NPM account was taken over and config can be modified to allow publishing using a token.
Well, that sucks! It’ll be interesting to learn how they obtained a valid second factor or 2FA bypass; that will inform the next round of defenses here.
Yeah, NPM should be enforcing 2FA and likely phishing resistant 2FA for some packages/ this should be a real control, issuing public audit events for email address changes, and publish events should include information how it was published (trusted publishing, manual publish, etc).
Instead they took away TOTP as a factor.
Scaling security with the popularity of a repo does seem like a good idea.
Are there downsides to doing this? This was my first thought - though I also recognize that first thoughts are often naive.
You don't want "project had X users so it's less safe" to suddenly transition into "now this software has X*10 users so it has to change things", it's disruptive.
TOTP although venerable was better than no second factor at all.
TOTP isn't phishing resistant
No it's not but it's better than nothing. Don't let the perfect be the enemy of the good.
It's not much better than nothing. It basically solves "I reused my password across sites" exclusively, that's it. If you're going to go through the effort of TOTP, it seems odd that you wouldn't just use a unique password.
If you use a unique password it's questionable if it adds any value at all. Perhaps in very niche situations like "password authentication is itself vulnerable due to a timing attack/ bug" or some such thing... but we've rarely seen that in the wild.
I disagree.
I use a password manager and systemically use long random passwords. An attacker would need to compromise my password manager, phish me, wrench me, or compromise the site the credential is associated with to get that.
Using local only TOTP (no cloud storage or portability for me, by choice) they would have to additionally phish me, wrench me, compromise my phone, or compromise my physical security to get the code.
None of these are easy except the wrench which is high risk. My password manager had standard features which make me more phishing resistant, and together they are more challenging than either apart. For example the fact that my password manager will not fill in the password on a non associated site means I am much less likely to fill in a TOTP code on an inappropriate site. Though there are vulnerable scenarios they aren't statistically relevant in the wild and the bar is higher regardless.
Now I happen to have a FIDO key which I use for my higher security contexts but I'm a fairly low value target and npm isn't one of my high security contexts. TOTP improves my security stance generally and removing it from npmjs.org weakened my security stance there.
TOTP seems effectively useless for npm so that seems fine to me
https://docs.npmjs.com/configuring-two-factor-authentication
I'm assuming the author must have been grandfathered in to TOTP?
One wonders if Microsoft/npm.js should allow new packages to be published immediately following an account email address change? I mean changes to email address are already recognized as potential attack vectors, so emails are sent to the old address warning of potential account take over. But this seems to have been done at night, so the warning email would not be seen yet. Even so a new package could be published and served to the world immediately. Unless I misunderstand something about the facts this would indicate an extreme lack of imagination in the people at Microsoft who already went through several cycles of hardening the service against supply chain poisoning attacks.
Was this an drive-by/auto-install attack?
There's another element to the solution here: runtime behavioral analysis. No matter how completely the maintainer's credentials are compromised, no matter how well the malware is concealed, it still has to act like malware (in the case of LiteLLM, credential harvesting, in this case a remote-access Trojan). It's possible to detect the behavior, rather than relying on supply-chain integrity.
We built a free tool that runs local behavior analysis on your machine, it's caught every supply-chain attack in the last couple weeks: https://www.producthunt.com/products/axios-litellm-detector
Very detailed and props to the security researchers, but the blog post has several indicators that it was written by AI, to which point I suspect their malware analysis was also done by a LLM.
I just wish it had more human interaction rather than have a GenAI spit out the blog post. It's very repetitive and includes several EM dashes.
It is harder and harder to trust any blog post anymore, the more AI there is. I used to read blog posts because of the personality and the precision level. Now both have been taken away.
Yeah, 98% of blog posts only exist for SEO purposes and so most of that is of course written by LLM
It was in part written by a LLM, but I believe some of the analysis was done by humans. You can just check where there's proper punctuation and em-dashes in the commented code
I used axios in the distant past but haven't used it whenever I had my say in the past five years. You don't need it, and for special things like retries I could roll my own just fine. Now ai will roll it for you
node:fetch is all you need, simple and effective
This is exactly to avoid this kind of issue that I decided to work on StableBuild. StableBuild pins and hosts a copy of your dependencies at a specific freeze date, so that your supply chain is never contaminated. This way, a compromised version published after your freeze date (even with the same version number!) would never reach your build.
Literally every package manager already does this.
Is this Jia Tan 5.0? I've lost count. You really should stop trusting packages (implicitly). Or don't. It's your funeral, not mine. See you at Jia Tan 6.0 April?
Not at all, it was a regular maintainer account that was hijacked (probably through phishing) and used to push a malicious payload, not a threat actor posing as a contributor and adding a backdoor like in the Jia Tan case.
I use Jia Tan as a figurehead for malicious maintainers. This clearly was a targeted hack. Does it really matter how long it took to get the job done?
I'd argue this has not much in common with Jia Tan apart from both being supply chain attacks, there is no malicious maintainer here, a trusted maintainer had their account taken over.
I guess the end result is the same, a malicious package pushed by an account that was thought to be trusted, but I think the Jia Tan case is worth being looked at differently than just simple account takeover.
It's just a longer backstory. All the same in the end. Hackers targeted a popular package. The lead maintainer was compromised. The pattern fits. There will be more of these.
So this and litellm one would’ve been preventable by proper config of OIDC Trusted Publishers.
I don’t buy the “wait 7 days” being thrown around as a guard.
Wouldn’t that just encourage the bad actors to delay the activation of their payloads a few days or even remotely activated on a switch?
Of course the "wait 7 days" are not a silver bullet, but it gives automated scanners plenty of time to do their work. Those automated scanners surely catch this `eval(base64.decode("..."))` stuff that some of those attacks used so in my book this dependency cooldown is a net win. I guess the skilled malicious actors will then up their game but I think it's okay to kick off an arms race between them and the security scanners in the dependency world.
That's a good point. In some level I'd prefer the delay to happen on publication of the package itself. Do any of these scanners have cryptographic attestations or similar?
PSA from the Claude Code leaks it looks like it's using Axios (although an older version)
(A bit off-topic; half-joking, half-serious)
What a great time to be alive! Now, that's exactly why I enjoy writing software with minimal dependencies for myself (and sometimes for my family and friends) in my spare time - first, it's fun, and second, turns out it's more secure.
This only limits the possibility of compromise, it doesn't remove it. Python itself could be compromised, or the package that your linux distro provides could be.
With AI agents the volume and frequency of supply chain attacks is going to explode. I think our entire notion of how to develop and distribute software safely needs to change. I don't have answers; "reflections on trusting trust" explains the difficulties we now face.
How is it we've made it this far and we still don't have any kind of independent auditing of basic publish security on NPM? You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
Because all mainstream packages are published via CI/CD pipeline not by an MFA'd individual uploading a GZIP to npm.com
Requiring a human-in-the-loop for final, non-prerelease publication doesn't seem like that onerous of a burden. Even if you're publishing multiple releases a day on the regular (in which case ... I have questions, but anyway) there are all sorts of automations that stay secure while reducing the burden of having to manually download an artifact from CI, enter MFA, and upload it by hand.
You can still have a step that requires a certain user/group to sign off, and you can still enforce that those users have MFA set up. Almost any serious shop that expects to pass audits already does this in some form or fashion before pushing code to prod.
another week another npm supply chain attack
Why is it with Javascript the culture is to use so many dependencies?
All sorts of reasons, but this isn't a left-pad situation. Axios's functionality is something provided by a library in a lot of languages (C/C++ with libcurl and friends, Python with requests, Rust with reqwest, and so on).
That's not to say it's inherently necessary for it to be a third-party package (Go, Ruby, and Java are counterexamples). But this isn't a proliferation/anemic stdlib issue.
Why can't we freeze the version of globally installed packages with npm?
Script to check if you've been pwnd if anyone needs it (or just ask an llm to make one for you): https://gist.github.com/shamwow/93101381686f23d21a85da4bac5b...
We're going to see this more and more and more. And it's not going to stop. Because nobody in the industry will use the simplest, industry-standard security practices. Because they don't feel like it. A software building code is the only thing that'll fix it.
1/5 of your CLI and 1/3 of your gui apps are npm based. Each one has 400+ dependencies , none notable enough to go viral when they are breached. And who knows what other packages are currently compromised. We all have 30+ node_modules on our disks, and 2/3 of them were shipped by outside vendors , packaged in an archive.
“I’m smart I use fetch instead of axios”. “I pin my versions” – sure but certainly one of your npx or Electron apps uses axios or another less notably compromised package.
Let’s
Ofc this happens the day we launch on product hunt. The last time we launched, AWS went down.
Has anyone else noticed there was a recent sudden flurry of 3000 deleted issues on axios/axios? The jump happened on March 23. Was this a first sign of compromise? Or just coincidence of an AI agent going rogue.
There are pretty much exactly 3000 deleted issues, with the range starting at https://github.com/axios/axios/issues/7547 (7547) and ending at https://github.com/axios/axios/issues/10546 (10546 which is 7547+2999)
Maybe just a coincidence but they have cubic-dev-ai edit every single PR with a summary. And that bot edits PR descriptions even for outside contributors.
Maintainer replied here https://github.com/axios/axios/discussions/10612
I think it could well have been the attacker trying to hiding any notifications of suspicious activity (email address changed, suspicious login) in the flurry of issue related emails.
Do we have a way yet to tell if something on our system is compromised? There's plenty of end user software built on node, like Gemini CLI and LM Studio.
I've been advocating to minimize the number of dependencies for some time now. Once you've maintained an open source project for several years, you start to understand the true cost of dependencies and you actually start to pay attention to the people behind the libraries. Popularity doesn't necessarily mean reliable or trustworthy or secure.
Agree. This is one of the major takeaways I've had from writing Go over the years -- which is even a Go proverb [0], "a little copying is better than a little dependency." Fortunately, LLMs make writing your own implementations of little dependencies super easy too.
[0]: https://go-proverbs.github.io/
I'll agree with that, and you would think it's common sense for any competent engineer, but for many people it's just an afterthought. Including senior and lead engineers. General matters like security are a political liaison, how can you raise the alarm/advocate for critical security improvements as an IC without making people around and above you look bad? How can you justify time invested into these things without raising the alarm? At the same time, you can't just ignore glaring security holes. It's a fine line to walk, and actually being realistic about the possibilities has found me nothing but enmity from peers and superiors due to it seeming like I'm throwing them under the bus.
In general, management was to see progress. I've come to find that technical details like these are an afterthought for most engineers, so far as the deadlines are being met.
It's one of these things that are under the water, tech side jobs. Everyone has to be on board, if your peers don't give a fuck you're just an annoyance and will be swimming counter-current.
Can relate. It's like; if you're less rigorous than the CTO, they would think you're incompetent. If you're more rigorous than the CTO, they would think you're overly pedantic; not pragmatic enough.
I started a project on Expo recently and my god, a thousand dependencies later, it was running.
Ok it's bad, but our npm projects are pinned in the package-lock.json, which I imagine most would be? So who would pull this besides security scanners?
`npm install` might be enough to pull it, unless you pin down to the patch?
I don't think that's right if it's in your package-lock it wouldn't pull it unless you npm update axios, or delete the package-lock.json and then npm install.
Updating my packages feels like playing Russian Roulette
Uh Axios. Even after being years out of NPM dev I remember that as the XHR thing for node. Whichs rings a big hit even to out of the loop people...
Another obvious ChatGPT-ism. The fact that people are using AI to write these security posts doesn't surprise me, but the fact they use it to write a verbose article with spicy little snippets that LLMs seem to prefer does make it really hard to appreciate anything other than the simple facts in the article.
Yet another case in point for "do your own writing" (https://news.ycombinator.com/item?id=47573519)
How anybody is still using NPM is beyond me.
Curious to know why are coding agents not detecting such risks before importing dependencies?
I'm assuming you are talking about agents like claude-code and open-code which rely on GPT functions (AKA Large Language Models).
The reason they don't detect these risks is primarily because these risks are emergent, and happen overnight (literally in the case of axios - compromised at night). Axios has a good reputation. It is by definition impossible for a pre-trained LLM to keep up with time-sensitive changes.
I mean that agents can scan the code to find anything "suspicious". After all, security vendors that claim to "detect" malware in packages are relying on LLMs for detection.
An LLM is not a suitable substitute for purpose-built SAST software in my opinion. In my experience, they are great at looking at logs, error messages, sifting through test output, and that sort of thing. But I don't think they're going to be too reliable at detecting malware via static analysis. They just aren't built for that.
At this point picking Node for a backend is a foot gun. Large companies have the funds for private, security vetted npm repositories, but what about small companies, startups, personal projects? Pnpm makes things more secure without install scripts, mininum package time, but it's still the same activity, does an extra parachute make skydiving any less inherently dangerous?
I'm not dogmatic about the whole "JS for the backend is sin" from backend folks, but it seems like it was the right call. You should stick to large org backed packages, or languages with good enough standard libraries, like Go, Java, Python, C#.
Many of the suggestions in this thread (min-release, ignore script) are defenses for the consumers.
I've been working on Proof of Resilience, a set of 4 metrics for OSS, and using that as a scoring oracle for what to fund.
Popularity metrics like downloads, stars, etc are easy to fake today with ai agents. An interesting property is that gaming these metrics produces better code, not worse.
These are the 4 metrics:
1. Build determinism - does the published artifact match a reproducible build from source?
2. Fuzzing survival - does the package survive fuzz testing?
3. Downstream stability - does it break any repos dependent on this project when pushing a release?
4. Patch velocity - how fast are fixes merged?
Here's a link to the post, still early but would appreciate any feedback.
https://hackmd.io/@carlb/proof-of-resilience
Carl, with all due respect, have you used AI for making this hackmd post?
"it's not just a waste of money — it's a security problem"
I am really passionate about these things, but I am not going to read something which you haven't written. Even sharing a prompt/rough-sketches/raw-writing might be beneficial but I recommend writing it by-hand man, we are all burnt out reading AI slop, I can't read more AI
You're right, I used an LLM to help write it from sketches. Gonna rewrite it properly because I think the ideas are worth exploring. Thanks for taking the time to read and reply.
In my opinion, its okay to use LLM to help find some sources and then validating them (but I also recommend using hand researching too as you might find some good things that you maybe weren't even looking for!)
but, please don't use LLM to help write it from sketches. Even show the sketch :)
Much of my writing is very sketch-y. Some people don't like it, but its mine and I am proud of it and I hope that even if you write sketches/refine them, you can be comfortable sharing your ideas in your words in the way you wish to write them carl!
My thinking is that, I improve my writing by well... practice itself. So I write publically and there are some thoughts which occur in my head during the writing process itself (PG has a good article about it recently)
In a world of AI, to me, Human writing is a breath of fresh air. Please don't fall into the rabbit-hole that you might need LLM to help write you.
These are just my 2 cents though, but I feel like I am definitely not alone in thinking so.
Have a nice day and I am looking forward for you to write the article yourself. Feel free to share me when you do with my mail as I would love to read it, as I am also passionate about the funding of open source :)
yet another npm supply chain attack, these are becoming as ubiquitous as gun violence in the US.
We have become numb to it.
One of my tools, bruno, was impacted but seems to be limited to cli via npm install [1]
[1] https://github.com/usebruno/bruno/security/advisories/GHSA-6...
This is the part that's tough — we push everyone to keep dependencies updated and automate it with Renovate or Dependabot, but that's exactly the pipeline that would have pulled this in before anyone noticed. Lockfiles and pinning help slow it down, but most teams pair those with automated update PRs which kind of negates the point. You can reduce your dependency surface area to lower the odds but one compromised maintainer on a top-10 package and none of that matters.
Rejecting any packages newer than X days is one nice control, but ultimately it'd be way better to maintain an allowlist of which packages are allowed to run scripts.
Unfortunately npm is friggen awful at this...
You can use --ignore-scripts=true to disable all scripts, but inevitably, some packages will absolutely need to run scripts. There's no way to allowlist specific scripts to run, while blocking all others.
There are third-party npm packages that you can install, like @lavamoat/allow-scripts, but to use these you need to use an entirely different command like `npm setup` instead of the `npm install` everyone is familiar with.
This is just awful in so many ways, and it'd be so easy for npm to fix.
pnpm and bun have approved lists.
npm is just dragging it's feet - and stuff like this is why people moved to pnpm, yarn and bun in the first place.
It’s time to run development in sandboxes. Docker or sandbox-exec (for Mac).
# If you have a projects folder containing multiple projects on macOS, you can run this script to recursively scan all subfolders for vulnerable axios versions and the presence of plain-crypto-js, helping you quickly identify potentially affected projects:
find . -name "package.json" -exec sh -c ' dir=$(dirname "{}") echo "==== $dir ====" cd "$dir" npm list axios 2>/dev/null | grep -E "1\.14\.1|0\.30\.4" grep -A1 "\"axios\"" package-lock.json 2>/dev/null | grep -E "1\.14\.1|0\.30\.4" [ -d node_modules/plain-crypto-js ] && echo "POTENTIALLY AFFECTED" ' \;
This is the second major npm supply chain attack this year and the playbook is identical every time: hijack a maintainer account, publish via CLI to bypass CI/CD, inject a dependency nobody's heard of.
The fix isn't better scanning (though Socket catching it in 6 minutes is impressive). The fix is npm making Trusted Publishers mandatory for packages above a download threshold. If axios can only be published through GitHub Actions OIDC, a stolen password is useless.
We run a fleet of AI agents that depend on npm packages. First thing we did tonight was audit every lockfile. Clean — but only because we aggressively minimise dependencies. The real victims here are the thousands of teams who npm install with ^ ranges and never check what changed.
My main question here is mostly why so many people still rely on axios for their fetch implementation. Native fetch has been a thing in the JavaScript world for so long, and the DX gains to using axios over it are miniscule. The only thing I can think of is axios instances, but you can easily write a tiny wrapper for fetch that would do the same.
This is a genuine question - if you still use axios, why exactly?
FWIW I vibe coded https://github.com/astrostl/surplies to detect evidence of the Axios and LiteLLM malware, using StepSecurity's writeups as a data source.
CNN reports it was North Korean hackers. https://lite.cnn.com/2026/03/31/politics/north-korea-hacking...
That’s probably the most interesting part of this whole story and it’s nowhere to be found here
https://docs.npmjs.com/trusted-publishers/#recommended-restr... This helps mitigate spear phished privileges employees pushing hacked npm packages entirely.
The postinstall script vector is getting all the attention, but IMO the scarier part is how the attacker chain works: compromise one package's credentials, use that access to pivot to the next target. Trivy -> LiteLLM -> now potentially axios. Each compromised package becomes a credential harvester for the next round.\n\nThe min-release-age configs (now in npm, pnpm, bun, uv) are a good start, but they only work as herd immunity — you need enough early adopters installing fresh releases to trigger detection before the 7-day window expires for everyone else. It's basically a bet that security researchers will catch it faster than your cooldown period.\n\nFor Node specifically: if you're still using axios for new projects, it's worth asking why. Native fetch has been stable in Node since v21. One less dependency in your tree is one less attack surface.
I haven't seen a bot here insert a \n into a comment yet
Three hours between the malicious publish and npm pulling the versions. If your CI ran an install during that window, this went straight to prod. Most teams I've worked with still have loose version ranges somewhere in their dependency tree even if they think they've locked everything down.
So here is the pitch: for npm / a new registry
1. Only the registry itself can build packages (only source provided) 2. Builds must be reproducable (no network or external files during build / publish) 3. New versions are hidden by default 4. Releases can only be published by an account, using a hardware 2fa token + password (no persistent login, no long lasting token) 5. All commits must be signed (maybe block web commits or add a cooldown of a few days?) 6. builtin scanners (using ai, virustotal, existing services) 7. if a security violation is found the version is instantly removed 8. Atleast 1 - 3 Days delay for releases 9. Hard no on binaries / post install scripts and binary data 10. blockchain like public record to see who published, updated, owns what
GitHub PR links
Axios has been one of my favorite libraries for over a decade. However, I must say I have never used it since starting vibe coding.
I wrote a tool that helps you check if your machine was compromised: https://github.com/aeneasr/was-i-axios-pwned