The current title is "Rubygems.org compromised." Isn't this just a gem that exploits the YAML bug, not an exploit on RubyGems.org? I believe this submission needs a title change.
Edit: I am incorrect. It seems that this uploaded gem was used to compromise RubyGems.org.
Yes, but the attacker would need to inject code into every .gem file that's pushed to rubygems.org. Because that source code would be highly visible, it would be simple to figure out what they're trying to do.
A more hidden attempt would be to simply alter the existing .gem files on S3. Once they have the access tokens from the configuration YAML files, they can do that from anywhere, not just the rubygems.org host. That keeps their activity under the radar and is slightly harder to track.
In that case, only newly installed gems in your system that were uploaded to rubygems.org before this incident would be at risk. If you run bundle update and pull in gems built from today onward, there is not as much to worry about. If you're updating gems and the bundle is pulling in gems from a few days ago, then you are at high risk.
I'm not disputing anything here, but I just want to emphasize that if they have the S3 credentials, replacing any gem is trivial. And they wouldn't need to replace every gem. Picking the 10 most popular ones would have pretty big fallout.
And, if like most people you don't verify your installed gems (not that you really can if they're not signed), you're going to have a very bad day. The "tracking" aspect is a situation where it doesn't matter because at that point, you're hosed anyway.
Also, walking S3 is not particularly fast. So if the plan is to walk all gem files to verify them, expect that to take a while. Hopefully they have bucket logging enabled.
We don't know the extent to which Rubygems was compromised yet. In a scenario like this, it's best to err on the safe side and avoid trusting the source for the time being.
Mind explaining to me how this shows that rubygems was compromised? It looks to me like someone just uploaded a gem called "exploit" which takes advantage of a recent exploit.
In Ben's defence, I doubt he's the only one who wouldn't have thought to check the whois data. I wouldn't have, for a start. There should have been an obvious contact email linked on the site itself for issues, not just a Twitter account.
I am not sure what this is getting down voted? This was not a dig at Ben. I was trying to be helpful, so they he could get in touch with the admin contact?!
It's disappointing to see this show up on HN without contact myself or the other rubygems.org admins. We've put rubygems.org into maintenance mode until the issue is fixed.
What I saw was Ben twerping to Ruby people asking who to contact about this first, then posting to HN when he couldn't find the right person. Can I ask, where's the security page on Rubygems.org that tells people who to alert when stuff like this happens? I'm trying to find it on Archive.org and drawing a blank.
If the whole world is pulling backdoored gems off a compromised Rubygems.org, panic seems like a reasonable first reaction. This isn't the Rails bug; there was nothing to be gained here from secrecy.
I became aware of this problem this morning when it was discussed on a public github issue. I think the problem has been known for a couple of weeks. I checked latest rack and active_support gem (would seem to be good targets...) to see if they had been backdoored but they were clean. Not sure if these gems are still clean or if any others have been compromised.
Now would be a good time to build up a quick script to compare the last modification time of the files on S3 against the updated_at of any gem record in the system. If the delta is too long, that would be a candidate for deeper investigation.
Ah, so were were supposed to look at the .yml files included in your exploit gem? Which contained some injected code that would then run on the rubygems server?
to see if they had been backdoored but they were clean
You can never be too sure. There's a whole contest (http://underhanded.xcott.com/) whose goal is "make something that looks legit, but actually does something evil / has an exploit).
If someone compromised a package repository and replaced OpenSSH with a backdoored version, I would want to know immediately, and relay that message as quickly as possible to as many other users as possible before it spreads.
It is better to be paranoid about it and get the word out before someone actually gets hurt. If the users get to it before the maintainer gets to it, at least someone got the word out.
Did you catch where was this payload injected? Did they have write access to the rubygems.org source? It would seem if they did, they didn't need to write this script at all and could simply read the configuration files directly.
After reading through the github.com issue posted in the comments, I get the impression this was targeted attack at rubygems.org (and not the user install base). There has been a tweet, also posted in the comments, suggesting that this was successful.
Even if most gem developers would understand, there's still a problem with the trust model: If you choose "high security", you cannot install a single gem without signature - so things won't work unless _all_ authors sign their gems. Gem signing would have to be a requirement, similar to packages on ubuntu's ppa. If you choose a lower security setting, a gem may just be repackaged and signed again by the attacker or the signature removed.
A solution that doesn't solve 100% of the problem is still not a bad solution. The most popular gems would all be signed, thereby dramatically reducing attack vectors.
No, as I said, this doesn't help. You would need to sign the most popular gems and all their dependencies and the dependencies dependencies. Given that a pretty much standard rails app can easily pull in hundred or more gems that's quite a bit of work. It's doable, but I only see that coming if for example rubygems requires a key and gem signatures.
Then there's still the unsolved problem of providing key trust - the current implementation relies on self-signed certificates.
And gem/bundler warn if the signature is not present or invalid?
The problem seems to be that work on the trust infrastructure has been dormant since 2011. The basic signing functionality is in place[1], but it's not enforced and consequently nobody is using it.
I hope this event will remind the people in charge to finish what they started. Otherwise it's a matter of time until something Really Bad™ happens. The code underlying rubygems[.org] is the stuff nightmares are made of (Marshal...).
It is very possible that something Really Bad™ has already happened, and we don't know about it. If it were me, I wouldn't let it out that I had essentially an unlimited backdoor to every system that installs gems.
On a related note, we are working on adding this requirement to Clojars, the Clojure equivalent of Rubygems. It's a lot of work and requires significant community buy-in, but would really pay off in moments like these.
When rubygems.org requires signed gems by developers whose keys have been put into the approved keyring, and denying access to everyone else. Until then, nobody will bother.
No, you miss the point. The idea is that if, say, Rails were signed, someone wiht an exploit couldn't attack rubygems to _modify rails_ without it being discovered, because they wouldn't be able to sign their modified rails.
The point of signing gems is not that any signed gem is neccesarily trustworthy. It's that any signed gem is neccesarily what the signature owner distributed, and has not been modified by someone else since.
But to make that so requires a bunch of things, it's not quite as simple as 'everyone just has to sign their gems'. But that's the idea.
The trouble is self-signed X.509 certificates are pretty worthless. X.509 certs are built around the concept that they're issued by a trusted provider.
I tried to get together a proposal to introduce OpenPGP signed gems and wrote some proof-of-concept code so there could be a proper Web-of-Trust for rubygems:
But the developers were resistant (and understandably so) to having gpg as a dependency for gem verification, and I didn't trust myself to write the crypto and a decent keyring validation algorithm from scratch.
Even with pgp you'd need a whole lot of infrastructure. Where to get the public key from etc. Ubuntu requires pgp keys for their ppa and provides a keyserver. Since a lot of gems seem to use github nowadays, maybe an option to provide a pgp key on github would be great. That would allow to trust at least the most interesting projects - rails, db-layers, rack, etc. which could limit the fallout of an attack against rubygems.org
My proposal was that rubygems would create a simulated certificate authority that operates similarly to the PGP Global Directory, which would allow registered users to have their keys signed by an authorized RubyGems.org signing key. (This is covered on the bottom of the page on the link above.)
In addition more active developers who go to conferences and what not would do all the proper WoT stuff to ensure that the RubyGems.org key being used was indeed the authentic one, and that gems in their circle of friends/co-workers were signed by the correct keys.
Yes, in general I think the proposal is sound. I just wanted to add that GH might be in a position to add "trust" quickly since the accounts already contain a little bit of information. Many people there have a paid account and so are at least known by credit card. It's not much and by no means perfect, but it's more than we have now.
The trouble is self-signed X.509 certificates are pretty worthless.
I disagree. If you obtain the public key from a second location that is not rubygems.org (e.g. from gem.homepage) then that raises the bar for a potential attacker. He now has to either compromise two sites in order to inject a malicious gem, or trick you into obtaining the public_key from a forged location.
I hope 'bundler' and 'gem' hear the wakeup-call and will start displaying a visible warning on unsigned gems (please also print the homepage-url to make it less of a pain for the user to hunt down the public key). After a grace period the default should be switched to -P HighSecurity (reject unsigned gems).
Yes, it's is not perfect, users may still import keys without thoroughly checking that the source is legit. However, it'll be a huge improvement over what we have now and greatly reduce the chance of a compromised gem to go unnoticed.
Point taken, but my day-job rails project, which isn't even that big, has 461 gems in my Gemfile.lock. (which lists all the explicitly included gems, as well as their dependencies, if you're not familiar.) That is a lot of self-signed certificates to track down, attempt some form of verification, and manually add as trusted.
My proposal would allow people who want a medium-level of security to verify that whoever uploaded the gem has (1) valid credentials to rubygems.org, and (2) control of the account owner's email, without requiring all that manual installation.
And it would allow people who wanted to set higher standards of verification to do so, basically for free, by configuring their settings in gpg.
I think your approach is problematic because it hinges on the integrity of rubygems.org (the very host that is compromised right now).
I do see your concern about having a large number of gems, but I believe that is a bootstrapping problem which could be relatively easily mitigated.
Trusted entities (rubygems team, 37signals etc.) could publish signed versions of their keyrings, i.e. public_keys that they trust. So, if you trust 37signals, you'd just download their ring, validate their signature, and import it. And that will probably cover most of your 400 gems.
Debian and Ubuntu solve this problem by having an archive signing key that is regularly rolled over and requires multiple people to re-assemble. The entire archive is cryptographically signed by this archive signing key, and each of the individual pacakges that are made up of that archive have been signed by the keys of the people who have been approved to be part of the keyring. This provides a cryptographic verification model that goes all the way up allowing you to verify each package comes from the right place. This would be a huge improvement over what there is now.
No, it doesn't solve every problem in the universe, but let us use this opportunity to improve the infrastructure.
We're basically on the same page, but coming from different directions.
Once you start having people import lists of trusted X.509 signatures from other trusted sources (e.g. 37 Signals) what you end up doing is basically creating the Web Of Trust that already exists in OpenPGP.
That is, instead of simulating a X.509 Certificate Authority in OpenPGP like I want to do, you're now simulating a OpenPGP web of trust in X.509 land.
I would have the rubygems OpenPGP key be the default for people who would want moderate security. Although a compromise could affect new users, all existing user would have the old key when they installed rubygems. That would immediately set off flags when gems signed by a forged key only were installed by the millions of users with the uncompromised install.
In addition, the more security conscious could trust for example 37signals or a co-worker and create a proper Web of Trust for validation.
This is the way OpenPGP works by default. You can even different trust levels automatically with this approach. Simply a rubygems.org signature would make the gem moderately trusted. Two additional signatures from two other trusted sources (37signals and a co-worker) would make the gem show up as highly trusted.
But rather than manually importing in an ad hoc fashion, you just sign the 37signals key locally, and gpg already takes care of enforcing all the rules, building the chain of trust, publishing peoples signatures to keyservers distributed world-wide, performing security calculations based on a user's custom preferences, etc.
So if you want the web-of-trust approach, best to just start with OpenPGP, instead of trying a NIH approach with X.509 certs.
Your approach is definitely more complete but also seems to require fairly substantial code changes (OpenPGP?).
My approach should be doable with a mere ~10 lines patch; Gem and bundler should default to '-P HighSecurity' and rubygems.org should reject pushes of unsigned gems.
Anyway, whichever implementation strategy they choose, I hope they will do it soon.
You can deploy X.509 in the same configurations as you can OpenPGP; X.509 is just a certificate encoding format, and the semantics are up to the application .
Ruby has better support for X.509 PKI than it does for OpenPGP, so using OpenPGP seems kind of silly in this case.
Yep, you can do WoT with X.509 if you write the semantics from scratch, and you can also do a CA approach with OpenPGP.
The thing is, for years no-one has stepped in and actually gone through the pesky little details of implementing all those semantics for rubygems via X.509 even though it's come up repeatedly. But maybe someday some expert will come along and fix things.
If you actually read the rubygems documentation on signing gems, it's pretty much "Okay generate your own self-signed key, then sign it with your self-signed key." That leaves a lot to be desired vs best practices. There isn't even an attempt to describe how you get your self-signed keys to users.
I just think leveraging the existing OpenPGP semantics, ecosystem, and battle-hardened gnupg codebase, is a lot easier than implementing an entire system from scratch on X.509 just because openssl is already installed everywhere and ruby has better bindings and gnupg is not and does not.
I'm just trying to take the approach you usually preach: Don't roll your own crypto code and leave the real work to the pros, and it seems like gpg offers a lower path of resistance.
Using OpenPGP is a bad idea here. All the mainstream rubies have access to high-quality X.509 in their mainline distros without a third-party gem.
Both OpenPGP and X.509 are going to require a little of fiddley semantic details that will take time to hash out. The bottleneck here isn't the technology.
I like, and routinely recommend, OpenPGP for solving app crypto problems. X.509 makes a lot more sense here.
can someone explain what happened here to those like me who are a little late. seems like the issue is resolved already or at least temporarily fixed.
All I can figure out from the url and the comments here is that there was a gem called exploit and that somehow the config files where posted to pasties.org
Edit: At least a PoC. How someone other than him (since hes denying compromising rubygems) found his PoC and wrote the actual payload and pushed the exploit gem is unclear.
1 week is too long for such a severe security issue. A proof-of-concept gem seems to be a logical step to "force" rubygems.org to fix this issue. However, publishing database.yml on pastie is TOTALLY 100% NOT COOL, and turns this from "whitehat who wants to help" to "blackhat that wants to destroy (or doesn't understand social norms)".
However, publishing database.yml on pastie is TOTALLY 100% NOT COOL ...
This whole thing is a joke, from the ridiculously unsafe YAML parser to the response of the RubyGems people.
If these are your community social norms, then you quite simply deserve the resulting shake-up. This is how you become inoculated against such incredibly poor engineering practices.
I don't quite get what you're trying to say. What "community social norms"? Are you talking about how rubygems.org handled this vulnerability from the beginning? How the community handled this incident after it happened?
And what does it have to do with my quote? I was just pointing out that in general I'm for PoC to highlight security issues when the vendor doesn't respond, but stealing database.yml makes you nothing more than a simple crook.
A full post-mortem is coming, but here's what happened from my perspective. I got mentioned on twitter about this thread while I was on the bus. I asked @evanphx to put the site into maintenance mode immediately.
Once I got on wifi, I downloaded the "exploit" gems and found that they used the Psych hack to post config/initializers/* and config/*.yml to Pastie. Luckily, none of our API keys were actually in those files - our config is a bit different since you can still run RubyGems.org on Heroku, which needs to put secret keys in the ENV.
I deleted the exploit gems permanently along with the throwaway account that posted them...that's why this post's URL is a 404.
I've reset our S3 keys just to be sure and any other keys we had on our production site. From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure. Once we get a real fix for this issue out, pushing gems will be enabled again.
Just a general PSA -
Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.
Yes, and thanks for telling us. Really, we should have disabled gem pushes immediately. Hindsight is 20/20, and I'm not sure why I didn't think of doing so earlier. The pain of not being able to push gems would have forced us to fix it.
I'm sorry about this. I don't know what else to say. I wish we didn't have to deal with this kind of problem in the Ruby community.
Respectfully, RubyGems has already been compromised. Not telling anyone has zero advantages to RubyGem users.
Telling people to avoid using RubyGems and to audit if they've used it recently helps protect users.
Responsible disclosure is about avoiding giving bad guys new attack vectors and ensuring that the maintainer has the best chance of fixing the issue before a blackhat discovers and exploits it. That doesn't apply in this case.
I agree, but how do people audit? I think it's time to start putting application dependencies, like rubygems and npm modules, under version control, not in the same repo, but in a different repo.
Mikeal Rogers has some good reasons why, but there gets to be too much commit noise if you have big dependencies, and put them in the same repo. A separate repo solves this.
Is this maybe an opportunity to re-evaluate the volunteer only aspect of RubyGems? I love the service (as I'm sure thousands of others do) and I'd happily pay for a subscription model that would fund a full-time developer/maintainer for the service. Based on how critical it is to the Ruby community I can't imagine that I'm alone in thinking this.
Hmm. That's for private gems. Interesting, but a different use-case. I'd chip in to pay for someone to maintain "rubygems.org" on a professional basis. It's awesome that some people take their private time to support such an essential service, but I'm with the grandparent that it might be good if it were run on a paid-for basis. Maybe through donations or similar, so that access is still free.
CPAN is all volunteer. We happen to have a few really good SAs volunteer to keep core infrastructure up (which PAUSE, our upload server, just moved onto) ... but it's still volunteer.
Hi Nick, instead of being a dick, I want to first of all thank you for understanding the urgency (despite being on a public bus, you took the pains to get it temporarily fixed somehow). Like others, I do agree that posting on HN does more good than bad, but at the same time, I do understand that it is also polite to inform the maintainers too (which I think in your case they didn't.). Anyway, thank you for being a responsible maintainer!
Once again, Thank you for maintaining rubygems.org! You're doing a great job :)
Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.
It'd probably go a long way if you guys considered making an easily-discoverable security page with information on who to contact, how you prefer that contact to take place and how long they should expect to have to wait to hear back from you.
If you want security notices to go to the Rubygems team, you have to set up a security page that tells people how to do that. Like everyone else, I appreciate your volunteer work, but no amount of goodwill creates the ability for people to read your mind.
Please post a security page. It literally doesn't need anything more than an email address and a PGP public key.
By the way: if Rubygems needs security help, it looks like there's quite a number of people who are willing to pitch in. When I published a FreeBSD crt0 bug back in 1996, I was given commit privs. I thought that was a pretty effective way to co-opt adversarial researchers.
I know dealing with this stuff is no fun. Try to keep in mind, though, that people like Ben Murphy and Hal (Postmodern) are on your side. Again, maybe consider formalizing that relationship a little!
The Rubygems team (or even just the person who writes the security page) can just generate a new one, for "security@rubygems.org" or whatever, and it can be shared by the team.
The purpose of the key is to allow people to report security vulnerabilities without worrying that by doing so they're giving ammunition to people snooping emails.
Maybe I dont understand PGP and this is a stupid question, but if the site is compromised, would it not be possible for them to just put a different key up? One which they knew the private key for? How would you know the public key on the compromised site has not been changed unless you could compare it to the PGP key from before the site was compromised?
You can't, but having a security.html page gives attackers the possibility to contact you privately and securely; this reduces the chance that they'll decide to "contact" you by completely pwning your public site.
What if the key is endorsed by the individual people on the team? Their personal keys, which they keep, must sign the security team key. A replacement cannot have this endorsement.
You don't want to send an actual exploit in clear text to a possibly compromised email account. PGP encrypts, and provides some level of validation that the message can only be read by someone on the security team.
We have a great many people relying on a volunteer run project here. It would be great to have someone sponsored to spend time maintaining the site.
Unfortunately I don't see any way to help fund such an effort. Is there an officially supported way to donate to RubyGems? Can we create a pool to help cover the cost of maintaining this resource?
There is some larger organizations using Ruby. So not hiring a guy full-time, but they might be interested in chipping in, to get a gem-repo that they can trust.
I also get an "invalid SSL certificate" warning when I try to visit rubygems.org. Here is the message in Firefox:
rubygems.org uses an invalid security certificate.
The certificate is not trusted because the issuer
certificate is unknown.
(Error code: sec_error_unknown_issuer)
Why is it that all these parsing issues with yaml and xml were ignored before it all blew up recently? Input parsing is usually the most venerable part of any application, and it's strange that these issues were ignored for so long. These are not super complicated exploits, they are using the _expected_ behavior of the yaml parser in terms of it constructing arbitrary objects.
Well, a unsafe parser (which was designed only for parsing trusted input) was being used by lots of people (including Rails) for parsing untrusted input as if it was a safe parser. You can debate about whether there should be a safe parser for YAML, but that's a separate issue.
Agreed, there should be a clear warning - in fact, the load method should be renamed unsafe_load. The root cause of this is probably unclear documentation and misunderstandings between the users and authors of psych.
Where are the adults? This is absolute amateurism. Why haven't Heroku or GitHub or some actual professionals in the Ruby community, stepped up to ensure that this kind of amateurism doesn't damage their ecosystem?
[Throwaway as I'm a small-fish in the Ruby community. I expect I'll get to use this account again soon though]
The response from the Gem folks has betrayed a serious lack of experience.
Has Rails been such an insular and young community for so long that there are truly no qualified engineers around with experience in this level of professional software development?
The problem is any YAML loader which supports tags that can name arbitrary classes and have existent classes whose initialization may `eval` the supplied values. Ruby is very publicly hurting, but equivalent vulnerabilities exist in other runtimes, such as Perl via [YAML::Syck](http://search.cpan.org/~toddr/YAML-Syck-1.22/lib/YAML/Syck.p...) or Python via [PyYAML](http://pyyaml.org/wiki/PyYAML) and [LibYAML](http://pyyaml.org/wiki/LibYAML). If your language can accept and load untrusted yaml and your language has `eval`, you're probably fucked and just don't know it yet.
The problem is that YAML is used by Rails & Co as a parser for untrusted user-provided data. It does not matter if there is buggy implementation in Python/Perl if nobody use them.
If other framework were using YAML in this fashion, yesterday will be the time to look for security holes in those implementations.
Why aren't gems protected by a trust chain all the way up like Debian does? Gems aren't even required to be cryptographically signed by the developer and from what I can tell are just thrown over http without any authentication. This seems terrible.
Oh the timing. So, there was a big meeting, where some of the devs pleaded the case that the recent wave of security issues in the Ruby world were a tipping point, and that we were over it. They presented a good case, and I think the architects were close to permitting the usage of Ruby and Rails in their current cases.
Then we get an interruption, as a couple of projects had updated their gems today, and there was a concern about overall security of the apps.
IT Security required the apps were offlined, and the green light was given to the PHP and Python guys in house (and a few retained developers) to start the process of rewriting the apps.
It will be painful, but, I think, the business has lost it's appetite for the constant Ruby idea of "Let's run with Yaml", before firstly "Let's walk safely with Yaml".
I'm glad to hear that at least one organization has done the right thing, and started the process of moving completely away from Ruby and Ruby on Rails. It's just the responsible and sane thing to do, given how the software and the community apparently can't be trusted.
I can understand it's bad timing, but the reaction seems ad-hoc. AFAIK neither python eggs nor PHPs packagist repository sign their packages, so in theory they're open to the same attack vector [1]. Judging from the docs, composer doesn't even allow signed packages. A better reaction would have been to fix the process: Allow only signed gems or cache and audit the packages you install. You don't need to depend on rubygems.
[1] Even OS package repositories or mirrors were broken into. I faintly remember a break-in where debian(?) had to check all their mirrors.
For those that don't know how this exploit worked, I think the "exploit" gem description provides a pretty good explanation:
'A Proof-of-Concept PoC gem that exploits a vulnerability in the Psych YAML parser, which allows the #[]= method to be called on arbitrary Objects.
If the #[]= method later calls eval() with the given arguments, this allows for
arbitrary execution of code.'
Jeeze, bad things happen (or could) to the greatest services ;) I'm late to the discussion, didn't have time to lurk around, nevertheless what I can agree the most is about gem signing. A gem is a package of software, and every single package distributor makes the author sign their work (think linux packages, being a Debian/Ubuntu developer I can tell you that if you don't get your package signed, it won't get uploaded, period, your key in some cases (Debian) needs to be verified in person to build a circle of trust. I think that besides the overhead (initially, I'm sure people will contribute scripts or integrate this so that the developer won't notice) this is really a needed step you guys should take.
I'm sorry to mention some RG projects are still "potentially" dangerous: https://gemnasium.com/rubygems
We'll do our best to help determining if these projects really need to be updated asap!
The current title is "Rubygems.org compromised." Isn't this just a gem that exploits the YAML bug, not an exploit on RubyGems.org? I believe this submission needs a title change.
Edit: I am incorrect. It seems that this uploaded gem was used to compromise RubyGems.org.
I think you're right, the title is inaccurate.
Running `$ bundle update` will not inject this into your app.
You'd have to intentionally add `gem "exploit", "~> 22.31.31"` to your Gemfile.
Couldn't all gems fetched from Rubygems be infected?
Yes, but the attacker would need to inject code into every .gem file that's pushed to rubygems.org. Because that source code would be highly visible, it would be simple to figure out what they're trying to do.
A more hidden attempt would be to simply alter the existing .gem files on S3. Once they have the access tokens from the configuration YAML files, they can do that from anywhere, not just the rubygems.org host. That keeps their activity under the radar and is slightly harder to track.
In that case, only newly installed gems in your system that were uploaded to rubygems.org before this incident would be at risk. If you run bundle update and pull in gems built from today onward, there is not as much to worry about. If you're updating gems and the bundle is pulling in gems from a few days ago, then you are at high risk.
I'm not disputing anything here, but I just want to emphasize that if they have the S3 credentials, replacing any gem is trivial. And they wouldn't need to replace every gem. Picking the 10 most popular ones would have pretty big fallout.
And, if like most people you don't verify your installed gems (not that you really can if they're not signed), you're going to have a very bad day. The "tracking" aspect is a situation where it doesn't matter because at that point, you're hosed anyway.
Also, walking S3 is not particularly fast. So if the plan is to walk all gem files to verify them, expect that to take a while. Hopefully they have bucket logging enabled.
it looks like someone has tried to steal rubygems.org db and other credentials. bundle update is now dangerous because you might get trojaned files.
But don't applications have to explicitly install this gem? If not, then I am incorrect.
We don't know the extent to which Rubygems was compromised yet. In a scenario like this, it's best to err on the safe side and avoid trusting the source for the time being.
Qrush, one of the guys behind Rubygems tweeted that the credentials of every yml file ended up on pastie: https://twitter.com/qrush/status/296638346283868160
Mind explaining to me how this shows that rubygems was compromised? It looks to me like someone just uploaded a gem called "exploit" which takes advantage of a recent exploit.
I'd like to know this as well.
It has been put into maintenance mode, so that is what you see now, until this is resolved.
It kind of says something when people can't tell the difference between your maintenance mode page and a hacked page...
Should he design a custom "We have been compromised" page before deactivating the site?
It was put in maintenance to stop pushing possibly infected gems to users.
If https://twitter.com/qrush/status/296638346283868160 is accurate (qrush is one of the rubygems maintainers), it worked.
The polite thing to do first would be to notify the maintainers of rubygems.org - not post the link to HN.
I didn't know who maintainers were :( I think i've notified one now.
$ whois Rubygems.org
Tech Email: ...
In Ben's defence, I doubt he's the only one who wouldn't have thought to check the whois data. I wouldn't have, for a start. There should have been an obvious contact email linked on the site itself for issues, not just a Twitter account.
This probably would be the first time in the past 5 years there was a non-spamtrap email address in whois
I am not sure what this is getting down voted? This was not a dig at Ben. I was trying to be helpful, so they he could get in touch with the admin contact?!
Don't sweat it. Getting the word as best you could was a good thing to do.
There's evidence that Rubygems.org has been compromised and you think the best course of action might be to keep that secret for a little while?
It's disappointing to see this show up on HN without contact myself or the other rubygems.org admins. We've put rubygems.org into maintenance mode until the issue is fixed.
What I saw was Ben twerping to Ruby people asking who to contact about this first, then posting to HN when he couldn't find the right person. Can I ask, where's the security page on Rubygems.org that tells people who to alert when stuff like this happens? I'm trying to find it on Archive.org and drawing a blank.
I looked for contact page first then panicked. I should have used whois :(
If the whole world is pulling backdoored gems off a compromised Rubygems.org, panic seems like a reasonable first reaction. This isn't the Rails bug; there was nothing to be gained here from secrecy.
I became aware of this problem this morning when it was discussed on a public github issue. I think the problem has been known for a couple of weeks. I checked latest rack and active_support gem (would seem to be good targets...) to see if they had been backdoored but they were clean. Not sure if these gems are still clean or if any others have been compromised.
consider me a fan of your work!
Now would be a good time to build up a quick script to compare the last modification time of the files on S3 against the updated_at of any gem record in the system. If the delta is too long, that would be a candidate for deeper investigation.
Hopefully they use S3 bucket versioning. Rolling back would be pretty easy then.
I'd be interested in seeing this github issue. Link please?
https://github.com/tenderlove/psych/issues/119#issuecomment-...
Ah, so were were supposed to look at the .yml files included in your exploit gem? Which contained some injected code that would then run on the rubygems server?
to see if they had been backdoored but they were clean
You can never be too sure. There's a whole contest (http://underhanded.xcott.com/) whose goal is "make something that looks legit, but actually does something evil / has an exploit).
Nuke from orbit.
i diffed the rack gem and active_resource gem from the versions built from git. i'm fairly sure the versions i downloaded were not compromised.
If someone compromised a package repository and replaced OpenSSH with a backdoored version, I would want to know immediately, and relay that message as quickly as possible to as many other users as possible before it spreads.
It is better to be paranoid about it and get the word out before someone actually gets hurt. If the users get to it before the maintainer gets to it, at least someone got the word out.
Your users need to know if their systems have been compromised .
Here's what the payload looked like: https://gist.github.com/3e4829f79dbd1be11295
Looks like this would create a pastie.org entry looking something like the this:
Searching pastie.org, I don't see any such entries.
These params
is just hardcoded pastie.org arguments that gives them somekind of safety against the stupidest attempts.
Maybe the post was removed, or the hash is different than the original so not all can find it.
I didn't change the hash, though I would have if I'd found the post on Pastie.
Did you catch where was this payload injected? Did they have write access to the rubygems.org source? It would seem if they did, they didn't need to write this script at all and could simply read the configuration files directly.
After reading through the github.com issue posted in the comments, I get the impression this was targeted attack at rubygems.org (and not the user install base). There has been a tweet, also posted in the comments, suggesting that this was successful.
It was in the gem metadata, which is YAML: https://gist.github.com/d891e876c53e55bf0920
rubygems.org is back up
When will people start signing their gems? And gem/bundler warn if the signature is not present or invalid?
I'd wager that most gem developers don't understand how it works, and there are basically no resources out there on it.
Here's the chapter on it in the RubyGems manual:
http://docs.rubygems.org/read/chapter/21
And that's all that I can find. Try to find something else, like a blog tutorial.
Even if most gem developers would understand, there's still a problem with the trust model: If you choose "high security", you cannot install a single gem without signature - so things won't work unless _all_ authors sign their gems. Gem signing would have to be a requirement, similar to packages on ubuntu's ppa. If you choose a lower security setting, a gem may just be repackaged and signed again by the attacker or the signature removed.
A solution that doesn't solve 100% of the problem is still not a bad solution. The most popular gems would all be signed, thereby dramatically reducing attack vectors.
No, as I said, this doesn't help. You would need to sign the most popular gems and all their dependencies and the dependencies dependencies. Given that a pretty much standard rails app can easily pull in hundred or more gems that's quite a bit of work. It's doable, but I only see that coming if for example rubygems requires a key and gem signatures.
Then there's still the unsolved problem of providing key trust - the current implementation relies on self-signed certificates.
And gem/bundler warn if the signature is not present or invalid?
The problem seems to be that work on the trust infrastructure has been dormant since 2011. The basic signing functionality is in place[1], but it's not enforced and consequently nobody is using it.
I hope this event will remind the people in charge to finish what they started. Otherwise it's a matter of time until something Really Bad™ happens. The code underlying rubygems[.org] is the stuff nightmares are made of (Marshal...).
[1] http://docs.rubygems.org/read/chapter/21
It is very possible that something Really Bad™ has already happened, and we don't know about it. If it were me, I wouldn't let it out that I had essentially an unlimited backdoor to every system that installs gems.
On a related note, we are working on adding this requirement to Clojars, the Clojure equivalent of Rubygems. It's a lot of work and requires significant community buy-in, but would really pay off in moments like these.
https://groups.google.com/group/clojars-maintainers/browse_t...
When rubygems.org requires signed gems by developers whose keys have been put into the approved keyring, and denying access to everyone else. Until then, nobody will bother.
Won't make a difference, it's possible to sign a gem with a metadata file that contains an exploit.
No, you miss the point. The idea is that if, say, Rails were signed, someone wiht an exploit couldn't attack rubygems to _modify rails_ without it being discovered, because they wouldn't be able to sign their modified rails.
The point of signing gems is not that any signed gem is neccesarily trustworthy. It's that any signed gem is neccesarily what the signature owner distributed, and has not been modified by someone else since.
But to make that so requires a bunch of things, it's not quite as simple as 'everyone just has to sign their gems'. But that's the idea.
Is there anything gem authors should be doing?
Sign your gems before pushing them: http://docs.rubygems.org/read/chapter/21
The trouble is self-signed X.509 certificates are pretty worthless. X.509 certs are built around the concept that they're issued by a trusted provider.
I tried to get together a proposal to introduce OpenPGP signed gems and wrote some proof-of-concept code so there could be a proper Web-of-Trust for rubygems:
https://github.com/grant-olson/rubygems-openpgp
But the developers were resistant (and understandably so) to having gpg as a dependency for gem verification, and I didn't trust myself to write the crypto and a decent keyring validation algorithm from scratch.
Even with pgp you'd need a whole lot of infrastructure. Where to get the public key from etc. Ubuntu requires pgp keys for their ppa and provides a keyserver. Since a lot of gems seem to use github nowadays, maybe an option to provide a pgp key on github would be great. That would allow to trust at least the most interesting projects - rails, db-layers, rack, etc. which could limit the fallout of an attack against rubygems.org
My proposal was that rubygems would create a simulated certificate authority that operates similarly to the PGP Global Directory, which would allow registered users to have their keys signed by an authorized RubyGems.org signing key. (This is covered on the bottom of the page on the link above.)
In addition more active developers who go to conferences and what not would do all the proper WoT stuff to ensure that the RubyGems.org key being used was indeed the authentic one, and that gems in their circle of friends/co-workers were signed by the correct keys.
Yes, in general I think the proposal is sound. I just wanted to add that GH might be in a position to add "trust" quickly since the accounts already contain a little bit of information. Many people there have a paid account and so are at least known by credit card. It's not much and by no means perfect, but it's more than we have now.
The trouble is self-signed X.509 certificates are pretty worthless.
I disagree. If you obtain the public key from a second location that is not rubygems.org (e.g. from gem.homepage) then that raises the bar for a potential attacker. He now has to either compromise two sites in order to inject a malicious gem, or trick you into obtaining the public_key from a forged location.
I hope 'bundler' and 'gem' hear the wakeup-call and will start displaying a visible warning on unsigned gems (please also print the homepage-url to make it less of a pain for the user to hunt down the public key). After a grace period the default should be switched to -P HighSecurity (reject unsigned gems).
Yes, it's is not perfect, users may still import keys without thoroughly checking that the source is legit. However, it'll be a huge improvement over what we have now and greatly reduce the chance of a compromised gem to go unnoticed.
Point taken, but my day-job rails project, which isn't even that big, has 461 gems in my Gemfile.lock. (which lists all the explicitly included gems, as well as their dependencies, if you're not familiar.) That is a lot of self-signed certificates to track down, attempt some form of verification, and manually add as trusted.
My proposal would allow people who want a medium-level of security to verify that whoever uploaded the gem has (1) valid credentials to rubygems.org, and (2) control of the account owner's email, without requiring all that manual installation.
And it would allow people who wanted to set higher standards of verification to do so, basically for free, by configuring their settings in gpg.
I think your approach is problematic because it hinges on the integrity of rubygems.org (the very host that is compromised right now).
I do see your concern about having a large number of gems, but I believe that is a bootstrapping problem which could be relatively easily mitigated.
Trusted entities (rubygems team, 37signals etc.) could publish signed versions of their keyrings, i.e. public_keys that they trust. So, if you trust 37signals, you'd just download their ring, validate their signature, and import it. And that will probably cover most of your 400 gems.
Debian and Ubuntu solve this problem by having an archive signing key that is regularly rolled over and requires multiple people to re-assemble. The entire archive is cryptographically signed by this archive signing key, and each of the individual pacakges that are made up of that archive have been signed by the keys of the people who have been approved to be part of the keyring. This provides a cryptographic verification model that goes all the way up allowing you to verify each package comes from the right place. This would be a huge improvement over what there is now.
No, it doesn't solve every problem in the universe, but let us use this opportunity to improve the infrastructure.
We're basically on the same page, but coming from different directions.
Once you start having people import lists of trusted X.509 signatures from other trusted sources (e.g. 37 Signals) what you end up doing is basically creating the Web Of Trust that already exists in OpenPGP.
That is, instead of simulating a X.509 Certificate Authority in OpenPGP like I want to do, you're now simulating a OpenPGP web of trust in X.509 land.
I would have the rubygems OpenPGP key be the default for people who would want moderate security. Although a compromise could affect new users, all existing user would have the old key when they installed rubygems. That would immediately set off flags when gems signed by a forged key only were installed by the millions of users with the uncompromised install.
In addition, the more security conscious could trust for example 37signals or a co-worker and create a proper Web of Trust for validation.
This is the way OpenPGP works by default. You can even different trust levels automatically with this approach. Simply a rubygems.org signature would make the gem moderately trusted. Two additional signatures from two other trusted sources (37signals and a co-worker) would make the gem show up as highly trusted.
But rather than manually importing in an ad hoc fashion, you just sign the 37signals key locally, and gpg already takes care of enforcing all the rules, building the chain of trust, publishing peoples signatures to keyservers distributed world-wide, performing security calculations based on a user's custom preferences, etc.
So if you want the web-of-trust approach, best to just start with OpenPGP, instead of trying a NIH approach with X.509 certs.
Yup, we seem to agree in principle.
Your approach is definitely more complete but also seems to require fairly substantial code changes (OpenPGP?).
My approach should be doable with a mere ~10 lines patch; Gem and bundler should default to '-P HighSecurity' and rubygems.org should reject pushes of unsigned gems.
Anyway, whichever implementation strategy they choose, I hope they will do it soon.
You can deploy X.509 in the same configurations as you can OpenPGP; X.509 is just a certificate encoding format, and the semantics are up to the application .
Ruby has better support for X.509 PKI than it does for OpenPGP, so using OpenPGP seems kind of silly in this case.
Yep, you can do WoT with X.509 if you write the semantics from scratch, and you can also do a CA approach with OpenPGP.
The thing is, for years no-one has stepped in and actually gone through the pesky little details of implementing all those semantics for rubygems via X.509 even though it's come up repeatedly. But maybe someday some expert will come along and fix things.
If you actually read the rubygems documentation on signing gems, it's pretty much "Okay generate your own self-signed key, then sign it with your self-signed key." That leaves a lot to be desired vs best practices. There isn't even an attempt to describe how you get your self-signed keys to users.
I just think leveraging the existing OpenPGP semantics, ecosystem, and battle-hardened gnupg codebase, is a lot easier than implementing an entire system from scratch on X.509 just because openssl is already installed everywhere and ruby has better bindings and gnupg is not and does not.
I'm just trying to take the approach you usually preach: Don't roll your own crypto code and leave the real work to the pros, and it seems like gpg offers a lower path of resistance.
Using OpenPGP is a bad idea here. All the mainstream rubies have access to high-quality X.509 in their mainline distros without a third-party gem.
Both OpenPGP and X.509 are going to require a little of fiddley semantic details that will take time to hash out. The bottleneck here isn't the technology.
I like, and routinely recommend, OpenPGP for solving app crypto problems. X.509 makes a lot more sense here.
can someone explain what happened here to those like me who are a little late. seems like the issue is resolved already or at least temporarily fixed. All I can figure out from the url and the comments here is that there was a gem called exploit and that somehow the config files where posted to pasties.org
The exploit was written by @postmodern_mod3: https://twitter.com/judofyr/status/296649167189725184
Edit: At least a PoC. How someone other than him (since hes denying compromising rubygems) found his PoC and wrote the actual payload and pushed the exploit gem is unclear.
Short timeline:
- 1 week ago: blambeau reported the issue to the rubygems people: https://github.com/tenderlove/psych/issues/119#issuecomment-...
- Postmodern wrote a PoC - https://gist.github.com/4674219
- Postmodern: "I posted it in a private chat room. Guess someone took it for a ride." - https://twitter.com/postmodern_mod3/status/29665192240284057...
- A gem called `exploit-36.44.16` was pushed to RubyGems. This contains a payload that only sends `uname -r` to pastie.
- A gem called `exploit-22.31.31` was pushed to RubyGems. This contains a payload that posted the config/database.yml to pastie. This gem was removed by the rubygems admins when they discovered it - http://news.ycombinator.com/item?id=5140109 & https://gist.github.com/d891e876c53e55bf0920 (that's the payload)
Other exploit-gems that was pushed (but we don't know what did):
- 16.17.49 - https://twitter.com/rubygems/status/296618422702309377
- 20.22.1 - https://twitter.com/rubygems/status/296617610622144512
- 12.5.4 - https://twitter.com/rubygems/status/296540884781109248
- 7.27.42 - https://twitter.com/rubygems/status/296537952320884736
My conclusion based on the current facts:
1 week is too long for such a severe security issue. A proof-of-concept gem seems to be a logical step to "force" rubygems.org to fix this issue. However, publishing database.yml on pastie is TOTALLY 100% NOT COOL, and turns this from "whitehat who wants to help" to "blackhat that wants to destroy (or doesn't understand social norms)".
this is the earliest version of exploit i've been able to find in the twitter timeline:
https://twitter.com/rubygems/status/296537952320884736 (12:38) 7.27.42 ~ 7 hours ago :(
but there was quite a few exploits pushed
This whole thing is a joke, from the ridiculously unsafe YAML parser to the response of the RubyGems people.
If these are your community social norms, then you quite simply deserve the resulting shake-up. This is how you become inoculated against such incredibly poor engineering practices.
I don't quite get what you're trying to say. What "community social norms"? Are you talking about how rubygems.org handled this vulnerability from the beginning? How the community handled this incident after it happened?
And what does it have to do with my quote? I was just pointing out that in general I'm for PoC to highlight security issues when the vendor doesn't respond, but stealing database.yml makes you nothing more than a simple crook.
Hi everyone -
A full post-mortem is coming, but here's what happened from my perspective. I got mentioned on twitter about this thread while I was on the bus. I asked @evanphx to put the site into maintenance mode immediately.
Once I got on wifi, I downloaded the "exploit" gems and found that they used the Psych hack to post config/initializers/* and config/*.yml to Pastie. Luckily, none of our API keys were actually in those files - our config is a bit different since you can still run RubyGems.org on Heroku, which needs to put secret keys in the ENV.
I deleted the exploit gems permanently along with the throwaway account that posted them...that's why this post's URL is a 404.
I've reset our S3 keys just to be sure and any other keys we had on our production site. From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure. Once we get a real fix for this issue out, pushing gems will be enabled again.
Just a general PSA -
Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.
Nick. I dont support today's PoC, I don't, really. But I told you. Twice in a week. I feel deeply sad now.
But thanks for your work anyway.
Proof of Concept?
Point of Contact
Yes, and thanks for telling us. Really, we should have disabled gem pushes immediately. Hindsight is 20/20, and I'm not sure why I didn't think of doing so earlier. The pain of not being able to push gems would have forced us to fix it.
I'm sorry about this. I don't know what else to say. I wish we didn't have to deal with this kind of problem in the Ruby community.
Thanks for all your work Nick!
Respectfully, RubyGems has already been compromised. Not telling anyone has zero advantages to RubyGem users.
Telling people to avoid using RubyGems and to audit if they've used it recently helps protect users.
Responsible disclosure is about avoiding giving bad guys new attack vectors and ensuring that the maintainer has the best chance of fixing the issue before a blackhat discovers and exploits it. That doesn't apply in this case.
Totally agree. I know it's a pain for the admins, but you can't say something like "security is omakase."
I think you meant to say "respectfully".
I agree, but how do people audit? I think it's time to start putting application dependencies, like rubygems and npm modules, under version control, not in the same repo, but in a different repo.
Mikeal Rogers has some good reasons why, but there gets to be too much commit noise if you have big dependencies, and put them in the same repo. A separate repo solves this.
http://www.futurealoof.com/posts/nodemodules-in-git.html
Yes I did, fixed.
Do you think it makes sense to disable pulling down gems until you know for sure nothing in S3 is compromised?
"THIS!", as the kids say.
Is this maybe an opportunity to re-evaluate the volunteer only aspect of RubyGems? I love the service (as I'm sure thousands of others do) and I'd happily pay for a subscription model that would fund a full-time developer/maintainer for the service. Based on how critical it is to the Ruby community I can't imagine that I'm alone in thinking this.
Just a suggestion!
I am not personally interested in doing this. Nothing is stopping others, though: http://www.gemfury.com/
Hmm. That's for private gems. Interesting, but a different use-case. I'd chip in to pay for someone to maintain "rubygems.org" on a professional basis. It's awesome that some people take their private time to support such an essential service, but I'm with the grandparent that it might be good if it were run on a paid-for basis. Maybe through donations or similar, so that access is still free.
Anyone know how CPAN works in this regard?
CPAN is all volunteer. We happen to have a few really good SAs volunteer to keep core infrastructure up (which PAUSE, our upload server, just moved onto) ... but it's still volunteer.
How do you manage critical stuff like security problems such as this one? Do you have a process for that or is it best effort?
Maybe funds support the RubyGems maintainers and the gem authors?
Hi Nick, instead of being a dick, I want to first of all thank you for understanding the urgency (despite being on a public bus, you took the pains to get it temporarily fixed somehow). Like others, I do agree that posting on HN does more good than bad, but at the same time, I do understand that it is also polite to inform the maintainers too (which I think in your case they didn't.). Anyway, thank you for being a responsible maintainer!
Once again, Thank you for maintaining rubygems.org! You're doing a great job :)
Cheers,
Neya
Who is Amanda? I think you mean Nick...
Yep, sorry :D changed it matey! Thx!!
It'd probably go a long way if you guys considered making an easily-discoverable security page with information on who to contact, how you prefer that contact to take place and how long they should expect to have to wait to hear back from you.
If you want security notices to go to the Rubygems team, you have to set up a security page that tells people how to do that. Like everyone else, I appreciate your volunteer work, but no amount of goodwill creates the ability for people to read your mind.
Please post a security page. It literally doesn't need anything more than an email address and a PGP public key.
By the way: if Rubygems needs security help, it looks like there's quite a number of people who are willing to pitch in. When I published a FreeBSD crt0 bug back in 1996, I was given commit privs. I thought that was a pretty effective way to co-opt adversarial researchers.
Great suggestion, I'll add a security page today.
Great!
I know dealing with this stuff is no fun. Try to keep in mind, though, that people like Ben Murphy and Hal (Postmodern) are on your side. Again, maybe consider formalizing that relationship a little!
What would the PGP key be for?
The Rubygems team (or even just the person who writes the security page) can just generate a new one, for "security@rubygems.org" or whatever, and it can be shared by the team.
The purpose of the key is to allow people to report security vulnerabilities without worrying that by doing so they're giving ammunition to people snooping emails.
Maybe I dont understand PGP and this is a stupid question, but if the site is compromised, would it not be possible for them to just put a different key up? One which they knew the private key for? How would you know the public key on the compromised site has not been changed unless you could compare it to the PGP key from before the site was compromised?
You can't, but having a security.html page gives attackers the possibility to contact you privately and securely; this reduces the chance that they'll decide to "contact" you by completely pwning your public site.
Publishing the key is what allows you to detect malicious changes to the key.
@JoachimSchipper do sec people inherently know to try hitting /security.html by convention, or should there be links to it from the main site?
They know to look for a link to the security page.
Github's is in their site footer.
37signals' is in their site footer.
Twitter's is linked off the sidebar in their "About" page.
Google's and Facebook's are the top search result for their site and "vulnerability" "security".
These are all fine options.
Thank you!
What if the key is endorsed by the individual people on the team? Their personal keys, which they keep, must sign the security team key. A replacement cannot have this endorsement.
You can publish the key on keyservers and get some trust on it.
e.g
http://pgp.mit.edu:11371/pks/lookup?search=37signals&op=...
http://pgp.mit.edu:11371/pks/lookup?search=heroku
You don't want to send an actual exploit in clear text to a possibly compromised email account. PGP encrypts, and provides some level of validation that the message can only be read by someone on the security team.
For encrypting the email with the exploit details. This is quite a common practice in the security industry.
Like signing gems? cough
Signing gems should be mandatory
Don't get me started on python eggs :)
i hope there is security@rubygems
Maybe it's time to heckle some sponsors (cough heroku cough) to pay for 2 full time admins for this critical piece of infrastructure.
Thanks for handling this issue & your work on the site. I look forward to read the post mortem.
Oh, and +1 for the security page
Thank you to you and all the volunteers who maintain Rubygems. Its an amazing repository that has helped all ruby developers.
Its amazing to see that one can just type 'gem blah' and magically things get installed.
something to think about here is, how long has this bug been in rubygems.org? how long would someone have had to modify arbitrary gems? fun times!
We have a great many people relying on a volunteer run project here. It would be great to have someone sponsored to spend time maintaining the site.
Unfortunately I don't see any way to help fund such an effort. Is there an officially supported way to donate to RubyGems? Can we create a pool to help cover the cost of maintaining this resource?
There is some larger organizations using Ruby. So not hiring a guy full-time, but they might be interested in chipping in, to get a gem-repo that they can trust.
What options do we have to work around this while the issue is being fully diagnosed?
I'm considering re-installing my gems directly from the github repos. Thoughts?
I also get an "invalid SSL certificate" warning when I try to visit rubygems.org. Here is the message in Firefox:
Any chance this is related?
According to Certificate Patrol, I'm getting the same cert I first received on 2012-12-11:
Why is it that all these parsing issues with yaml and xml were ignored before it all blew up recently? Input parsing is usually the most venerable part of any application, and it's strange that these issues were ignored for so long. These are not super complicated exploits, they are using the _expected_ behavior of the yaml parser in terms of it constructing arbitrary objects.
Because when you write "Rails vulnerabilities are not Rails'" (http://www.revision-zero.org/rails-vulnerabilities-are-not-r...), most people respond "yes it is".
Well, a unsafe parser (which was designed only for parsing trusted input) was being used by lots of people (including Rails) for parsing untrusted input as if it was a safe parser. You can debate about whether there should be a safe parser for YAML, but that's a separate issue.
Is "which was designed only for parsing trusted input" written anywhere in Psych doc? Psych is shipped with Ruby, is it written in Ruby doc maybe?
Agreed, there should be a clear warning - in fact, the load method should be renamed unsafe_load. The root cause of this is probably unclear documentation and misunderstandings between the users and authors of psych.
Where are the adults? This is absolute amateurism. Why haven't Heroku or GitHub or some actual professionals in the Ruby community, stepped up to ensure that this kind of amateurism doesn't damage their ecosystem?
[Throwaway as I'm a small-fish in the Ruby community. I expect I'll get to use this account again soon though]
This is exactly what I'm wondering. Are there adults in the Ruby community?
The vulnerability itself is simply ridiculous:
https://github.com/tenderlove/psych/issues/119
The response from the Gem folks has betrayed a serious lack of experience.
Has Rails been such an insular and young community for so long that there are truly no qualified engineers around with experience in this level of professional software development?
The problem is any YAML loader which supports tags that can name arbitrary classes and have existent classes whose initialization may `eval` the supplied values. Ruby is very publicly hurting, but equivalent vulnerabilities exist in other runtimes, such as Perl via [YAML::Syck](http://search.cpan.org/~toddr/YAML-Syck-1.22/lib/YAML/Syck.p...) or Python via [PyYAML](http://pyyaml.org/wiki/PyYAML) and [LibYAML](http://pyyaml.org/wiki/LibYAML). If your language can accept and load untrusted yaml and your language has `eval`, you're probably fucked and just don't know it yet.
The problem is that YAML is used by Rails & Co as a parser for untrusted user-provided data. It does not matter if there is buggy implementation in Python/Perl if nobody use them.
If other framework were using YAML in this fashion, yesterday will be the time to look for security holes in those implementations.
Why aren't gems protected by a trust chain all the way up like Debian does? Gems aren't even required to be cryptographically signed by the developer and from what I can tell are just thrown over http without any authentication. This seems terrible.
So what source should app developers use until this is cleaned up? gemcutter.org and rubyforge.org?
Oh the timing. So, there was a big meeting, where some of the devs pleaded the case that the recent wave of security issues in the Ruby world were a tipping point, and that we were over it. They presented a good case, and I think the architects were close to permitting the usage of Ruby and Rails in their current cases. Then we get an interruption, as a couple of projects had updated their gems today, and there was a concern about overall security of the apps.
IT Security required the apps were offlined, and the green light was given to the PHP and Python guys in house (and a few retained developers) to start the process of rewriting the apps.
It will be painful, but, I think, the business has lost it's appetite for the constant Ruby idea of "Let's run with Yaml", before firstly "Let's walk safely with Yaml".
I'm glad to hear that at least one organization has done the right thing, and started the process of moving completely away from Ruby and Ruby on Rails. It's just the responsible and sane thing to do, given how the software and the community apparently can't be trusted.
I see where you're coming from, it's a shame that so many bugs are available in Rails.
It's a tipping point, but not of the kind that they thought.
I can understand it's bad timing, but the reaction seems ad-hoc. AFAIK neither python eggs nor PHPs packagist repository sign their packages, so in theory they're open to the same attack vector [1]. Judging from the docs, composer doesn't even allow signed packages. A better reaction would have been to fix the process: Allow only signed gems or cache and audit the packages you install. You don't need to depend on rubygems.
[1] Even OS package repositories or mirrors were broken into. I faintly remember a break-in where debian(?) had to check all their mirrors.
status updates posted here: http://status.rubygems.org/
I never knew the gems were not subjected to version control at all on the server. Can't they put everything into a git repository somewhere?
For those that don't know how this exploit worked, I think the "exploit" gem description provides a pretty good explanation:
Jeeze, bad things happen (or could) to the greatest services ;) I'm late to the discussion, didn't have time to lurk around, nevertheless what I can agree the most is about gem signing. A gem is a package of software, and every single package distributor makes the author sign their work (think linux packages, being a Debian/Ubuntu developer I can tell you that if you don't get your package signed, it won't get uploaded, period, your key in some cases (Debian) needs to be verified in person to build a circle of trust. I think that besides the overhead (initially, I'm sure people will contribute scripts or integrate this so that the developer won't notice) this is really a needed step you guys should take.
Damn it! What a mess! This is some depressing press for the Ruby community.
Being totally new to RoR (trying to learn it), I'm trying to get my head around the scope of this.
When did the compromise happen? Was it compromised yesterday or only found out yesterday?
I have default gems installed on my system and haven't updated anything since the last big Rails security issue that was reported a bit ago.
It'd be great to get some guidance on what to do.
I'm sorry to mention some RG projects are still "potentially" dangerous: https://gemnasium.com/rubygems We'll do our best to help determining if these projects really need to be updated asap!