This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack?

Edit:

1.The bitcoin hacks, March 2012

2.HTP hack, April 15, 2013 (CF exploit)

3.Second HTP hack April 16, 2013 (Another CF exploit)

4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014

5.This hack

I'm not counting their domain name and various other parts of their infrastructure provided by third parties being compromised, but if I was the list would be significantly longer.

Maybe it's time to ditch that coldfusion stack?

[Linode developer here]

Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.

we're still doing routine audits of the existing stuff.

This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.

I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.

When did you start rewriting it? It doesn't take years to replace this stuff.

I was hired in July and have been driving most of this effort and we're shipping rewritten versions of some parts of our infra soon™

I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous.

First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked

The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800...

Today's attack is some kind of unspecified or unknown breach involving Linode manager.

I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues.

Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.

Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.

For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.

Really? Thomas Asaro told us they were all in jail.

A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.

We were aware of it for probably an year before anyone bothered to spend 10 minutes looking at coldfusion source. That's all the time it took.

V interesting. Do any of the other VPS providers strike you as more secure alternatives?

I'd avoid VPS providers in general, but AWS is on a whole different level than linode. They actually understand what they're doing well enough to do live xen patching etc.

But yeah, people get hacked through their hosts all the time. Best approach is colo with minimum access for the dc staff.

    MySQL server that allowed anonymous logins 

Has anyone got more information on this? Various Google searches keep pointing me at the other four hacks.

https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...

They try to downplay it pretty hard.

Thanks for this link. Although it talks about:

    database accessed using old forum credentials

So I'm not sure "anonymous login" would be an entirely accurate description.

The page was slashdotted for me, here's a copy:

"Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.

"A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.

"This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.

"The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.

"You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be.

"The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.

"We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode."

Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff.

Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large customers who just don't mind their stack disintegrating beneath them every sixty days at most.

So, what happened during the AWS outage this past Fall? Or did you restructure to provide failover beyond just moving platforms?

Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks.

Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to linode made this rather front and foremost in our minds) - so nothing that's happened at AWS has ever been anything more than an inconvenience - we've managed five nines since the move - before, we managed one.

Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.

I don't think Telehouse East is correct. They're with Telecity, and in their PowerGate facility, i.e. out in Acton: http://www.telecitygroup.com/our-company/news/2010/linode-ex...

Unless they've moved?

Strange how this works. I moved everything to cloud providers because of availability issues. Now a few years later, I'm moving everything back to dedicated hardware in several different datacenters because of availability issues. Thanks, Docker!! <3

Are you buying your own hardware and colocating, or leasing hardware? If the latter, which provider(s) are you using?

We are leasing hardware, and we are leasing a lot of OVH (in addition to a number of smaller providers). OVH does a good job, and if you build your infrastructure to load-balance around failure (which does not happen often), you'll hardly notice it. Often we get an email from OVH telling us that support techs have been automatically dispatched to a node which we hadn't even alerted about yet. Docker is basically a miracle drug for this kind of thing. Our hardware costs are less than half of what they were with EC2. Huge savings.

We've been watching the unfolding situation closely as well and have decided to migrate/duplicate on AWS. Glad it worked our for you.

At what point should Linode start thinking damage control? DDoS, breaches, lack of transparency...

Wow, these guys can't get a break.

You make your own luck.

If they can't be bothered to invest in their tools and processes then this is the sort of thing that happens.

Yeah it's not like ColdFusion powered web sites enabling billions of dollars in commerce and what with all other web development tools being free of security vulnerabilities and all.

They've been using a CF stack that is fundamentally broken from the foundation up for years and was aware of it.

Its part of why they hired someone in July to rebuild it in python.

You signed an NDA and have done a full analysis of their code? Cool.

Sure wish they had sent out an email notification to users instead of a slashdotted blog post. Now the question is how long can Linode stand in the face of these sorts of hacks and network attacks in the face of stiff VPS competition.

(Linode Employee) Already got it covered, we are sending out an email to everyone in batches, but pushed out the blog first since it can be seen by everyone right away.

I'm still waiting for mine (6 hours since this was posted to HN).

I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too

If the Linode Manager database only stores the hash, how would they know the password?

How did you find out about the illicit login?

If the Linode Manager database only stores the hash, how would they know the password?

Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently.

The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters).

This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts.

But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.

This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.

This only works if the input password has low entropy.

If you're generating every single possible password up to e.g. 8 characters the password's quality doesn't matter, only the length does.

A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present.

But the most effective "rainbow table"-like table is a look-up table with the followings:

* leaked password in plaintext, associate with email and any ID (forum username??)

* hash all of those passwords without salt

* hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed.

If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.)

I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available.

It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!

If we presume the attackers had access to the system handling authorization, then the attackers introducing code to ship passwords offsite as users log in isn't really a stretch of the imagination.

Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked).

< naqod> alexf: any legitimacy to this https://news.ycombinator.com/item?id=10845619 ?

<@alexf> naqod: again I'm not in ops so I don't have the deets, but my gut reaction is to say No effing Way

You'd imagine that by now they wouldn't be so quick to deny this stuff.

As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?

Double jeopardy, I was previously convicted of 50700 counts of unauthorized access to various coldfusion sites. Among those was linode.

This one?

http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-te...

Yeah.

How exactly did you get caught? opsec/comsec violation?

Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps).

The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession.

They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me.

We would've appealed but there was no point as the sentence was essentially nothing.

I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit.

Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.

I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.

Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.

I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.

That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.

Looks like the reason their blog is down is because... it's now being targeted by a DoS:

http://status.linode.com/incidents/kldhjpjnfnkj

Attacking a blog talking about the hack? It sure seems that someone has a grudge against Linode. :-/

At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...

It makes more sense than you might imagine...

Linode, Github, Stackoverflow, Imgur, they've all been targeted. But what do they have in common? In a word: popularity. The core reason these sites are targeted is because it is impressive to others.

The source of this is typically two fold:

- For the lolz. Someone with a botnet just wants to show off, taking down something known gives them more notoriety.

- For a sales pitch. Someone has botnet capacity that they want to sell, and "I took down Linode" is a great way of demoing that to potential buyers.

The first one is more common when someone finds a new method of traffic multiplication and just wants to show it off (i.e. they trick a third party into DoS/DDoSing a target). The second one is legitimate criminal enterprises.

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here.

Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images.

In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.

We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.

We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well.

In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015.

We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials.

In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it.

[1] https://www.pagerduty.com/blog/july-2015-security-announceme...

Nice writeup.

Keeping logins of ex-employees on 3rd party systems is a no-no though I admit full removal might pose some hurdles.

To clarify, we did not leave the users enabled in the Linode Manager. When you delete a Linode user, Linode shadow deletes them in the DB by setting an inactive date on the row.

The honeypot user would not have been able to access the account had the credentials been valid, but based on the information given by Linode we did see someone attempt to log in as that user only once around the time of the compromise.

Saw your tweet (https://twitter.com/theckman/status/684484772316360705) that linked to this post. Did a quick search to get your technical background and your LinkedIn profile states you used to work for Linode? I think it's important to share that info when you're telling your side of the incident. Your past relationship, if you left on bad terms, could play a role in your motivation to post.

I can absolutely understand your concern. I originally had it in my post, but removed it because I was worried it would detract from the details of our compromise at PagerDuty.

I worked at Linode for just under three years, and worked on quite a few different things there. I started on support and moved on to a development role (including writing ColdFusion). I left Linode on good terms. California is much more enticing than NJ, so I wanted to relocate. Plus I was interested in doing more of an Ops role, instead of working including customer-facing web applications. I'm still enjoying it. :)

I think there are lessons that can be learned whenever a company has some sort of security incident. This is especially true if they are willing to publicly disclose details of the incident. We've wanted to provide what limited information we had, but wanted to wait until we had confirmation that Linode was the vector.

While there is some relief in finally determining what we believe to be the vector of our attack, it's very unfortunate that Linode engineers are dealing with the fallout right now.

so you're saying you wrote the code that was responsible for the breach(es)? It also sounds like you knew about these issues before you left Linode but didn't point them out while you were there?

I'm curious what kind of losses you've seen since your own data breach. I'm sorry but you have far too many plausible ulterior motives for taking the position that you are taking.

1) As someone else pointed out, you're an ex-employee of Linode. You went out of your way to hide this fact. I'll refrain from listing all of the very obvious reasons why your word on Linode should be taken with a grain of salt at the very least.

2) Being able to blame Linode for your own data breach is a fantastically easy (although lazy) way to pacify customers about the fact that their personal data was just pilfered by someone on your watch.

All that being said, what have you presented that can be proven? All that can be proven is that you're an ex Linode employee. Everything else is hot air that we're all meant to take your word. Tons of appeal to authority in your explanations. You keep invoking some mysterious third party "expert" security group who conveniently agrees with everything your own company "discovered." If you were actually confident in your own abilities and that of your team members, there wouldn't be an immediate appeal following every attempted assertion.

Plus, even if you really did hire someone, what company isn't going to just say "yes" and agree to whatever PR campaign their customer is saying while dumping wheelbarrows of cash into their pockets? Frankly I don't believe you, and I find your consistent drumming against Linode to be highly suspicious in the wake of these attacks. You're not involved, are you?

The Linode post you're referring to is just saying that they expired everyone's passwords. That's not admitting anything, especially not admitting anything about a separate incident from a year ago. What lawyer would ever take this and say "okay, you can legally publicly blame linode now?" No lawyer worth his salt. In other words, you're full of shit. Your story if full of holes, tells, and I think you should stop posting so much garbage before you're on the receiving end of a lawsuit or are considered a suspect.

Why the hell have they not emailed their customers about this!

This is not the kind of thing I want to learn from HN.

They're doing so as we speak. It takes a while to send 400,000 mails if you actually want them to be delivered to inboxes.

400k emails isn't that many though. :(

Glad they posted it. I just logged into one of my linode boxes that I don't use for very much but keep around, and it was rooted... doh.

Could be just a coincidence if you keep SSH open on standard port. SSH bots don't sleep.

The host was only accessible via private key or Linode's LISH shell. That's what seems most suspicious.

There is some minor evidence remaining in the .bash_history that is curious.

I'd be interested in a write-up about anything you find in that .bash_history or logs, would you consider writing one?

With Linode's extended DDoS I have not been able to get into the Manager for a couple weeks. I'd really like to cancel my account with them (and they do keep billing) but I don't appear to have any tools short of a chargeback. Anyone else in the same boat?

You don't mention contacting support. Shouldn't that be your second avenue after trying to access the account dashboard?

I have always contacted support by opening a ticket in the Manager. The main website is also down. Is there an out-of-band way to reach them?

This is a perfect poster child for decentralized authentication. Centralized stores of passwords and secrets are just asking for trouble. When will we learn?

So, outside of the major cloud providers, what are the good alternatives?

This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.

Don't use Linode.

The actual answer is much more sinister than that.

Which is kind of hilarious.

What is the actual answer?

Last I heard, compromising Bitcoin exchanges for lulz and mad profit. Allegedly.

So if the rumor is true, he wasn't technically lying.

I don't think that if all you have is 'allegedly' it is up to you to make such accusations here in public.

That crosses into personal attack, which is not ok, even if it's just insinuated.

We detached this subthread from https://news.ycombinator.com/item?id=10847715 and marked it off-topic.

It's not a personal attack, it's something that ryanlol publicly claims in ##security on freenode FFS.

If it wasn't, I wouldn't have referenced it

From what I gather, he thinks law enforcement is hilarious.

Not the first time they poorly handled a security issue: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...

There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.

I enabled two factor on my Linode account recently and saw this worrying copy:

If you lose your token and get locked out of the Linode Manager, email support@linode.com to regain access to your account.

Should you need us to disable your Two-Factor Authentication, the following information is required:

An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card. > An image of the front and back of the matching government-issued photo ID.

There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.

EDIT: I also find it odd that you have to manually generate a scratch code, and they don't automatically generate it for you. Again, all of the other 2FA setups I've gone through have done this.

That's it. I'm done with Linode. I will be keeping one server on Linode, but my new startup where I need several load balancers, Web servers, DB replication will be hosted elsewhere.

Do you guys know any good providers? Not AWS.

Can I ask why not AWS?