- 47comments
- 146comments
- 22comments
- 207comments
- 51comments
- 10comments
- 39comments
- 228comments
- 69comments
- 235comments
- 12comments
- 43comments
- 151comments
- 25comments
- 2comments
- 49comments
- —discuss
- 170comments
- 15comments
- 127comments
- 12comments
- 87comments
- 108comments
- 8comments
- —discuss
- 247comments
- 24comments
- 144comments
- 15comments
- 16comments
I would like to see stats from Tier1/Tier2/IX for that. Krebs claims it's 665Gbit/s https://twitter.com/briankrebs/status/778404352285405188 Such attack must be visible in many places, however not a single major ISP reported that in mailing list. Previous smaller attacks were reported 'slowing down' some regional ISPs. Perhaps ISPs got better.
theoretically if it was a well distributed bot net then it could be a few megs from a TON of different sources.. and since akami had their own ASN it could just be jamming up Akami and not other AS
I've seen the graphs from prolexic, the claims are legit
In a world of 10 and 40 gigabit NICs, why is 665 considered big?
Because you have to classify and filter out the spam packets before they reach the intended host and content, which is really hard to do at line rate, especially if you also plan to serve useful traffic.
I really like this comment. I think you're saying that moving data is easy, while computing the data hasn't kept up?
Then again, I could be reading into this too much, and the computing part has always been a bottleneck at backbone level.
It seems to me that he's saying in order to move data at volume, you have to compute on it.
Computation is tricky. Per-bit, if you can handle the network input, you're probably able to fire packets up to the OS layer.
But when you need to run stats on the incoming data, e.g. an ML classifier of "bad/not bad" or "stop/passthrough", you might be O(n^2) or worse. Moore's can't hang.
Interesting. What ML algos are used to classify packets against DDoS?
I think that's the secret sauce for these quys. I'd be surprised if you can find out a lot about current techniques without signing an NDA and leaving your mobile phone in a box at security.
None, really. It's mostly filters against common types of attacks at L3/L4, then OODA. Variations from normal get looked at and custom filters applied as appropriate.
And of course, there's lots of NOC to NOC back channel comms around this stuff constantly to stay relatively on top of things.
Not sure if this applies to DDoS, but a baseline ML method for security is outlier detection. For example, (1) you get a dataset that is mostly "good" data, with some "bad" data (2) you cluster it using something fast like k-means (3) data points are labelled as outliers if they fall further than some threshold from a cluster center.
Linespeed is pretty fast, and there's a lot moving through. Routers are a bit like GNU grep -- they way to be fast is to not touch most of the data.
The more you have to touch, and the deeper you have to touch it, the more expensive it gets.
The real trick is figuring out what's good, what's evil, and downrating the latter whilst allowing the good. Given peering relations, BGP routing, and the sorry state of much of those protocols, tracing problems to their source, quickly, and getting a useful response, is difficult.
At line rate, with millions of small packets coming in every second, even counting the number of packets per flow, a prerequisite for some of the simplest mitigation strategies, is really hard (often requires either expensive hardware like ternary content-addressable memory, or exotic data structures like counter braids that are very expensive to decode). A lot of the time people try to get around this by probabilistic sampling, etc. Similarly, MICA's ability to handle key value lookups at line rate on commodity hardware was considered a big success in the database community: https://www.usenix.org/node/179748. Hopefully that should be a good indicator of the challenges inherent in performing any sort of nontrivial computation at line rate, even really, really simple computation.
(This is why most DDOS mitigation strategies involve getting peers to load balance their traffic when it's still manageable, rather than buying bigger and bigger pipes; it's also why ultimately a large part of the responsibility for handling DDOS attacks rests on the shoulders of ISPs).
I still don't get it. Why not just absorb it? Serving static HTML and related data is fast. A single rack in a datacenter could easily serve this with out breaking a sweat.
Sure, this costs Akamai money they don't want to spend, but is such an attack noteworthy? Eh.
You'd still be filling the pipe. And possibly filling many small(er) pipes at interconnection points. The last meters of pipes into a rack isn't where the issues are.
If the tiny "GET" requests are flooding in at 1Tbps, then the responses are going to be orders of magnitude larger, maybe somewhere around 500Tbps
I'm curious, what sort of datacenter racks do you run that can serve 665 Gbps of traffic? For instance, Google's ToR switches (in its own datacenters) as of Jupiter (2012) were 16x40G, which is just 640 Gbps. Obviously, all of Google can serve a lot more than that, and you can get absurd stuff like 640-port Infiniband etc., but you seem to have pretty unrealistic ideas about network capacity. And, as another comment pointed out, that's just the input for this attack... the output would be a lot larger.
you need 665 gigabits of idle capacity to... wherever the attacker is coming from. If the attacker can send five gigabits from some town in vermont, and your provider(s) can't get five gigabits plus normal traffic from that town in Vermont to your scrubbers, then legitimate users within those networks will be denied service, even if users on other networks are just fine.
Essentially, it comes down to the fact that getting packets from point a to point b requires a lot of cooperation, and cooperation is difficult. Yes, yes, if you bought me the fiber, I could build you a 665 gigabit network, on the kind of money that a nerd could come up with, (not counting the fiber) but interconnecting that network with other people's networks? yeah, that's gonna cost you. Settlement-free peering is a thing, but it is really difficult to set up and maintain those relationships.
The limitation isn't the NIC, it's the processor (and possibly storage). The 10 and 40 Gigabit NICs exist but processing them on a conventional PC is hard.
It is, just about, possible to perform actions on every packet in a 10Gb stream on an x86 machine. You have to use a userspace stack, handle packets across multiple cores, and be VERY careful with what you are doing so you don't do cache misses. At 10Gb/s you're talking only a few hundred clock cycles per packet - anything that doesn't work as planned causes massive backlog.
Now try serving (dynamic) HTTP to that.
Because 665 is a bigger number than 10 and 40
:-)
Akamai has ~140 000 servers around the world. Attack was probably spread around many locations that's why you don't see any report on mailing lists.
The ddos attacks seem to be getting larger these days.
I've recently seen a ~200 Gbit/s hit us.
Does anyone have good resources around mitigation? I was looking at the BGP flowspec but was hopefully that someone might have come across other tactics?
Yes, here is a very good described by OVH http://media.frnog.org/FRnOG_24/FRnOG_24-3.pdf
Consumer bandwidth is increasing.
Take a look at FastNetmon: https://fastnetmon.com/
tl;dr Akamai was hosting his site pro bono. His site was being DDOSed, which cost Akamai a ton of money, so they kicked him off since they were literally only losing money on the deal.
Yes, courteously and graciously verified by him on Twitter:
"Before everyone beats up on Akamai/Prolexic too much, they were providing me service pro bono. So, as I said, I don't fault them at all."
https://twitter.com/briankrebs/status/779111614226239488
Why would it cost Akamai a ton of money? If they already have the infrastructure and automation in place to mitigate such attacks, it wouldn't be a lot of work and would not require additional resources to mitigate this attack? And as others have said: isn't this bad PR?
all those botnets ddossing Krebs = less DOSses for the rest of us = less new clients for Akamai ;)
You still have to pay for bandwidth and if the attack began impacting SLAs and paying customers, that's bad for business.
Let us not permit companies to co-opt language for their benefit.
If it was genuinely pro bono ( lit: for the public good ) then they would have taken all steps possible to keep the site online since the public good was served more by having Mr Krebs online than not.
However, in this case they were hosting him free-of-charge because it was good publicity for them. That's a very different scenario.
"Please make sure to secure your own oxygen mask before helping others"
Krebs is the one who used the term "pro bono".
Besides, pro bono isn't literally "for the public good", it is literally "for good".
Finally - that is a ridiculous standard to hold everything categorized as "pro bono" to. Law firms oftentimes take on cases/clients that can't afford their services, pro bono. Because they call it pro bono, does that necessitate that said law firm should continue to fight all pro bono cases in court until either A. they win or B. they go bankrupt? Of course not.
Well, to be fair to the parent poster (and without going deeper into the merits of this side-argument), I'd note that the expression "pro bono" as generally used in English is actually short for "pro bono publico" which does in fact mean "for the public good".
Law firms (and other professional services firms) call it "pro bono" when they use their specific skill-set to provide their services to those (e.g. the indigent) who couldn't otherwise afford them.
In that example, it's the fact that the indigent can get access to quality legal representation which is itself considered the "public good".
I get that. My point about law firms is that most of them won't take pro bono cases all the way to the supreme court (or equivalent). i.e. there is a limit to how much manpower they are will to expend on a charity case.
In the same way, expecting Akamai to provide free service to Krebs until the end of time because it was referred to as pro bono (even if it was them, which it wasn't) would be silly.
tl;dr - Akamai provided a service that could be seen as publicly beneficial. As long as they were providing free service to Krebs, they were doing something that was arguably pro bono. Them no longer choosing to provide that service does not retroactively detract from its public benefit.
Cloudflare should pick up the site for good advertising..
Cached version where Krebs paints picture of Cloudflare supporting DDoS:
http://webcache.googleusercontent.com/search?q=cache:0uf9RIu...
Isn't this the point at which Cloudflare is supposed to gain a handful of PR points for putting him back online, pro bono, and then doing a write up on how effortlessly they handled the bandwidth with eBPF?
Unfortunately, Krebs has (correctly) repeatedly attacked Cloudflare for sheltering most of the most prolific DDOS attackers. I doubt that's going to happen.
Care to fill in the details for me? Do you mean to say the most prolific DDoS attackers work for Cloudflare? Or that their network somehow (?) shelters them? What do you mean exactly? This sounds interesting.
There's another comment that links to one of his posts about it (http://webcache.googleusercontent.com/search?q=cache:0uf9RIu...).
Cloudflare hilariously acts as a reverse proxy for booters and does not respond to abuse reports against them.
I can confirm that this is true. "Hilarious" is not the word I would use to describe it, "potentially criminal" I think is closer to correct.
I'd call it business savvy.
They seem to think they have firm legal ground.
Keep in mind that most reverse proxies will do the same thing. The only difference with Cloudflare is that you don't know the destination IP.
Not even remotely. If the government steps in with a subpoena for the origin host IP or an injunction to stop protecting the site they'd stop. Someone on the internet asking them has no legal power to do so.
https://news.ycombinator.com/item?id=12568256
Why send cloudflare the abuse report rather than the police who are paid by the government to investigate crimes. I have little understanding for this current trend of letting suspected criminals go un-investigated while all focus is on private third-parties that is held up to act police, judge and jury.
Because one would think that it would be in CloudFlare's interest not to harbour criminals on their network. This logic seems to work almost everywhere else on the internet, including privacy friendly hosters in iceland. It's mostly just CloudFlare who replies to every abuse report with the same "WE R A REVERSE PROXY", no matter what the actual issue that was raised with them was.
If they were any smaller, their IP ranges would just go into the rogue-isp-blocklist, and that would be the end of that. But because they're mixing in the criminals with their normal customers, that's not really possible.
And since I am unlikely to be in any jurisdiction that CloudFlare is in, nor do I have any chance of finding out who these criminals are because CloudFlare is protecting them, going to the police here wouldn't really achieve much.
Because one would think that it would be in CloudFlare's interest not to harbour criminals on their network.
It's also not in their best interest to take on the burden of deciding who the criminals are.
He might be referring to vDOS using Cloudflare: http://webcache.googleusercontent.com/search?q=cache:kaymYsb...
https://webcache.googleusercontent.com/search?q=cache:kaymYs...
I have personally witnessed Akamai use strongarm tactics to "prove" you need their, "protection" in such a ridiculously high profile instance I am sure they have no shame.
If this is a CloudFlare Vs Akamai attack Krebs isn't saying, but I would put dollars to doughnuts it is.
While I'm open to believe you if true, you need proof, links to affected people giving details, etc... Not your personal anecdote that may or may not have happened the way you describe it.
Otherwise your post is merely an unsubstantiated personal attack against akamai.
I've talked about it before in lots of contexts and no one believes me anyway. Why bother?
well if you've never offered factual proof to back your "I've seen things" statement, that's not a surprise. If you've ever given proof to back it up, I would love the corresponding link.
Do you check your email?
Cloudflare protects everybody who signs up, including control panels for "booter sites", which are web pages where you can allegedly buy a DDoS attack for an hourly rate.
Kind of like the police protects gangsters from getting shot by other gangsters, but you would really like them not to do that, so that the gangsters can just shoot each other.
In this case, Brian Krebs tried to convince Cloudflare to kick off the booter sites, so they are unprotected, and can DDoS each other. Cloudflare didn't put any effort into that idea, and now he's apparently angry that he didn't get through to them.
That's not called being "angry". That's called being principled. Other things that are considered principled include posting your opinions with your name on them instead of cowardly resorting to a throwaway account like you just did.
Cloudflare is not the police. They're a private organization that makes a profit from offering "protection" for people getting DDoS attacked. They enable the people doing the DDoS attacks by protecting their booter sites (https://www.google.com/search?q=ddos+booter). That's called a racketeering operation (https://en.wikipedia.org/wiki/Racket_(crime)), and that's illegal. There are laws against it. Just because our crappy government is too incompetent to file charges doesn't mean it isn't illegal.
If Cloudflare thinks they can foster criminal activity through their network because they're running a juiced up nginx proxy, they're wrong. The "slippery slope" argument is absolute nonsense. As Krebs himself pointed out, they already remove sites that are hosting phishing attacks and malware.
Cloudflare, it's time to do the right thing here and stop protecting DDoS booters. Your policies are helping to damage the internet and censor people, whether they're illegal (they are) or not.
They enable the booter's free speech; they aren't enabling their actual attacks. If comparison to police doesn't suit you since they're special, think of doctors.
You're seriously comparing DDoS attack markets to doctors?
It takes an intense contortion of the concept of freedom of speech to apply it to this malicious and illegal activity. I guess under that logic it's also okay for a personnel security company to host (or hide behind their nginx server) a hitman-for-hire marketplace too, as long as they're not the ones doing the actual killing?
DDoS attacks (and their store fronts) are not about freedom of speech. They are, always and everywhere, about the suppression and censoring of speech through violence. Protecting them means that you are protecting violence.
But if you're adamant on this being free speech, fine. Where's the free speech criticism for Cloudflare shutting down those phishing and malware distribution sites? Why are DDoS attack sites magically different and deserving of freedom of speech protection?
Honest question -- how do you feel about someone downtown with a bullhorn, calling for terrorism or some other malicious act?
Cloudflare will obviously respond to law enforcement requests of what the origin server is. Krebs is not law enforcement, and neither are other DDoSers. What is your problem?
That would be fine and all if Cloud flare wasn't removing other Internet threat from their networks (phishing, malware,...).
They remove them all, except the one whose threat they benefit from (cloudflare has a direct interest in the ddos threat being as big as possible).
Claiming they are protecting their free speech is a load of bollocks.
Both the phishing and malware attacks would be performed over CloudFlare's networks, while the DDoS attacks would not.
What happens over CloudFlare's networks in the case of DDoS providers would essentially be the agreement of a business contract.
The attacks are paid for and managed by the customers through the web portals that run behind CloudFlare. How is that not enabling the attacks?
One could imagine management and payment via phone calls or snail mail. The actual attacks couldn't happen like that.
Free speech? As an example, criminal conspiracy and solicitation of murder are both crimes of speech, and claiming a "free speech" defense will just make a judge laugh at you.
Yes, but that's a decision for a judge to make, not a private company.
So if someone hosts a site for selling drugs but doesn't sell it himself that is free speech too? It didn't help someone who called himself a Pirate and made a Silk Road.
If only...
Let me quote [0]:
So, if I report a scammer CloudFlare will forward my information to that criminal, putting me at risk. Gee, thanks!
and
Or to put it in the words that they answer every abuse request with:
Which basically translates to "We don't care, we want to pretend that we are not responsible for our actions."
[0] https://www.cloudflare.com/abuse/
FYI I was impressed by your post and just tweeted a link to it to them.
Cloudflare has a pretty simple policy. They only censor content when they legally have to, or when it's child porn. That actually opens them up to a lot of heat from people who aren't big fans of the KKK, the Westboro Baptist Church, or botnets. BUT they don't specifically allow botnets as a weird method of promoting them, it's a widely applied policy.
I would bet things would be a fair bit easier for them if they agreed to take things down which most people don't like, but from my position they are taking a very principaled stand for free speech. Are people on hn actually arguing we want more censorship on more places on the web?
Nobody in here is proposing that Cloudflare censor unpopular speech. We are asking that they stop protecting for-profit DDoS attack sites that are destroying the internet and using violence to censor people's ability to speak. That isn't a freedom of speech debate, it's a debate on the ethics and legality of defending and protecting criminal activity that financially benefits them, a timely topic now that this activity is actively threatening the ability of the internet to function for any kind of speech http://www.webhostingtalk.com/showthread.php?t=1599694 http://webcache.googleusercontent.com/search?q=cache:0uf9RIu...
I mostly agree with you but let's not take it too far.
A DoS is not a violent act. I am mostly ignorant of these things but I think attacks if this kind are a service that test our capabilities. My fear is that there might be calls for legislative actions against "DoS attacks" which would then apply to people sitting at home pressing F5.
How would such a law be different from the current laws? If you sit at home pressing f5 with malicious intent and succeed at bringing a site down, you're committing a crime.
I don't know about you, but criminalizing the act of pressing F5 with any intent seems firmly on the way to Aaron Swartz-like cases to me.
What if you are just fed up of waiting for a site to reload and press F5 a number of times? And what about the (probably majority of) instances where the "attacker" is simply a person who unknowingly downloaded malware onto their computer to get free smileys or whatever?
If you figure out how to build a 600Gbps DDoS attack with Firefox, you are correct, that still qualifies as a DDoS and you can go to jail for it already. People have been tried in court for using Low Orbit Ion Cannon before, in a few extremely isolated instances. A DDoS is a DDoS, but intent is obviously important, and you do need to actually cause a problem for there to be a crime. I think clicking reload a couple times would be a stretch here for enforcement, perhaps it's possible but AFAICT it's not yet happened.
But we aren't talking about protest with a reload macro here, these are for-profit criminal botnets. And one if them just took down the largest DDoS mitigation network in the world. Which means there aren't many sites on earth left they can't take down. Much smaller attacks have nuked Github for days. Who's next to get "freedom of speeched"?
I think a better solution is to have ISP s work together to warn and cut off access to botnet infected computers. They have the technical ability because they have strikes for copyright. Perhaps it could be a soft ban like an hour long ban or something.
But if two billion people decide to stay at home and continuously press F5, you should get freedom of speeched. I think that's the equivalent of a picket line. Not talking about automated tools other than "refresh page every second".
"I will send DDOS for $xxx, send paypal to ###@example.com"
That's the the extremely unpopular speech that you're proposing to censor. The instant you say "oh but that's different" because of the contents of the speech, you're interjecting your own opinion about that speech.
Which, actually, is fine, but don't play that off as not being speech.
At the level where Cloudflare's network isn't actually being used to send the DDOS attack itself, it's also still speech.
Cloudflare will close accounts when asked, backed by court order. The problem is on today's Internet, that's nigh impossible, which realistically means it falls to Cloudflare to interject an opinion on what's good and bad, but so far they've avoided that as effectively as an ostrich burying it's head in the sand, and so are effectively supporting many bad actors.
While I understand your position, the particular line you quote is not protected as 'free speech' because it's advertising to sell a criminal act for money ... I could be wrong.
I agree that it's an ethics problem, and a non-trivial one at that.
It seems like another problem caused by the fact that code can be data and data can be code. By which I mean, both are information. 'Free speech' implies the intent to be communicated to people, and can be considered 'data'. However a DDoS is a bunch of information with the intent of affecting the behaviour of computer systems, and can be considered 'code'.
The problem lies in discriminating between the two, given that "bits don't have colour", as explained here: http://ansuz.sooke.bc.ca/entry/23
I'm not at all sure what the right answer is, here. I'm also not 100% convinced that Cloudflare has the right approach, but I'm leaning to "yes", considering the alternative.
(by the way, you'd probably be interested in watching the youtube clip jgrahamc posted elsewhere ITT, with someone from Cloudflare saying some words about their perspective on this dilemma: https://news.ycombinator.com/item?id=12564876)
Protecting unpopular organizations is taking a principled stand for free speech. Protecting people who profit from breaking people's web services is not.
"We don't take it down unless it's illegal" is a simple policy, but to be a good policy it needs judgment as well.
They proposed, but he didn't reply:
https://twitter.com/eastdakota/status/779063982984355840
https://twitter.com/eastdakota/status/779129927543033856
I wonder why. Any ideas?
I imagine he is still talking with Akamai(they did not comment after all) and expects to be back after attacks die out. Switching would burn that bridge.
Akamai was good to him by providing services pro bono, so I doubt he wants to knowingly become a PR stunt for CloudFlare.
I'm surprised that the Azure or Google Cloud teams aren't on top of this. They want tech people to pay attention to their stacks, why not host a high profile site like this to gain the respect of the industry?
I feel like Brian Krebs is a public good at this point. Would love to see Google foster a better web by hosting him!
They should, and get agreements with CDN's / ISP's to forego charges in case of a DDOS. If anyone could pioneer such an agreement, it would be Google.
You want them to push the idea that isp and other middle men networks should not be dumb pipes, but charge a different pricing depending on traffic type and intent?
Your comment may have the best of intentions, but that's how you take net neutrality out the window.
New way to get on google for free: make an AMP site. Until they start charging for that at least.
Considering they charge per GB data transfer, even if your site survive the attack you'll get an amazingly expensive bill at the end of the month.
There could be something like a DDoS insurance.
Insurance only works when you can get good coverage (lots of premiums being paid). Akamai and CloudFlare are DDoS insurance, and BK just got dropped.
You don't see insurance companies rushing in after a disaster, this is no different.
For both Google and Amazon the cheapest transfer you can have is 0.02 USD/GB, which in case of such attach gives 5.5k USD / hour, or 400k USD for three days of the attack.
That's for outgoing (egress) transfer. Incoming (ingress) transfer is free with both Google and Amazon, and a DDoS attack is mostly incoming.
You just have to make sure your infrastructure doesn't try to auto-scale up to actually handle all the traffic or suddenly you have thousands of high-powered instances to pay for...
Amazon's actual ELB IPs can handle relatively little traffic, "prewarming" is required in order to add more ELB instances--during a DDOS attack, you'll be overwhelmed almost instantly. Route 53 uses DNS round robin, which is trivial to bypass if you're planning on a DDOS attack (by targeting a specific IP). Google actually gives you an anycast IP, so they're a better option.
All that being said: the idea that only ingress traffic matters during a DDOS attack isn't quite right. If the connections are legitimate, you either need to be able to detect the attack attempts (requires expensive coordination and mitigation techniques, especially if the attack is much larger than what a single NIC can handle) or actually serve back the content (which will make your egress skyrocket).
The enterprise doesn't care. It'd take a hellluva an event to get marketing behind such a stunt:
Akamai is probably way bigger than Cloudflare, so if they don't take customer that got a 620Gb DDoS I doubt Cloudflare will.
I think the bigger issue was that he wasn't a customer (they provided the service pro bono), not that Akamai wouldn't keep a customer that got hit with such an attack.
Traditionally, journalists and celebrities getting freebies get /better/ service than regular paying customers.
I don't see why being pro bono would matter in an established company?
Cloudflare has this on their website: "600Gbps: Largest DDoS attack stopped" - https://www.cloudflare.com/under-attack-hotline/ But I suppose attacks may differ in other ways that Gbps.
If I remember what cloudflare's article said, that 600gbps attack was a reflection/amplification attack, which for them is likely easier to filter out then just large amounts of direct DDoS traffic, which is what this was evidently.
If you're under attack, try us out. We'll aim to surprise you in the most unsurprising and dull way possible, by keeping your site online throughout.
Shouldn't it be the job of the police to protect his web property. The police, or another government agency, protects citizens offline, why not online? Why do we have to rely on private entities for basic protection online? Time for an online fire department or something similar?
I'm pretty sure I've heard some peoples describe CERT/CIRST centers as "web firemen".
Not that I'm opposed to something like that, but depending on how big a target you are private security is already a must in the real world.
have things changed so much that cloudflare has enough capacity to seriously contemplate handling something that Akamai could not handle?
You've gotta get the bandwidth to your filtering servers before you can filter it. DDoS mitigation, as I understand it, is first and foremost a matter of having more capacity than the attacker.
Yes.
No, it's just that Akamai (probably) didn't want a "customer" that they were hosting pro bono to affect paying customers.
It would be interesting to try out some of these new p2p website technologies like IPFS/WebTorrent with these high profile sites who are frequently attacked.
+1 for IPFS
Hosting static blogs is really easy on IPFS (and if you absolutely can't live without comments: use disqus) but the URL's are cryptic and you either need a public IPFS gateway to access the site - which could get DDoS'ed - or run your own.
Another alternative is ZeroNet but you still need to run the client to access the site.
IPFS team member here:
If the URLs are cryptic, you can use dns to make them look nicer. Take a look at the TXT record for ipfs.io, as well as the TXT record for _dnslink.ipld.io
Both of those websites are hosted through ipfs and have A (or CNAME) records pointing to our gateways. You can also access this locally if you happen to be running an ipfs daemon at http://localhost:8080/ipns/ipfs.io
A little hard to find, but this is well described in https://github.com/ipfs/examples/blob/5d77470eb5ae944aa54093....
That's awesome, not sure if this was around last time I tried but it is still hard to find, thanks! :)
At this scale it must also cost a ton of money to carry out this attack, I wonder if there's a vulnerability that we don't know about that let them do this so easily?
It doesn't; you're using compromised machines to initiate the attacks, which is free to you.
You're almost never using machines you compromised. Instead, you're using someone else's compromised machines: a botnet-backed DDoS-ing service, which you rented on the open (black) market, at the going rate. Bigger attacks still cost more in such setups—whether or not the botnet nodes were free to acquire, the resulting network is still a scarce resource whose price rises with demand.
(Of course, in the very special case of Krebs, the people he is reporting on frequently are the owners of the botnets, who can of course use their own botnets freely.)
Well, no, if you are committing crime you can just use illegally pwned computers.
Quite impressive. You know your blog is good when folks will try to take down a CDN to supress what's on it. He's also had heroin mailed to him in combination with a swatting attempt before: http://webcache.googleusercontent.com/search?q=cache:gEjqPfc...
holy shit. that is terrifying.
Normal game streamers get swatted and sending heroin would be easy with the silkroad-alternative market online.
Does that in any way negate the fact that it is terrifying?
No.
Could have been child porn.
An almost horrible, related story:
When I was in middle school and the internet was still fairly new (we had just gotten it) a classmate of mine hatched a terrible plan to get rid of a teacher he hated. He waited until the teacher was out sick one day then during the substitute's typical teaching pattern of having us "read these 3 chapters, answer questions then keep your head down until class is over" he jumped on the teacher's computer. After reassuring the substitute that he was allowed to he tried desperately to find child porn. His plan was to save it into semi hidden folders onto the computer then later on turn the teacher in for having child porn.
Fortunately this classmate wasn't able to find any and eventually gave up. But I've always remembered his plan. It's terrifyingly believable that if someone managed to get into your computer and download child porn, there is likely little recourse or way to prove you did not do it.
In this case such prove should be possible. Files creation dates would show that the files were downloaded when the teacher was absent. The dates would make it easy to associate this accident with your class and then, after asking a few questions, with your classmate.
Those dates can easily be altered by someone knowledgeable though.
And easily sold to a jury as proof positive after they were altered. I've feared for a lot time that this type of thing is currently in the government arsenal of threats and weapons.
But file creation dates are easily changed by a skilled hacker.
And also by the teacher. So maybe he altered them? That's what the DA would sell to the jury...
And what if the teacher was not on leave or sick, but just in the school's cafeteria, or briefly in another class, etc?
Good luck proving anything with the dates, especially after several months, where nobody remembers who was where at that random day in the past.
This is basic criminal defense stuff, actually. "What time did you go to the cafeteria? Did anyone else see you there?" Then talk to the people who he says saw him.
Are there any cameras in the school that might have captured his visit to the cafeteria? Review the footage.
How did he pay for his food? Did that create a record that can establish a time and location?
Are there any witnesses who saw the kid sit down at the computer? Like the substitute teacher, for instance?
None of this is unique to child porn cases. Establishing or disputing time and location is basic trial strategy. All a defendant has to do is create reasonable doubt, not conclusively prove innocence.
And they'd remember they saw him after 6 months, and even more so that they saw him leave in 15:20 instead of 15:10, because?
No, but all this make "I didn't change them because filestamps in file are in an hour I wasn't there" difficult.
Heck, the teacher himself will probably not remember where he was at the time the timestamps show...
The real problem is that the minute it was found the teacher would probably be fired, or at a minimum put on administrative leave, the media would catch word of it and publicly tarnish him, parents would demand swift action, and before the investigation even had a chance to get going the guy's life would be ruined. In the end, even if he was found innocent, the lasting damage would done.
Honestly, even if you can prove you were framed 100% and are found not guilty, the media has likely already run several stories regarding the teacher who's computer had child porn on it. The school will be forced to fire them. The school board will likely bar them from ever working in another school in that city again. When you Google that person's name the top categories will likely be regarding their trial / arrest.
Once an accusation of child pornography (creation or possession) is put out there if any of it gets exposed to the public by the way of the media (and it will) that person's life is seriously ruined even without prison time.
The problem seems to get worse, too and I certainly don't have any good ideas regarding it. Except maybe re-tooling a new search engine that somehow can take context / validity into account but that's exceptionally difficult to do. And even then if someone gets their news or information from any other source you're still screwed.
Aren't CP laws set so that it doesn't really matter if you did it, the possession by itself is a crime, no need to prove intent?
There are 50-60 parents in any given class. A fair number of them can be expected to make trouble no matter what the police does.
Yes, and every image is a separate charge.
Because think of the children.
Possession and distribution of child pornography is perfectly legal if you're the FBI though.
There's a further nuance there as well. Say you work for a site that accepts UGC images (like a major social network for example). It's required that you have a team who works with law enforcement to handle investigations / takedown / preservation. Anyone who works for said team unfortunately thus has to see those images and have them, at least temporarily, on their machine, but there is zero legal precedent to protect them. It's a legal grey area and thus I have and continue to recommend anyone in such position to refuse to risk it.
You have the defense of "we are specifically required by law enforcement to do this" which is pretty strong. It's no different from a locksmith being asked by a cop to break into someplace the cop needs to be.
I think the psychological trauma would be a much bigger concern for most than some vague legal threat.
Each video frame is a separate charge...
Hang on, just to clarify: if someone else puts a file on my PC without my knowledge, without me ever having seen or opened it, he could call the police on me, and it would get me convicted?
Yes. In such a case, a presumption of innocence would be true in theory but of no value at all in practice. Such an attack could even be carried out remotely -- an attacker could compromise a machine remotely, then plant incriminating evidence on the compromised machine (i.e. child porn, terrorist literature, drug-dealing evidence, etc.), then alert the authorities.
This plausible scenario is another reason to vigorously protect one's computer against external attacks.
I won't say that it doesn't happen (bad outliers always exist); however it would be very unlikely you'd be convicted for it.
A few years ago when I worked in computer forensics there was this big myth that if you went on porn websites and one of the images was underage you'd get done for it. However intent is a big part of law and so there would always have to be something along with "just an image" showing some intent to have obtained and viewed it.
There was a person in Finland mass scraping the alt. newsgroups for porn, and they got some cp on their machine inadvertently. Conviction, publicity and a ruined life promptly followed.
If someone else put 10 kilograms of cocaine in your house without your knowledge, without you ever having seeing or opened it, he could call the police on you and you would get convicted.
Whatever happened to reasonable doubt, innocent until proven guilty, all that stuff?
What kind of court routinely convicts people who didn't actually do the crime?
In the US it's not obvious that an anonymous tip would establish probable cause for a search.
They don't even have to call the police. Simply put it in your Dropbox, which can be done from many devices (it's Dropbox after all).
Dropbox (and all other "cloud storage" providers) actively scan for CP material and will turn you in to the police automatically.
I need a link to an instance of this before i will believe it
In the UK, somebody can send you an obscene file over Whatsapp, unsolicited, and get you convicted without you even opening it. [0]
We have sleepwalked into being a police state and barely anyone seems to care.
Police have also started trawling Reddit to look for thought crimes. [1]
[0] http://m.theregister.co.uk/2014/08/05/whatsapp_smut_convicti...
[1] https://m.reddit.com/r/unitedkingdom/comments/53y1wi/a_reddi...
From article no. 0:
"<...> police stopped them for unrelated matters and discovered the shock images upon inspecting their mobile phones"
What?? So police in UK can now ask to inspect inside your phone without any warrant or even reason?
You should try reading what you post first.
1. They weren't convicted of any crime.
2. The person was charged because he racially slandered someone. Not because of a thought crime.
I can't believe they both pled guilty! Is it possible in the UK to appeal a conviction if you plead guilty? IANAL, but I think their are some narrow cases where it can be done in the US, for example if you can show that your legal counsel did not provide competent representation.
The UK is all sorts of fucked up. You're not lawfully allowed to withhold passwords to encrypted data you're in possession of, meaning someone can slip an encrypted USB drive in your bag, you get arrested, and sent to jail for god knows how long because you're unable to give a password to a drive that isn't yours (and maybe doesn't have any password).
At least here in Germany you would have a very hard time proving your innocence.
It's even worse: The police can do it too once they got access to your computer for some other reason: Say, provoking a three letter agency.
Is there actually any nation on Earth where powerful and/or connected people can't screw you over if they really want to and if you have no other powerful or connected people on your side? (and no money, etc.)
The thing about this stuff that's really alarming is that any random script kiddie could also do this by coaxing your machine into downloading something. That greatly increases the surface area of people who can screw you. Given the abysmally awful security profile of a lot of consumer software and devices this is very plausible.
Yes. In US law this is called Absolute Liability. Formally, absolute liability means that a conviction does not hinge on the presence of mens rea.
You're likely familiar with this concept in the context of speeding tickets. All that needs to be proven is that you were, in fact, speeding. It doesn't matter if you could not have been aware of your violation due to, say, a speed-limit sign that was blown away by a storm. If you were provably doing 55 in a 45 zone, you have no recourse.
The argument for absolute liability with speeding violations is purely practical, I believe. The reasoning is that it's not a crime, per se, so the trade-off of individual protection vs expediency of trials is deemed worthwhile. Clearly, the same is not true of child pornography convictions.
Actually, that's "strict liability".
That probably does matter, since exceeding the speed limit properly posted is usually the actus reus of speeding, so even to the extent it is a strict liability offense, the absence of proper signage for any reason (except when the speed limit is either the states maximum highway speed limit or a default limit for some other condition which does not require signage, in which case notice is provided by the law setting the default for the conditions, and the sign is a reminder) makes it so that no offense occurred. [0]
[0] Also, given that states do generally have default speed limits that apply in the absence of signage, one could easily argue that the absence of signage is itself a positive indication that the default speed limit applies, making available a U.S. v. Kantor-style "good faith" defense even under strict-liability principles. [1]
[1] https://en.wikipedia.org/wiki/Strict_liability_(criminal)#Un...
In the UK, possession of Child Porn is a strict liability offence: https://en.wikipedia.org/wiki/Strict_liability_(criminal)
Is it?
http://www.legislation.gov.uk/ukpga/1978/37
[...]
There are some amendments in the Sexual Offences Act 2003, but I don't think they turn it into a strict liability offence.
That's not true. The wording of the act's relevant to CP specifically say things like "knowingly" which is well-ruled to mean Mens Rea is required.
Real-life sexual offences, as in actually attacking a child, is strict liability.
I stand corrected.
CP laws in most countries are insane, because every politician knows what it would take to defend against reinforcing the anti-CP arsenal.
that reminds me of randomRambo's story. On stream he was searching for a file on his computer, and a file with a name containing CP. Long story short, police was called, "he molests his kid on stream" (totally untrue), police came, arrested him on stream. took more than half a year for him to get declared innocent.
here is him talking about it: https://www.youtube.com/watch?v=CzdFOpRTvyU
Even if declared innocent, being charged on suspicion of possession of such porn is a life-long stigma that never wears off. For example, it will be impossible for a foreign citizen to obtain a US 'ESTA' visa waiver after such suspicions.
The google-cache page doesn't seem to be loading right for me - It's still trying to pull off extra content like css and images from the regular site. That said, I found the article in way.archive.org, and it has images and formatting intact:
https://web.archive.org/web/20151115154842/http://krebsonsec...
Thanks for giving a link to this post!
Oh, that's a much better version! Thank you. I thought I linked to the text-only google cache, but perhaps not.
Another good mirror: http://archive.is/CVSm3
Google cache: you have to click Text-only version which isn't trying to load anything from the original site:
http://webcache.googleusercontent.com/search?q=cache:kaymYsb...
(it's the "strip=1" parameter in the URL)
Wow. Imagine if he hadn't gotten access to the forum and preempted the police call.
I wonder: Is he fluent in Russian? I doubt google translate could help you out with that.
Actually yes, he taught himself some Russian, from what I remember.
If you're going to be reporting on his beat, you have to be at least somewhat fluent.
That's an interesting read but I'm really not sure why Krebs posted the real name of "Fly's" wife. She doesn't seem to be involved in her husband's activities at all, so what's up with that?
"fair game"? The Russian hacker did the same with him as he posted copies of Krebs' credit report, directions to Krebs' home and pictures of Krebs' front door.
Plus, threatening to kill Krebs' wife.
But since we're talking about ethics here: Sure, now that the hacker faces 30 years in prison, he's not short of probably sincere apologies. I could really believe that he now has changed his view and accepted his guilt. It makes me ponder the thought of if I were Krebs' to not only feel sorry for the guy but (if it were legally possible) to dismiss the charges against him.
Consider the much more likely outcome of the hackers' plan: That it worked. Would the hacker had the same sense of guilt then? Or the same sense of forgiveness as Krebs or me seems to have? Maybe. We'd never know unless he did. It's more likely, he would have enjoyed Krebs' ruined live. Maybe even continued to threaten his wife and family. Just for the fun of it.
Because doing that in the anonymity of the web makes it easy to misbehave in ways no one ever would in front of the public eye and even less in the eye of his family and friends.
I tend to be a bit skeptical of the sincerity of apologies first made after the perp is caught.
If you ever press charges against a criminal, follow through with it.
If you forgive them, they might just do it again.
I don't think he deserves 30 years either, but he should still go to prison.
Of course he's not going to get the charges dropped against this guy. Krebs should be pushing for the most harsh penalty possible against this guy so he can continually point to him as an example of what happens when you threaten him or his wife.
I think thirty years is extreme, but I think the hope is that this will serve as an effective deterrent against this kind of crap.
Fortunately in the US justice system victims do not determine sentences. That said, I agree that hackers who maliciously target an individual with disregard to collateral damage should get the maximum sentence.
"Fair game" excuses collective punishment, then? I'm not so sure.
Also, a "wife" is a person. How do you justify retaliating against someone by harming someone else? You steal my car, I beat up your wife- hey, "fair game"?
(Not parent poster but...) No, but you steal his car, you can't complain if he steals yours.
I don't think Krebs was right to dox the hackers wife.
She's her own person. If being married to an asshat is a punishable crime then many people are even worse off than they realise.
But Krebs retaliated in kind.
I do however think that's a punch he should have pulled.
Is the wife in any actual danger from his reveal? I imagine Krebs thought no and that's the difference.
It might destroy their relationship by scaring her into leaving him, which is perfectly fair game. You dont deserve to have a relationship with your wife if you actively try to destroy someone else's.
Is the strongest argument - but still too weak.
So, should Krebs have flown over to Russia and beat the shit out of her? That would have definitely "scared her into leaving" the guy.
Again: how is it "fair game" to punish one person for the actions of another?
Relatives of a victim are generally protected by the press, but relatives of a criminal are not. Right or wrong this doesn't seem outside of accepted USA norms. Other countries are another matter, where often criminals identities are protected.
Yeah, but the question is about right and wrong, not accepted USA norms.
I put it in quotes because I have a hard time considering it "fair" myself.
The main point here is: this anti-social behaviour of the Hacker was made possible because of his anonymity.
Anonymous towards the legal system but anonymous towards his private life as well.
It is fair to assume he kept his cyberbullying activities to himself because she wouldn't support that.
The hacker's wife was probably more endangered by continuing to be living next to a criminal.
I think this is naive and gullible. This person needs and deserves a rehabilitation attempt, which he's very unlikely to get in prison. The psychopathology of someone who doesn't merely threaten you with death, but an explicitly named loved one, is a very dangerous, damaged, malicious person. Merely being found guilty and in prison does not at all ensure this person accepts responsibility for what they did or that it's wrong.
What someone does while they're being watched isn't a good judgement of their character. What matters more is how they behave when they're not being watched.
The people he deals with don't have any rule book they follow. Likewise, it's up to him who he wants to go after. Clearly, he decided that turn around is fair game and went after his wife.
Unfortunately when you deal with criminals, you can't really justify any behavior on either side since they are both operating outside the bounds of the legal system. Outcomes are mooted in the context of the world they are operating in when it's devoid of rules, honor and morality among the participants.
I don't know why he did it, but there's a difference between a wife and a wife's name. There's a decent chance that even if the person isn't involved, her name is or may at some point be.
It's not difficult to open a bank account in your spouse's name without his/her knowledge, and use that account to do things he/she wouldn't do.
This recent talk about DDoS attacks is worth a watch if you're curious about why it's a hard problem to solve: https://www.youtube.com/watch?v=79u7bURE6Ss
Starting watching this and he gives a great explanation of the issue. Thanks for sharing.
https://twitter.com/briankrebs/status/779111614226239488
"Before everyone beats up on Akamai/Prolexic too much, they were providing me service pro bono. So, as I said, I don't fault them at all."
I tried to get to an article on Krebs' site from a Bruce Schneier blog post, and couldn't, then bumped into this post in HN.
It's a pity Akamai booted him off; on the one hand, I can understand that it would significantly impact on their SLAs to other customers, but on the other hand it's a shame they don't have a lower impact network to re-host him on, and use this as a learning lesson on how to better mitigate such DDoSs...
Is it according to terms/conditions of Akamai?
What is 'it' that you are referring to?
Akamai were providing him hosting free of charge, but the size of the attack will have had not insignificant financial impact on them, and on their customers. It's completely understandable that they've chosen to terminate their hosting agreement at this time.
https://twitter.com/briankrebs/status/779111614226239488 https://twitter.com/briankrebs/status/779062433902170112
It would be interesting if he started writing on Medium (not saying technically advisable, just interesting). I wonder if he'd ever consider trying that.
Why would it be interesting?
To see how Medium would deal with massive DDoS attacks and whether they'd kick Krebs off.
Why would he give away advertising dollars?
Also, medium is bad. Everyone now thinks if you publish something on medium, the writing is suddenly a masterpiece.
Who profits from this attack?
Someone's ego.
I'd love to learn more about these botnets. I wonder about things like What's the average time that a compromised computer stays in this net. What is the typical computer (grandmas old PC running XP). Do the ISPs ever get involved to kill bots running on their networks?
for the most part ISP are not set up to defend or stop DDoS from within their network. Do you go to a road building company to stop car thieves?
If that road building company also collects tolls and controls access to their roads you sure as hell do
Well that's the state. His analogy is like saying 'why would your fiber splicer police this.'
no you don't because that's a bad analogy. We do however go to states and could ties to protect the infrastructures they paid the contractor to build.
ISPs are uniquely situated to stop this kind of ddos because the traffic originates from IPs they don't own. The traffic has a spoofed from address. And as a rule, the ISP.should only need to send traffic out of a neighborhood from the block of IPs that is assigned to that neighborhood. You can put a filter on every switch or even every interface allowing only traffic from the IP or IPs on the other side of the link to send traffic. A company like Comcast could make it default part of account setup scripts. If everyone did that, these would disappear over night.
Aren't there botnets that use their actual source address when spamming a DDOS target? How do you defend against that type of attack?
If a bot is using its actual source address, you can block its IP at the edge (and even ask the ISP to investigate). Thanks to IP spoofing, that's totally ineffective, so instead the only way to make the attack stop is to null route the host.
How does IP spoofing even work outside of those DNS reflection attacks mentioned on Krebs' blog? [1]
I constantly see references relating to DDoS attacks about how IP spoofing is such an obvious trick to use but I've never seen any way to actually do it. Why wouldn't every device on the internet spoof their IP?
[1] https://web.archive.org/web/20160922021000/http://krebsonsec...
https://spoofer.caida.org/summary.php - compromise a device in one of the ASes not marked "unspoofable." Those ASes do not consistently perform packet ingress filtering.
That's not to say that DDOS attacks stop being possible, but at least they become traceable.
I appreciate the response. I had no idea things were this bad.
Thank you, I've actually been interested in the answer to this question for many years! Never was able to solve it via googling.
What do you do when you have 10, 20, 50 million bots using their actual source address? Do you just block all 50 million devices? If so, then that would be a great way to initiate denial of service on those devices.
Actually, that would be a good way to take all those insecure devices off the internet, or at least prompt someone to do something about it (even if temporarily, like hitting the "factory reset" button) before reconnecting them.
Most countries don't allow cars on the road that are unsafe due to lack of maintenance. Perhaps it's time to do something similar for internet-enabled devices that cause serious harm to others. Hold the user, manufacturer, or network operator responsible for harm caused by their lack of maintenance.
Yes there are. but using your source address means that you are confined by the upload bandwidth of your hosts, instead of some mis configured DNS server with a 1000Mbps up in a datacenter. You'd just have to work a lot harder to get the bandwidth.
to add to that, You'll defiantly get some mis configured servers with 1000Mbps uploads. And those will be really easy to pick out of the lineup. And then you'd probably be able to call the DC and say that they should block that IP at their boarder and they would probably also comply because there's a good chance that customer that was doing 110Mbps and won't want to pay for 1000.
As it is now, because the source is spoofed, you can't really take the source offline, only take the destination down to keep the other hosts in close proximity running.
With a TCP connection you can pick the source and drop the handshake, basically never start the connection. Some of the windowing can be used to make a tcp connection less of an issue as well.
Most ISPs have a clause in their Terms of Service stipulating that you won't use the service for criminal activity. I'm pretty sure a DDoS can be argued as a criminal activity. So, it is in the ISPs right to stop service to those nodes. --Yes, I understand the consumer isn't the criminal, more so they are the victim and the crime took place on their service.
Further, technologists tend to be pretty good at solving problems. I know this isn't the ISPs problem, but it is a flaw in the network, I'm simply wondering if anyone is attempting to solve this problem at the network level rather than simply building bigger caching services to protect those that pay for protection.
This is an interesting video examining a botnet.
https://www.youtube.com/watch?v=PgDY3C8SEmY
This was a mixed traffic botnet. There are speculations that a large chunk was from compromised IoT devices.
How many IoT devices are out on the public internet at the moment?
and for the existing IOT devices, are they the same thing, or were different exploits used for different devices?
How many on the public internet? Too many. Here are close to 100k IP cameras on the public internet: https://www.shodan.io/search?query=ip+camera
Again, the speculation that it is IoT devices is unfortunately just that. However massive compromise of internet connected embedded device is not new: http://internetcensus2012.bitbucket.org/paper.html
Are there any good quality articles you can link to discussing this?
Good quality? I'm not sure. Just speculation on this one post: https://www.hackread.com/brian-krebs-website-665-gbps-ddos-a...
Got a source for that? Thanks
Just this link: https://www.hackread.com/brian-krebs-website-665-gbps-ddos-a...
I'm a bit concerned about the reliability of that report since there are no strong proofs for their claim.
Interesting. Here's a bit more on IoT botnets
http://blog.level3.com/security/attack-of-things/
Good luck updating those embedded linux devices, or even alerting the people who own them
A botnet of coffee machines, thermostats and Teslas? :D
Are there web servers or software that blacklist IP addresses that disconnects after a short time and redirects them to a static page?
Yes, but they can be flooded as well. Every system is going to have a finite capacity. And if the capacity is exceeded, the system will slow. If the capacity is significantly exceeded, it will become unreachable itself.
So if the capacity of your system is X Gbps, then it will start to have problems if the attacker sends X + 1 Gbps. And will probably be completely unreachable if the attacker sends X * 2 Gbps.
But more specifically, whoever launched the attack cost them that money.
Also, ha:
PING krebsonsecurity.com (127.0.0.1): 56 data bytes
I would say that is a clever move, but to be honest that is the most he can do now.
https://twitter.com/briankrebs/status/779144394360381440
It might be more useful to return the IP address of whoever made the DNS query.
This could trick the computers that make up the botnet to either attack themselves on the public interface (more resource-intensive than trying to DDoS your own loopback), or even better, their ISP's resolvers (it would force the ISP to do something about it).
With the recursive nature of DNS, I imagine that could get a little hairy as the DDoS'ers would then be targeting whichever DNS servers they were using.
Reminds me of: https://twitter.com/troyhunt/status/716408697266679808
He should get a Facebook page and publish a copy of all his posts on it.
Interesting idea, but probably doesn't go far enough...
Perhaps he should re-post his blog articles everywhere: Facebook, flickr, tumbler, watpad, wordpress, various feedback forums, etc.
Combat a DDoS attack with a DPD (distributed publishing defense - just made that up)
That would harm his business model, which is advertising.
It could work with Facebook Instant Articles, he may even be better off using it since they source the advertising and have been out trying to poach and source quality content.
I was going to say a different version of this.
Real men mirror.
Krebsonsecurity deserves to be on git and use something like Jekyll. Mirror it instantly in a hundred different places.
Why do you have to bring gender norms into this.
I'm sorry if it came off that way .. I thought the reference was well known.
http://quotes.yourdictionary.com/author/linus-torvalds/19029...
"real men do / don't do X" long-predates Torvalds, for example https://en.wikipedia.org/wiki/Real_Men_Don%27t_Eat_Quiche
Well it was specifically about mirroring. In that context, the Torvalds quote is actually pretty popular.
But I apologize nonetheless - it was not my intention to make a sexist joke.... Just a geeky one.
And give Facebook ad $ for nothing?
Isn't this whole thing a bit silly? I mean what's the point? They just spend time on making him the best marketing, he'll double his audience/readers, no?
The point isn't to cost Krebs readers most likely. It's to show off how awesome this bonnet is.
I'd guess the DDoSer is jumping with joy over this news actually, because now the DDoSer can advertise his service with "I DDoSed Krebs so hard Akamai had to drop him!"
They have shown who is stronger and probably got some PR for their DDOS services.
so long for using a CDN to protect from DDOS attacks...
There are different tiers. Some people can get away with using a CDN because they aren't that big of a target.
This was prolexic, a ddos protection service purchased by Akamai.
I understand that this is burning bandwidth for Akamai, but seriously, taking into account what is at stake here, I think they need to do their share and continue to support Brian.
Why don't we switch to a distributed network with a DHT like freenet? So many benefits, including not being able to take down content via DDOS.
The DHT can still be attacked. Here's three methods:
1: Take out the bootstrap nodes. These are several nodes that bootstrap a new client into the DHT system. BitTorrent, Inc. keeps a couple such nodes. On first boot, the client registers it's DHT address and collects a few from the bootstrapping node. The client could then can traverse the network itself. By knocking out these nodes, newly started clients now have to browse the whole IP space for possible DHT clients, which is not feasible.
2: Attack the peers themselves. A malicious program could traverse the network searching for DHT peers in the same way. At first, it would only collect a large number of DHT addresses and their corresponding nodes. Once a sufficient mass is gained, each is targeted with a low level DDOS to knock them offline to further requests. Most of these peers will be homes and local ISPs, which can't effectively deal with DDOS traffic themselves. Others trying to connect to a down client will eventually remove them from their own address space for later queries.
3: Poison DHT peers. This is probably the hardest, but once complete could poison an entire network with a switch. On each of your compromised Bot machines, you make a valid DHT node. Make a LOT of these (like a Botnet). For the most part, participate correctly with the DHT network. Collect as many valid/real DHT user and content addresses as you can and host them in your nodes. When it's time to attack, prevent these valid DHT addresses from resolving on inquiry. Even better, make them go in the wrong direction and infinitely pass around requests to other poisoned bots in your ring to prevent resolution but not hang the process. This is especially useful for content attacks because it attacks the content addresses themselves.
1a. A system could be made for bootstrap links that have the addresses of a few nodes in them.
1b. When you have 10 million nodes like torrents, you can go searching random IPs. As long as many nodes bind to the same ports.
2. Sure, if you have comparable bandwidth to the entire network you can take it down. But that's a lot harder than overwhelming a single target. Nobody can send 20mbps each to millions of IPs.
3. This is the method that takes the least resources, but pretty good countermeasures can be made.
You can say the same about any network. What you are describing is an attack on the WHOLE network (such as what Bruce Shneier recently reported), not a specific file. You can try to bring down the internet, but not a site.
Here's a link to the last post from his website. Google did not appear to have this cached:
https://archive.fo/t94ve
relevant quote from that: ››Many readers have been asking whether this attack was in retaliation for my recent series on the takedown of the DDoS-for-hire service vDOS, which coincided with the arrests of two young men named in my original report as founders of the service. I can’t say for sure, but it seems likely related: Some of the POST request attacks that came in last night as part of this 620 Gbps attack included the string “freeapplej4ck,” a reference to the nickname used by one of the vDOS co-owners.‹‹
I'd err on the side of caution and consider that string a Red Herring without more evidence to corroborate the claim. It is a nice coincidence though...
I'm just here for the popcorn :)
This is bad PR for Akamai and a tactical error for them to boot Krebs even if they were providing free service.
To some, the implication would will be "they couldn't handle it" so why should I trust the DDOS they are heavily promoting on their site?
At minimum they should comment on the situation, at best restore his service and learn how deal with high profile clients.
Even if they tried to mitigate and quietly semi failed (like 30% packet loss) the PR would have been better. It could be that such attack takes down their entire network hard. Verisign said a year ago to us they could mitigate 2Tbps for comparison.
Akamai can easily mitigate this.
They just don't want to provide it for free.
And pretty much no one can afford to use their services @ 655Gb/s for that long unless they had billions of $.
Well a single 48 hr mitigation costs 12k/yr contact. Unlimited mitigations probably cost about 130k/yr. Either this negative PR is worth 12-100k in savings for them or they would drop any paying customer if the attack is over a certain threshold?
They might bill you that money, but on average it costs them less to provide that service to you, otherwise they wouldn't make any money.
Because we don't know the operating margins and the distribution of DDoS costs per customer, we can't infer how much this particular attack would cost Akamai.
According to Kerbs (https://twitter.com/briankrebs/status/778398865619836928) Akamai did mitigate this DDOS, they kicked him of after the attack, not during.
Maybe because of the scale of this attack, relative to the business value Krebs provides Akamai? Shedding an attack of this magnitude (by dumping Krebs) was probably well worth the PR hit Akamai took.
Brian Krebs' wasn't a paying customer right? Akamai provided the service pro-bono. Perfectly acceptable for them to suspend service if it becomes more than trivial in terms of cost or it puts their paying customers at risk.
Wow, I figured that everyone that had hired vDOS would be irritated but that is pretty impressive. Still it says a lot for how effective he has been at rooting out this stuff, not like the TierN infrastructure folks have managed to track this stuff down with their resources.
I've always wondered if your domain is under a http DDoS attack, couldn't you in theory update your DNS A record to another ip and take other servers down (maliciously)?
Most DDOS attacks are launched directly against the IP, so it wouldn't really help.
Here's a "philosophical" question with regards to the internet, and perhaps even it's future. Given that a currently anonymous attacker, and likely not a "state" player (i.e. not a governmental entity with almost unlimited resources) has managed to DDoS a single website, does this portend that unless there are significant changes to the way the internet infrastructure works, we are seeing the demise of the WWW?
Kind of like a reverse wild-wild-west evolution, where the previously carefully cultivated academic and company site presence, gradually degenerates into misclick-hell? And the non-technical, non-IT savvy masses, in a bid to escape this all, end up in a facebook-style future where media is curated and presented for consumption (or perhaps in future, facebook-type entities end up with their own wild-wild-west hell)?
I have a strange feeling that we are seeing the decline of a city/civilisation; once you used to feel safe walking out at night, knew everybody in the neighbourhood, could leave your doors unlocked... and now, you don't dare to go down the lane to the left in case you pick up a nasty virus, and if you hear a knock on the door at night/email from DHL, you don't dare to even look through the peephole/preview the JPG!
Perhaps its the government? Lots of people think all these NSA programs are evil surveillance, could be it's just the glue that has saved us from just this.
I'm personally amazed that people don't get hacked more often TBH... I can't think of any instances where non-technical people have been pwned in my own life.
I personally have a pa55word that I use for sites I don't trust, but the accounts never seem to fall or even falter. It's amazing really.
There's too many people in the general population to target them one by one, but
... the machines participating in the DDOS are (almost entirely) hacked, yes?
You are not the first to come up with this idea. This same thought has been posted every year for the past 20 or so years in mailing lists, forums or Usenet (thought lately, not too often to Usenet).
I think prevention should be emphasized. If there wasn't so much garbage plugged into the Internet, there wouldn't be huge botnets to send DDoSes. There are few groups that scan the Internet for vulnerable systems, and rather than compromise them, send notices to the ISPs. In Canada, the CCIRC does this. But they only check IP blocks assigned to Canadian ISPs and enterprises.
Plus, why do so many ISPs still allow spoofing of IPs? It isn't 1999 anymore.
We should start a grass roots group to talk to everyone they meet, and get people to update their OSes, devices, and get rid of crap.
Which pretty much illustrates the worst-case outcome: spam and trolling rendered completely worthless.
The quickest way to do achieve that would be to hold ISPs legally responsible for any damage caused by their failure to block spoofed traffic from their own network.
I'm not sure how well this would work outside of the U.S. though. Not everyone is as litigious as Americans are.
Should not a router be able to this filtering ? Making Internet Exchange Points do it would be even more quicker. Once big IXPs do it smaller would follow suite for the fear being cut-off and eventually ISPs.
Filtering can only be enforced at boundaries where an operator can say "Link N will only/never have traffic for net range foo/16." And it isn't always possible to make strong blanket statements like that.
Netadmins can make those kinds of statements about traffic originating from with their own networks because they set the rules. But at an interconnection the types of networks connecting, and the purpose of the connection might mean there is little meaningful anti-spoofing protection that can be done.
For example: I send a packet to google, it passes from AS 123 through AS 456 to AS 789. How is AS 789 going to tell the difference between a packet from me, and a forgery originating from AS 456?
It cant. One solution would be blackholing AS 456 from AS 789 at the requrest of its members. Hopefully this will teach 456 to stop misbehaving.
Though we do assume that AS wont itself misbehave and send a spoofed packet to one of its member peer and most of time its true.
We have to worry about misbehaving ISPs for which previously mentioned filtering works.
I dont think so. IXP can force peers to provide their IP Space even if its whole internet. At least they wont be able to spoof IP outside of their space. If they do spoof ddos from their own space the above solution would probably suffice.
EDIT: I just realized peer already has to give destination ip ranges. So IXP dont have to force anyone.
The latest attack wasn't using spoofed traffic, from my understanding. Hacked devices were directly sending traffic.
And I ask you this, how is an ISP supposed to know if a device is hacked, or for example, is a webcam uploading a stream to a redistribution site. It can take days to chase down all the IPs, even in the US, and get the ISPs to deal with them.
"Here's a "philosophical" question with regards to the internet, and perhaps even it's future. Given that a currently anonymous attacker, and likely not a "state" player (i.e. not a governmental entity with almost unlimited resources) has managed to DDoS a single website, does this portend that unless there are significant changes to the way the internet infrastructure works, we are seeing the demise of the WWW?"
Peak Internet:
http://blog.kozubik.com/john_kozubik/2010/12/peak-internet.h...
Some are guessing the DDOS was because of this recent post of his, about a large DDOS network.
http://webcache.googleusercontent.com/search?q=cache:kaymYsb...
You'll be redirected in... never redirected.
I think its time for some serious financial incentives for ISPs to start getting serious about routing (or rather not routing) garbage. Financial fines for every DOS originating from your AS, or blacklisting if you are a repeated offender.
There is an incentive: it is the cost of transit. However, there usually are not a lot of zombies per single ISP for the access level ISP to even see any abnormal traffic.
The best thing is that access ISPs need to implement BCP38 (https://tools.ietf.org/html/bcp38). And shutdown all open recursive DNS servers. It would be great if Microsoft didn't ship such a retarded DNS server too. I would say that most ISPs do not do this.
NTP really should be replaced with something better. There are still large numbers of NTP amplification attacks going on. The big issue with NTP today, is that by default ntpd in daemon mode, is also a NTP server and responds to NTP requests. And so many of the two bit home routers run ntpd.
But the reality is, that no one is even reporting DDoSes right now. I work at an ISP, and I haven't seen a DDoS report in the past year. We pro-actively scan for open DNS and open NTP services. But many DDoS attacks just use regular HTTP/HTTPS, are hard to detect at the individual network connection level. Do you think Akamai sent out a single notice to any ISPs, saying "The following X IPs are sending excessive traffic to site Y, and are suspected to be part of a botnet"?
Should've used CloudFlare.
So if Akamai can't hold an attack of this size, who can?
Or is it that they actually can hold it off but it costs too much money?
Cloudflare and OVH (that got a 1.1 Tb attack around the same time, the biggest in history) certainly have the capacity to hold attacks of that size.
Akamai may also have the capacity, but bandwidth is not free.
Still not a good move for Akamai, though.
I get him speaking out for them about the hosting having been free, but Akamai is now the CDN that got bullied into kicking someone of their service against their own will.
Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure.
I know it's a crude comparison, but we don't negotiate with terrorists for a reason.
I agree and I could see them reversing their stance now with the press around it. More than likely a middle manager approved the pro bono usage and once it got expensive that manager found him/herself in hot water over the resources being consumed. A few apologetic emails to Krebs from that middle manager, but saying he had to go, and boom we find ourselves here.
I would bet money that the attack was truly epic... to the point where it was impacting (or was about to impact) other Akamai customers. Because you are right, this is terrible PR, and Akamai knows it.
DDoS mitigation is fundamentally a problem of "who has more bandwidth?" - if the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over.
The problem is that the economics, right now, are heavily tipped in favor of the attacker.
Seems like it not only was one of the most expensive attacks so far (and by far the biggest one to have ever hit Prolexic, according to them), it also made little use of reflection of amplification making it much harder to mitigate.
That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers.
I... wonder if anything different would have been done for a paying customer. I mean, if the attack was big enough to take down other customers, and if Akamai had the choice between kicking one customer and all customers being down?
I'd expect them to have gone further and probably invested/committed more resources to it.
Why's that? The theory is that it was impacting other customers. If the notion is that they kick off the low-revenue person to preserve the larger revenue stream, then it would seem that the only way to get real protection from Akamai is to be their largest customer.
Krebs was not a "low-revenue person" he was a "no-revenue person" they provided the service pro-bono. With customers they'd have a contract, and while I don't know Akamai's contracts I assume they either have specific service clauses and/or "use clauses" where protection costs get charged to the customer.
Nope. Contract basically says "here are the service levels Akamai commits to meet" and "Customer's sole remedy is to cancel service if Akamai does not meet these levels." Looking at their contract language does not make me feel any better about whether they'd drop a paying customer in similar circumstances. (Quoting from a version available online at http://contracts.onecle.com/akamai/msa.shtml)
8.1.3 Akamai shall meet or exceed the network availability, capacity and operations levels as set forth in Section 2 above; provided that Customer's sole remedy for the breach of this provision by Akamai shall be the termination rights set forth in Section 10.2 below.
10.2 TERMINATION UPON DEFAULT. Either party may terminate this Agreement in the event that the other party materially defaults in performing any obligation under this Agreement and such default continues unremedied for a period of thirty (30) days following, written notice of default; provided, however, that in the event this Agreement is terminated by Customer due to Akamai's breach of its representations under Section 8.1.3 above and failure to cure, Customer's sole remedy shall be its election to terminate the Agreement without further liability to either party (except for Customer's obligation to pay all accrued and unpaid fees outstanding at the date of termination).
Having not committed a 620 GBps DDos attack on a blogger I don't like lately, how on earth do you generate that much traffic without reflection or amplification?
Asking for a friend. /s
Pure speculation but you can get and click a VPS/dedicated server with a 1Gbit/s pipe almost everywhere. Most of these machines are not regularly patched and run often vulnerable PHP software.
I guess if you have a few 0-days for common stuff that is often hosted you can easily!? collect a few thousand servers that blast out traffic.
Additionally more machines have faster access to the internet as fiber gets more popular, if you have a decent sized botnet that can pump out traffic on average with 50mbit or so you can also get some serious traffic, not sure about getting 620GBps, through. I guess if you mix all of these and put some amplification attacks on top it's possible.
http://webcache.googleusercontent.com/search?q=cache:http://...
You hack into a bunch of servers, PCs and smartphones. You don't even need that many with modern connections.
Yup. Cable subscribers in the USA alone have enough aggregate bandwidth to knock Akamai totally offline if 620 G-bit is something they can't handle. My own uplink is 10 M-bit. 100,000 compromised connections that speed is enough for T-bit scale attacks, and compromising 100,000 machines in the new world of "Idiots of Things" is a no-brainer given almost nobody working in the IoT space even gives a nod towards security.
620 Gbps, not GBps. Don't create more confusing between (B)ytes and (b)its. You and I know its bits, but other readers might not.
Oh. My. God. This is in text so it is going to come across as sarcastic but I 100% guarantee you it is not, I am 100% serious. I have been making that mistake for over a decade and yes I do know the difference but I've still been writing it down wrong every single time it's come up. Why did nobody tell me before now?!
I am genuinely grateful you pointed out my error, it's these little things that people judge us on when we write emails and technical documents.
Please excuse me now though, the Englishman in me demands that I hide under the bed out of shame for the next 26 hours.
I certainly did not expect a response like yours. (Yes I would have totally taken this sarcastically!) I am glad I helped you learn something :)
From the article: "The assault has flooded Krebs' site with more than 620 Gbps per second of traffic — nearly double what Akamai has seen in the past."
Sounds pretty epic.
It's accelerating!
If it takes a mere 620 G-bit to screw Akamai, then they're obviously not much of a content distribution platform. I only need a few thousand compromised machines in the right countries to run 5+ T-bit scale attacks. This isn't the 90s any more, and Akamai has no reason to not have improved bandwidth capabilities, unless they're doing like the Telecoms companies by saving that money and never using it to improve their own infrastructure.
I was impressed by that number too, but after reading your comment it seems awfully small. In 2016, what kind of attack do you expect CDN like Akamai to handle?
Chattanooga has symmetrical G-bit fiber to the home. 5,000 Chat-town residents got uppity, think of what would happen to Akamai if they couldn't deal with 600-ish G-bit.
Yeah if that network was designed to handle continuous 1gbit per home.
It is, as it's a municipal network and not a shitty company-owned one. Designed right from the ground up from day one, much like the fiber service in Sandy, Oregon (300/300 for $40, no limits, caps, throttling, nada.)
Don't worry about his comment. He has no idea what he is claiming. These recent attacks are labeled record breaking because they actually are. This one at 620Gbps and the recent ~1Tbps against OVH are the biggest in history, and still 5x less than what he claims.
Of course we will get there sooner than any of us in infosec want, but he is almost an order of magnitude off of what realistic threats look like.
Almost an order of magnitude? You obviously don't know what that means. I'm at half a magnitude of order off by your supposed words (protip: An order of magnitude meas you add a zero to the end of the number you're using,) and you know not much about CDNs if you don't think that multiple terabits of traffic are flowing through Akamai every second already.
Currently, I do the physical networking builds for a mental health company. We're already deploying 100 G-bit in these offices as primary connection trunks, because a lot of these services will be done remotely over video and audio.
I could open up a 20,000 user Camfrog Video Cluster chat room and could saturate a T-bit connection link just like that the second it's half-full. Have you ever used (let alone seen)a T-bit scale program before? Camfrog's been out for over a decade.
Tangent: orders of magnitude are exponential - if you want to say "half an order if magnitude" you have to do it along the exponential curve. 5x is about 0.7 orders of magnitude; half an order of magnitude is a bit over 3x.
Especially since Akamai has placed themselves in the cdn market as "WE cost more, but we are the best there is." Hell, a ton of other "cdns" are merely reselling Akamai with friendlier contracts and features.
I'm going to call shenanigans. Do a quick google for the largest DDoS attacks on record, and this is one of them if not the largest. Pulling from places like Arbor, and their yearly reports, the largest previously seen were ~500Gbps. I seriously doubt you and "a few thousand" machines can magically be 8-10x stronger than the largest attacks on record.
I would love to see some sources that ANYONE can get close to that number. Short of the NSA bringing its full power to target a specific pipe, I don't think we're there yet.
I'm willing to bet you're not including compromised backbone routers and symmetrical gigabit-fiber connections. There's enough of the latter in the USA, to homes, to do that much and then some.
But what you're arguing isn't reality. Show me a source article where someone has compromised a backbone router, and then used it for DDoS. This is almost exactly what I was addressing when I said "Unless you use the power of the NSA to target a single pipe." Even in a hypothetical scenario where you have gotten your hands on one: How long do you think companies are going to let their half million dollar router be consumed for a DDoS before they take notice?
I think its pretty obvious you don't understand how internet traffic really flows, when you think "all I have to do is compromise 600 pc's with a Gb connection and I can launch a 600Gbps DDoS."
"I think its pretty obvious you don't understand how internet traffic really flows, when you think "all I have to do is compromise 600 pc's with a Gb connection and I can launch a 600Gbps DDoS."
I've been doing networking for 26 years. One of my largest jobs was mitigating Slashdot effect for two high-profile sites. I know very well how a DISTRIBUTED denial of service attack works, can work, and have done many of my own in checking security measures for those whom I consult. Compromising backbone routers is actually fairly simple. Too much reliance upon software stacks and not enough reliance upon sound hardware logic design that's proofed against attack in the first place.
Is there no way to stop such attacks by coordinating with your upstream ISP, or with the sources of the traffic? Why do backbones allow it to be carried? Is the problem that these attacks are too many different individual streams to identify and filter?
It seems like there ought to be some way to hierarchically punt the problem to network operators. "Your network is contributing 10 gigabytes per second to this DDOS attack. Identify the sources and shut them down." - times each identifiable traffic stream.
Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Maybe a good-faith collaboration of different ISPs could result in quick shut-downs. Or if networks don't cooperate, then the next-nearest border does it for them (and if they don't like the policies under which their neighbor suspends the traffic, they can sign up to do it themselves). Imagine something like a mini automated DMCA type request. "This IP is DDOSing me", signed by the operator of a reputable network, having the effect of suppressing origin traffic from the IP when received by a reputable network. (Any abuse of the mechanism causes the network to lose its privilege to participate, and DMCA requests related to that network fall upon its neighbors.) Perhaps the suppression would be destination-limited so as not to be vulnerable to too much abuse of the mechanism.
It is possible to mitigate, but it takes time.
Now hackers have a new attack vector.
Without net neutrality economics would likely fix this. "We'll limit our 60GBps traffic to this network block because our peering arrangement makes that expensive."
Couldn't an ISP just throttle traffic to a particular network block on receipt of an authorised request.
And that seemingly provides a general solution. Record output to particular network blocks and throttle traffic when it peaks beyond statistically normal bounds? Basically, applying damping.
The problem is a lot of traffic is spiky if you look at it from the view of any given network provider to a site that isn't always getting traffic.
So distribute the DOS traffic enough and it will be very risky for providers to throttle it without appearing to be overall slow for a lot of legitimate traffic.
The other problem is determining what an "authorised request" involves. As it stands it is already a problem that people can - and now and again do - manage to mess up routing tables by sending broken route announcement, re-routing large address ranges to the wrong location.
Too much of internet routing still relies on a large amount of trust. We unfortunately need less of that, not more.
Eventually that might lead us to a situation where we could properly authenticate and authorise requests like what you suggest, but today it is high risk.
One thing perhaps ISPs could do is share data on these attacks, and work to shut off the botnets after the fact- so they can't be used again. So for example if Akamai shared information with ISPs about which orgin IPs were involved in the DDOS attack, the ISPs could reactively reach out to the affected customers. Or the ISPs could even try to examine their own logs after hearing a report of a large scale DDOS attack to determine if any of their own nodes were part of it. There may be also measures ISPs cold take on their CPE routers to detect infected machines on the customer network and warn them more pro-actively.
And governments can easliy silence "nuisances" like human rights lobbysts.
Governments can already block traffic on their border and can organize attacks again foreign targets (remember China vs Github case).
Oppressive governments can do even worse things to human rights activists, like banning a person from international travel (like Mr. Snowden or Wikileaks founder) and getting them extradicted from other countries.
The Internet is supposed to be a dump series of pipes. However, in practice you can of course try a traceroute and then complain to the ISP, which will or will not cooperate. If the ISP does not cooperate, then you can call their upstream provider and kindly ask them to call their customer and yell at them, but note that money is flowing in the wrong direction for that to work.
Plus you get the problem that you are usually seeing DDOS traffic from innocent bystanders, if the ISP is shutting of a compromised home router, then their customer will not have internet for some length of time.
Why not simply block traffic from the specific unknowing customer to the target? I.e. Allow all normal traffic from the customer to go through, except for the traffic that was identified as being part of a DDoS? Is it expensive for an ISP to do that type of intelligent/rule-based routing?
It would probably take too long to get the ISP to comply, and I suppose the targets keep changing with botnets
Shutting off Internet for people housing compromised devices does not seem like a bad thing, and it is probably the best solution that exists.
It probably seems like a bad thing to the customer paying for internet, and since they're the one paying for service they do get some say in the matter.
Which do you think costs Comcast more? Temporarily disabling 20,000 customers' internet connections, or forcing Akamai to drop Brian Krebs as a customer?
I think someone who has equipment engaged in DDoS attacks might be a bigger cost to the telecom than they are paying anyway. Those 20,000 customers are either using disproportionately more bandwidth and resources or they are at least unwittingly involved in illegal activity, so Comcast would have either a financial incentive or social obligation to temporarily suspend their service. The company should be complaining to the customer, and they should be worried about losing their service. If every competitive provider had these very narrow standards, customers should be more worried about not having Internet because of their security illiteracy.
You can capture the IPs, but they aren't going to be the true originating IPs. Usually they are just random IPs generated on the fly. Or sometimes the attacker will pick a company or ISP and spoof the packets with those IPs.
There "is" such a way, see RFC 5575, but it seems to be little known/implemented/deployed/enabled/something.
5575 is a BGP extension that says "for packets from x to y, do z". Assuming a router knows on which if its input ports such packets arrive (and during a DDoS it doesn't have to wait long for the next packet), it can disseminate the flow specification towards the actual source(s) quickly, so the packets can be dropped quite far from the DDoS target, in the ideal case as soon as it reaches an honest ISP.
Egress filtering should kill much of the spoofed-origin traffic and this much of the rest — if deployed.
I'd love to know why 5575 isn't deployed. Memory concerns maybe?
You cannot stop attack because web is decentralized. You cannot tell other Autonomous System to stop sending traffic to you.
Because they get paid for any traffic passing through. The more traffic they have the more profit they earn.
http://packetlife.net/blog/2009/jul/6/remotely-triggered-bla...
What would have been a bunch of positive earned media is now a flood of negative reaction and lingering questions that they buckled under the pressure.
The biggest DDOS ever, and Akamai dumps the client rather than defend it.
However they spin it, doesn't look good.
Exactly this.
Frank Leighton needs to make a statement about this, immediately, and he better pull the Rabbit of Caerbannog out of his hat.
Definitely. The lesson I'd take from this is that Akamai isn't serious about DDOS protection.
For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim.
It strikes me as especially bad that they're doing it in the moment. It'd be bad enough if they said, "Sorry, Brian, this is too big a distraction; you've got 90 days to find a new home." But that they're dropping him in the middle of an attack? That means I can't trust Akamai.
They gave Krebs the service free of charge.
I mentioned that above, but I think this misses my point, which is:
If Akamai can't provide their service for free, then they shouldn't provide their service for free.
A CDN's whole business is resilience, which in this case makes them the bodyguard, not a bystander.
Whatever your opinion of Cloudflare, it seems clear to me that Matthew Prince keenly understands this, hence him reaching out and offering to step in and get Krebs back online.
tldr; If Akamai can't do the one job they exist to do in the face of an (albeit well armed) assailant, then they're the problem, not Krebs.
They gave him service for free and covered many, many smaller attacks, including the 20-100Gbps attacks he reported earlier this month. The fact that they decided not to cover one of the largest attacks ever documented pro bono isn't great but I don't think any of their clients would fail to understand the difference between a favor and a signed contract.
"This was flagged to my attention and I've reviewed all the interactions between the author and our team [cloudflare]. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection.
The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network."
https://news.ycombinator.com/item?id=5214480
That was 3.5 years ago. CloudFlare's ability to deal with DDoS has changed substantially in that time and we deal with enormous attacks day in day out automatically.
So how would you feel about 600gbit+? I'm genuinely curious, would it be an interesting challenge or an immediate avoidance? As a business customer myself and a big fan of CF's general attitude to these sorts of things, I know be very happy to see the blog post on it.
We've offered to help Brian Krebs out. We don't see any reason we could not handle an attack of that size. We've already seen gigantic attacks and have a very well developed automatic infrastructure for dealing all manner of attacks at different layers and an experienced 24/7 network team.
Honestly I didn't know about this instance, but as another poster has mentioned, I don't believe this reflects where Cloudflare is today, and that's who Akamai is competing with.
On a side note, they really don't appear to be making a concerted effort to get out in front of this which tells me that they either aren't aware of the reaction or don't think it's a big deal.
Either way, it just makes them look bad.
But he didn't buy anything. We can't extrapolate what happens to paying customers from the experience of a non-paying customers.
That being said, I definitely agree with your thoughts on insurance.
So, if he had paid one cent (thus being a paying customer), you could extrapolate?
I don't see how the price is in any way relevant here. They promised to protect him, and they failed to do so. Claiming afterwards that the premium was too low isn't the way this works.
Also, I doubt that it actually was "for free". He may not have paid in money, but likely in the form of (at the time positive) PR, for example.
You're right the actual amount of money is not relevant. What is relevant is that the contract he signed with them is not the contract paying customers sign when they do business with Akamai.
Since we have no idea what was in the contract this guy signed and it's all speculation, this discussion is totally vacuous and pointless.
How do you know what they promised? They could have promised protection, or they could just as well told the guy "hey, here is some free caching for you, m'kay? No strings attached". Hell, it's possible he didn't even sign anything, and there wasn't a contract at all!
If he were a regular paying customer, I would make the assumption that the contract he signed is likely the same, or similar to the contract I would potentially sign, and this would put Akamai in a very bad light to me.
Since the contract this guy signed is not the contract I would sign, I cannot rationally infer any information from this incident, good or bad.
If Akamai emails me and offers me some free service, then yes, this information would be valuable and relevant. Until that day, I can't make any use of this information.
You are missing the point which is that Akamai can't handle this DDoS.
Do potential customers care if Krebs is a paying customer or not? He went with them as they offer this service which apparently doesn't work as well as advertised.
There is no indication that Akamai can, or can't handle the DDoS. The only information we have is that they are only not willing to do it anymore for this particular customer. There is no indication that they won't do it because they lack the technical capacity to do it. Just as well they might not do it because this thing is financially disadvantageous to them.
As a potential paying customer, what they can and can't do is covered by their SLA, and that's all that matters. If they break their SLA they own the customer compensation. This incident is irrelevant.
Of course I don't actually know what kind of SLA and indemnification Akamai provides. Maybe it's bad. Then after analysing the contracts I would make an informed decision. These things are what I use to make decision, not random stories with no technical or business details on random blogs.
From what I've seen (quoted elsewhere in this thread) there isn't any significant penalty for Akamai if they are unable to mitigate, or choose to not mitigate, a DDoS attack. They might negotiate other terms, but I doubt it. DDoS mitigation is, by its very nature, a best-effort service, and reputation for not giving in to attackers is more important than any contract terms you're reasonably going to be able to get.
I haven't read anywhere that they failed. Isn't the timeline:
- Akamai mitigates attack just fine
- After the attack is over, Akamai does the cost/benefit analysis and decides to stop providing the free service.
The attack was ongoing when Akami gave Brian Krebs 2 hours to find alternate hosting.
The attack showed no signs of waning as the day wore on. Some indications suggest it may have grown stronger. At 4 pm, Akamai gave Krebs two hours' notice that it would no longer assume the considerable cost of defending KrebsOnSecurity.
http://arstechnica.com/security/2016/09/why-the-silencing-of...
I had some friends who worked at Akamai. I always got the impression that they were very serious about addressing anything which could disrupt service, including DDoS.
Yup. And it's those people I feel bad for. I'm sure I would have been one of the tech people saying, "We must not give in! Let's use this as incentive to keep upping our game. That's the only way we'll win in the long haul."
You're entirely ignoring the fact that there's no way Krebs could've possibly been paying Akamai enough to tank the attack.
DDoS protection isn't insurance, Krebs gets attacked 24/7. Only an utter moron would be willing to sell Krebs DDoS insurance.
Which means nothing at all in a world without alternatives, hosts capable of tanking attacks like that number at two or less. But I get the impression you're not looking to spend hundreds of thousands of dollars a year on DDoS protection anyway.
They were hosting it pro bono. He never paid them enough to do anything. And yet...
But a smart person would cover him for free as a way of proving that they could handle the worst the DDoSsers gave out. To prove that they stick by their customers.
Most people who buy insurance never really use it. So what are they buying? A feeling of safety. Just think about the various insurance company slogans that come to mind.
I can almost guarantee CloudFlare will offer to step into the breach.
In an ideal world, they would not cave... but someone has to pay the giant bill. Who would chip in?
The site must be constantly under attack, and it must cost Akamai a fortune in real $$ all year long. And, when their service is performing well, nobody is talking about it, so there must be very little positive PR.
If someone is ready to foot a 620 Gbps bandwith bill all year long, I am pretty sure Akamai will be more than happy and able to scale up further.
Too bad there is no always a cleaner/smarter solution than pure bandwidth and $$ to fight those attacks.
As someone who actually pays for DDOS protection, this doesn't alter my opinion of Akamai.
I already know that Akamai is expensive and not particularly good at it. They are a CDN who is trying to make some money on the side with unused bandwidth, and will protect their CDN business if it comes down to it.
They don't invest in active defense. In fact, I know folks who actually had to block malware being served from Akamai-owned IPs!
I suspect that is why they gave service to Krebs for free in the first place--they need the marketing.
krebsonsecurity.com is now resolving to localhost. I guess he doesn't want to give the DDoSers a target.....
I'm wondering if the rising scale of these attacks & the seeming ease with which sites can be taken down will ultimately result in an "authenticated" internet - ie. you can't even connect without identity verification.
We already see publishing through FB Instant Articles etc. moving in that land on top of the current internet, to combat these types of firehose attacks, the only solution may be to take authentication one level deeper into the connection level.
That of course sounds good to security agencies as that's the end of anonymity online.
I doubt it. I think a lot of these attacks (though possibly not this one) would not even be possible if ISPs did egress filtering of packets with spoofed IPs, and yet ISPs don't seem to be implementing even this feature. It seems unlikely that they would leap frog this and start requiring some sort of identity verification, just to solve DDoS problems.
Additionally, in situations that don't make heavy use of amplification (where egress filtering doesn't help much), the way it's usually accomplished is by compromising a bunch of hosts - home computers, routers, etc, and assembling a botnet. In those cases, if your device is compromised, it would authenticate as you anyway, so such a scheme would solve nothing.
But if you had authenticated access, you could find exactly which C&C server controlled that botnet node and then who controlled that C&C server right? All these attacks depend on some form if amplification - if only to go from C&C servers to botnet. If just being on the network required authentication, you could trace back network connections and ID the controller even if attack was by a botnet.
I don't think the imperfect mapping between IP and user is the real problem with tracking down botnet controllers. You have a network of compromised hosts, and you can disguise yourself as one of them. There are lots of schemes for this sort of thing when you have a whole bunch of compromised machines under your control. For example, even if you made it illegal to run a Tor node, you could still just have your compromised network of hosts start running their own Tor (are you going to arrest all the people who just got a virus that starts a Tor network?) and you deliver commands from a hidden service. That's a huge effort on the part of society (banning Tor, implementing that, etc) that's easily circumvented.
In any case, like I said, you can't even get ISPs to do egress filtering of spoofed IPs, so even if it were going to solve DDoS, I don't think you'll get them on board for all the complications of implementing the protocols necessary and buying the equipment necessary to log all the traffic necessary to track down botnet controllers (who may be in a country where knowing who they are won't help you much anyway).
Time to use Github pages.
Too bad, I had some nice reads on his website. Hopefully this will only be temporarily...
It's funny how my mom after reading "record cyberattack" would be wondering how many poor people died but what it means is that somebody was downloading images from website many times.
I'm really interested to read his blog now. Any way I can find a readable version for his blog posts?
https://web.archive.org/web/20160922124922/http://krebsonsec...
Something about the platform-centric world we're in now is that this sort of attack doesn't have the blocking power it once did: you can mirror your content on Twitter, FB, G+, etc. and cross-link so people can still read your stuff. This makes the "denial" part pretty watered down; it's a wonder people even bother with these sorts of attacks anymore for non-services (i.e., for regular media material like text, photos, etc.)
Of course, maybe the goal is to deny someone ad revenue, but that seems awfully low-status for such a high-profile attack: "Yeah, we really got 'em! Denied 'em AD REVENUE for a whole week!"
Such attacks are possible because Internet is decentralized. There is no way to tell peers that you don't want to get traffic from some AS.
And investigation is difficult because attacking nodes might be in different countries, in some of which DDOS attacks are not illegal.
Maybe it is time to start building international firewalls to protect local infrastructure?
That would be quite the Great Firewall...
There are a number of factors that go into play (did the site use custom SSL, what edge locations were they providing caching in, etc), but had Kreb been a normal paying customer, this could easily have been a over a million dollar bill (if it was sustained long enough to alter his 95th percentile bracket) in the cheapest case. If things like custom SSL are in the mix (which Akamai charges absurdly high prices for), or lots of traffic from more expensive POPs, or lack of already having pricing commiserate with high volume traffic commitments, the bill could've been 5-10x that amount or more.
The first thing a lot of people are thinking (and saying) is "switch to Cloudflare". But there's another name I think needs to be said - OVH. OVH can withstand a Tbps scale attack as far as I know, and it provides this to pretty much anyone. They have a pretty good interface and some of their plans are extremely cheap. They're also great at standing up for free speech, which I really appreciate.
OVH got hit with record attacks yesterday too: https://twitter.com/olesovhcom/status/778019962036314112
Also, they claim to (soon) be able to filter up to 5 Tbps attacks: https://twitter.com/olesovhcom/status/778831449206231041
Yes, and my cheap $20 Kimsufi (OVH's ultra-cheap brand) still didn't drop off the internet.
Yeah, been quite happy with their DDoS protection myself for hosting game servers. And short of drawing the massive ire of organized crime and/or nation states, that's about the biggest DDoS bullseye you can paint on your back.
Krebs also hasn't publicly criticized OVH like he has CloudFlare, so I could see that working out well. Would be great press for them, too.
So the attackers win..
These 'attackers' give Krebs' more publicity than he would ever be able to generate himself.
It's also useful to point out that Krebs' hasn't been the only target as half a dozen other large targets were attacked http://www.webhostingtalk.com/showthread.php?t=1599694
Brian Krebs is a hero. Are Akamai executives cowards for dumping him? I'd like to add that law enforcement are heroes.
And it's honorable he wants to meet Fly in person, recognizing him as a human being. I haven't read it yet but I'm assuming the reference to 12-step hints that Fly's having some post alcohol binge regrets.
I'm sure alcohol makes it easier to hurt other human beings, which is why violent people are often drunk. I'd be ashamed of myself if I woke up realizing that I'd spent my life actively trying to harm other human beings for money, feeling no remorse until Karma (here defined as law enforcement officials) finally caught up with me.
After looking into this more I see that even Krebs himself does not blame Akamai for dumping him so I regret that part of my post.
It's kind of stupid to me that the massive and advanced cdn of akamai protect something as non-important as a blog against such a major ddos attack. If they were doing it pro-bono wouldn't the prudent action be to mitigate ddos's until a certain treshold and then actually assess the value of what you are protecting? A good lesson to have learned, I believe.
But no, they'll drop this client which had to have continually given good referrals.
If you're curious what the source of the DDOS attacks are from, here is a recent one that hit OVH:
https://twitter.com/olesovhcom/status/779297257199964160
This is much higher than the Akamai attack on Krebs too. Welcome to the wonderful side-effects of the totally insecure firmware of IoT...
Unbelievable, they enjoyed year of free publicity from association with him, and this is how they repay him. Its bad enough that they couldn't handle the attack, despite all the bragging about their multi-Tbps capacity...