Hacker News

Top stories

Live mirror
30 storiesupdated just nowView source snapshot
  1. Jemalloc 5.4.0(github.com/jemalloc ↗)
    20comments
  2. Astra for Law(openai.com ↗)
    491comments
  3. The scourge of x86 emulation(fex-emu.com ↗)
    9comments
  4. Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint(prismml.com ↗)
    118comments
  5. Bend – A language that blocks AI mistakes via proof, on CPU and GPU(bend-lang.com ↗)
    205comments
  6. Qwen 3.8 Omni Flash(qwen.ai ↗)
    63comments
  7. Hister: A private search engine for the pages you visit and the files you keep(github.com/asciimoo ↗)
    160comments
  8. A heap overflow and SSO misconfiguration to compromise OpenAI internal repos(hacktron.ai ↗)
    124comments
  9. Pre-Greek: The lost language hidden within Ancient Greek(linguisticdiscovery.com ↗)
    26comments
  10. When the fractional part of a float fixes your shader(crocidb.com ↗)
    discuss
  11. Wax motor(wikipedia.org ↗)
    64comments
  12. Fujitsu launches made-in-Japan next-generation CPU FUJITSU-MONAKA(global.fujitsu ↗)
    222comments
  13. Shapelearn Qwen 3.8 27B (13.1 GB VRAM)(byteshape.com ↗)
    6comments
  14. How to Write with an LLM(sockpuppet.org ↗)
    84comments
  15. Flet 1.0 – Build cross-platform apps in Python(flet.dev ↗)
    52comments
  16. Telstra outage: The night a network decided the year was 2006(netnod.se ↗)
    22comments
  17. Apple detectives solved mystery of ancient tree and rewrote the history of fruit(scientificamerican.com ↗)
    5comments
  18. Ask A Monk – A digital wilderness for thoughts with no immediate answer(askamonk.online ↗)
    20comments
  19. Diplodocus, Long Thought Exclusively American, Turns Up in Spain(sci.news ↗)
    37comments
  20. Fixing an NZXT Signal 4K30 part 2: the green/pink video bug(downtowndougbrown.com ↗)
    8comments
  21. The most important product decision is what you don't build(liamnugent.me ↗)
    32comments
  22. CrowdSec Source Code Leak(crowdsec.net ↗)
    44comments
  23. Why I didn’t sign the Fields medallists’ letter(gowers.wordpress.com ↗)
    358comments
  24. How do we prevent mathemathics from devolving into the Medieval Era of secrecy?(mathoverflow.net ↗)
    95comments
  25. Waymo in Singapore(waymo.com ↗)
    108comments
  26. How Uber Protects Against Retry Storms(uber.com ↗)
    36comments
  27. Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data(arxiv.org ↗)
    39comments
  28. Show HN: Snapdrop: Instantly share files between devices. No setup, no signup(snapdrop.me ↗)
    34comments
  29. The American Religion of Self-Storage Facilities(newyorker.com ↗)
    390comments
  30. Khipu (Quipu) Field Guide(khipufieldguide.com ↗)
    discuss

MarkMonitor left 60k domains for the taking

420 pointsby 5y agoian.sh
99 comments
5y agoHN ↗

Oops. People pay a lot of money to MarkMonitor because their promise is to not make such mistakes.

5y agoHN ↗

When I was a customer, screwing up a parking page isn't the kind of mistake my employer was paying them not to make. My employer was paying them not to let outside groups change our domain's DNS settings.

But that was for domains we were actively using. I don't even remember if we had a parking page on the domains we only had to sit on; if we did, while it wouldn't have been great, it also wouldn't have been a big deal for them to show something attacker controlled for a little while. Parked domains weren't in active use, didn't have links going to them, and probably shouldn't seem credible to others.

I think we might have set up the parked domains to just redirect to our main website, but yeah.

5y agoHN ↗

it also wouldn't have been a big deal for them to show something attacker controlled for a little while

This is understating the risk. If an attacker obtains control over a website for even a brief period of time, they can go to a certificate authority and obtain a domain validation for the domain (including all subdomains) that's valid for one year. At the end of that year, they can use the domain validation to obtain an SSL certificate that's valid for up to one year, even if they no longer control the website. This means that following a brief takeover, the domain continues to be vulnerable to attack for two years, which would be a problem if a parked domain is transitioned to active use during that time.

5y agoHN ↗

Nowadays a bit lessened by CT logs, so you (or MM) can at least notice and cause the cert to be revoked.

5y agoHN ↗

Yes, although the attacker can defer logging the certificate to CT until the moment before the certificate is used in an attack, so you would not have much time to respond.

5y agoHN ↗

How? The CA submits records (precertificates) to the logs before it signs the cert.

5y agoHN ↗

thanks, I didn't know both steps (pre-cert and actual cert submission into logs) were optional.

5y agoHN ↗

Logging is not a policy requirement of the trust stores.

It's a condition for your certificates to work in some popular browsers (notably Chrome and Safari) but not a policy requirement.

This is on purpose. Google themselves for example will get a certificate issued, say, today, for shiny-new-google-product.example and then only at the last moment do they log that certificate when spinning up the web site https://shiny-new-google-product.example/ to launch shiny-new-google-product -- you can't find out about it in the CT logs because it wasn't in there earlier.

Now, the readily available and especially free certificates most people use are logged before issuance as you describe using the poisoned "pre-certificate" feature from RFC 6962 (in theory this will one day be replaced by a 6962bis actual pre-certificate document rather than poisoned X.509 certs) but that is not at all mandatory, it's just convenient because it requires no workflow changes. You get a certificate, it's a bit bigger (it has SCTs baked inside it) and it just works.

If you do things the hard way, you must ensure your server application software understands what SCTs are and can send them to clients as necessary. You save some bytes talking to clients that don't want the SCTs, and you get to have this just-in-time logging behaviour if you want that.

The other reason it isn't against policy to issue without logging is that some archaic systems that are subject the Web PKI rules but aren't actually talking to the public Internet, and especially to web browsers, do not have logging, it's not a thing for them. This will probably go away in the next few years, these systems age out, but they still existed when I last looked.

5y agoHN ↗

At the end of that year, they can use the domain validation to obtain an SSL certificate that's valid for up to one year, even if they no longer control the website.

Can you explain this further? I don't understand.

Are you saying they could MITM the request that the CA makes to the website when it tries to do domain valudation? If that's the case, why is it limited to 2 years? Why couldn't they continue to do this indefinitely?

5y agoHN ↗

No, they don't have to MitM the CA's domain validation request. While they have brief control over the website, they use domain validation method 3.2.2.4.18 (Agreed-Upon Change to Website v2)[1] or 3.2.2.4.19 (Agreed-Upon Change to Website - ACME)[2] to legitimately complete domain validation by making a change to the website.

Due to domain validation reuse[3], the certificate doesn't have to be issued right away. The attacker can wait and request the certificate up to 398 days later, without having to do domain validation again.

[1] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...

[2] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...

[3] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...

5y agoHN ↗

So if I understand the issue correctly, MarkMonitor pointed all of those domains to S3, without first creating S3 buckets for the domains in question?

5y agoHN ↗

Yes, they first pointed them to S3, then created the parked domain page for each of them. There was a window where anyone could have claimed the bucket.

5y agoHN ↗

And ultimately they still own the domains and can always point them to a new IP address at any time. The window for exploitation seems kind of small and temporary.

5y agoHN ↗

The window for exploitation seems kind of small and temporary.

That depends on whether you are attacking the clients or the server. If an attacker obtained a domain cert or wildcard cert, while in control of the domain, then the attacks can continue.

5y agoHN ↗

How? They now have a cert for that domain but the domain owner can point the domain to any IP address he wants. The article mentions this, but I don’t see the vuln. Can you explain?

5y agoHN ↗

I think we are going to need a little bit more. How does the attacker get traffic to route to them?

5y agoHN ↗

DNS is not (yet) generally secured. An attacker who e.g. controls a wifi AP can just substituted their own DNS responses and redirect users to their own addresses, and their servers have valid certificates for the domains that they're supposed to be, so from the user side everything looks fine.

5y agoHN ↗

Yes. When the attacker can interpose their own server on the network, seen by valid clients, then the attacker can redirect the DNS of those clients to the fake server.

This is a MiTM attack that allows the fake server to be accepted by valid clients (on that network), stealing their credentials and then potentially their information from the server. There are actually many attacks that can happen here, one of which is simply to record the credentials, while passing the traffic through to the server and back from the server to the clients.

Even though the true DNS server has had its DNS record's IP address changed to point towards the correct and true server, the clients are unaware of this change, as their DNS caches have been poisoned to point towards the malicious server.

PS There is a defense against this, certificate pinning, but that is not used much in practice.

5y agoHN ↗

DNS is very easy to spoof and redirect. There are proposals to secure it (DNSSEC, DNS-over-TLS, DNS-over-HTTPS) but none are widely used and instead the server's certificates are used to both authenticate the correct destination and encrypt traffic to it.

Redirecting traffic is much easier than generating certificates so a valid cert held by a bad actor can be a serious vulnerability.

5y agoHN ↗

Redirecting traffic is much easier than generating certificates so a valid cert held by a bad actor can be a serious vulnerability.

This is bullshit. If you can redirect traffic you can almost always create a certificate unless you can only redirect a very limited set of traffic.

Redirecting traffic is literally all you need in order to be able to use certbot to generate a new cert.

5y agoHN ↗

It's a bit silly, there's no meaningful defense defeated here. Since the BGP routing system is almost entirely trust based a determined attacker will be able to perform a MITM attack on your domain and get a domain validated TLS certificate issued.

There's no real defense against this, TLS only stops dragnet attackers.

5y agoHN ↗

Only if they notice it among 60K+ domain names. There might be an automated system that silently proceeds to the next domain if the bucket has already been claimed.

5y agoHN ↗

Agreed. The title is misleading...there were no domains 'taken', which would imply a change of ownership, especially in the context of a registrar.

5y agoHN ↗

'for the taking', ie: they were vulnerable to a domain takeover. The title is entirely accurate. They didn't say they were 'taken'.

5y agoHN ↗

Good point, I can see that perspective. I guess I should have used 'ambiguous' instead of misleading. My first thought, with current wording, is that they were allowing transfers out or letting domains expire. Perhaps I spent too much time in the registry business.

5y agoHN ↗

I also understood it as if they lost the domains.

5y agoHN ↗

A lot of windows are small and temporary.

In this case the attacker could have served anything out of the S3 bucket and conducted full blown phishing (google.ar, coinbase.ca), and generated valid certificates.

It's a big deal the order of operations here and the risk MarkMonitor introduced to all their clients.

5y agoHN ↗

This should be something added to an AWS Security Hub check, e.g. do you have Route53 pointed to an unregistered domain. Otherwise, I can't find fault with AWS at all. Now I have to review our domains on Monday.

5y agoHN ↗

Please bug your TAM if you have one about this. I’ve bugged ours. S3 should not serve a bucket as a website without domain verification. In the interim, we’ve built middleware where a bucket serving content can’t be removed until the dns record has been.

5y agoHN ↗

Or use CloudFront with an S3 bucket as backing for this use case, like you'd expect? CloudFront has domain name verification.

5y agoHN ↗

The point is that without domain verification, it won't stop someone else from registering that bucket (which is what happened with the domains in this article).

5y agoHN ↗

The point is that S3 isn't a CDN. If you use it as a CDN, it's on you to ensure it'll work for your use case. CloudFront, however, is a CDN, and as expected, has domain verification.

5y agoHN ↗

Whether it's a CDN is irrelevant. This is already a supported use-case for S3 which is why it even has this functionality.

It's one of many products that supports serving under custom hostnames and all such products should have domain verification.

5y agoHN ↗

I've seen this attack with cloudfront and an S3 bucket.

You can verify your domain is going to a cloudfront you own, but it doesn't verify that the origin is a bucket you own.

5y agoHN ↗

I hear about subdomain takeovers with S3 all the time. It seems insane Amazon doesn't require any domain verification.

5y agoHN ↗

What do they require? You can just claim any S3 bucket you want that's been 'abandoned'?

5y agoHN ↗

No, S3 buckets named after hostnames are served when said hostname is pointing to S3 servers .

Bob owns bob.com and sets CNAME bob.com.us-east-1.amazonaws.com. Eve creates the bucket and can now serve any static content she wants.

5y agoHN ↗

Maybe Bob shouldn't have set cname bob.com.us-east-1.amazonaws.com if he didn't own the bucket?!

5y agoHN ↗

Often they did have it but it was decommissioned or deleted.

5y agoHN ↗

That's easy to say when there's single or double digit number of people involved, but three or four (or five!)?

5y agoHN ↗

That seems pretty reasonable to me. How should AWS verify domain ownership if not control over the CNAME?

5y agoHN ↗

it doesn't verify control over the CNAME from the person creating the bucket, that's the criticism, instead it lets any AWS user claim it. (The usual verification method would be something like requiring a TXT record to be set with a secret that's tied to the bucket, and I guess for route53 users they could integrate settings from there)

5y agoHN ↗

Right, but the owner of the domain needs to create a CNAME record that points to AWS for a bucket they don't control. How is that not purely on them?

5y agoHN ↗

If thousands of people keep making a mistake when using your product, you can go "purely on them, not our problem", true. Or you can do something to your product to make the mistake less likely to cause a problem.

5y agoHN ↗

The bucket can be created before they can do so, which is exactly what happened to MarkMonitor here. Now, technically you could create the bucket first before setting the CNAME, but this can quickly go wrong (especially when handling hundreds of domains) and it regularly does. There's no need to keep this footgun around.

5y agoHN ↗

S3 is not a CDN. S3 is a distributed file system. Use a CDN for CDN use cases (CloudFront) and you wouldn't have these issues.

There's a chance a normal AWS user may not understand the above distinction, a sophisticated actor like MarkMonitor (whose business is THIS) should. Instead they had a security incident. That's it. They used the wrong AWS service. They configured massive numbers of domain DNS records incorrectly. They risked their customer's reputations.

5y agoHN ↗

Google provides several options for site verification. They provide a string you must place on your website using one of the following ways:

- DNS TXT record

- HTML meta tag

- Upload a file with a randomized name to your site root

5y agoHN ↗

This is the reverse situation. What's going on here:

- Bob makes bob.com and sets the CNAME to be bobstaticsite.s3.aws

- Bob forgot to make a bucket called bobstaticsite.s3, and Alice, scanning the DNS records, creates it instead. Now Alice is serving data on Bob's domain

Am I missing something? How would AWS S3 stop you from creating a CNAME? Or how is this their responsibility? Don't make that CNAME entry pointing to a bucket you didn't create yet / don't own?

One final thing: S3 is not CloudFront. And CloudFront, AWS's CDN, does require domain name verification.

5y agoHN ↗

AWS can't stop you from creating the CNAME, but AWS could stop Alice from creating the bucket, or (IMHO better) not serve HTTP(S) traffic for bob.com from that bucket unless it is confirmed that the bucket belongs to the owner of bob.com.

5y agoHN ↗

Am I missing something? How would AWS S3 stop you from creating a CNAME? Or how is this their responsibility?

They (AWS) should not accept traffic on that hostname until ownership - tightly bound - is proven.

This also helps avoid related cases, such as deleting a bucket (which now frees it up) - other services like Heroku have similarly been vulnerable to this takeover hack because of a lack of strict verification: https://0xpatrik.com/subdomain-takeover-providers/

Every new bucket, service or account handling traffic for a domain should require re-verification. That verification should never persist beyond the lifetime of that resource.

5y agoHN ↗

All of which are much weaker than the CNAME, since with a cname you can do all 3.

5y agoHN ↗

If they cared, a reasonable choice would be to lean on the Web PKI. It is usual in the Web PKI for the leaf certificates to be certified both to identify a server (which is what is ordinarily done) and to identify a client. The only name usually provided for the subject is a DNS name, but in this case that's exactly the identity we want to prove.

That is, you'd prove you control www.example.com to AWS the exact same way www.example.com proves it is www.example.com to a web browser.

5y agoHN ↗

If you control the cname its trivial to get a cert.

5y agoHN ↗

Sure, but the point of this choice is to make it easy for a third party to verify your identity, the exact same problem as for web browsers visiting an HTTPS site.

5y agoHN ↗

What's the method of attack here? Bob at some point stops using the site and deletes the bucket but not the DNS entry, and you notice that and create a new bucket with that now-available name?

5y agoHN ↗

Or as happened here, bob points the site to S3 because he plans to put something there later, and Eve beats him to it.

5y agoHN ↗

No

This company never created the S3 buckets at all but set them in the CNAME, for domains that were purchased but never used

They can change the CNAME at any time, it’s a pretty dumb but inconsequential exploit. Potentially some purchaser didn’t use generally free and default whois privacy and could be associated with offbrand content.

Potentially someone observant can earn a bunch of ad dollars on a popular domain.

5y agoHN ↗

I don't think it's as inconsequential as you say. Coinbase.ca and google.ar were two of these domains. The author noted that TLS certs could have been minted and then used in MITM attacks, if, for example, Coinbase began operating in Canada.

5y agoHN ↗

This feels like a typical "why we can't have nice things"-scenario. You're in control of DNS. You decided to create the DNS-record to point to S3. It's a little odd to then come complaining when Amazon does what you wanted it to do.

A fun exercise could be to boot up a machine on pretty much any cloud provider and set up a simple webserver to respond to all Host-headers. Heck, I'm pretty sure you could make nginx only respond requests where both the Host-header and A/AAAA-record were correct with some Lua magic. In this scenario, would you blame the cloud provider or the administrator of the domain?

(Not necessarily you-you, but you get he idea.)

5y agoHN ↗

Relatedly, the thought of properly storing your web server's full access logs indefinitely comes to mind, for a baseline of "until I write some exactingly specific filtering logic and I get 0 unclassified results back out". Doesn't have to happen immediately, but by all means hoard 50GB of traffic logs until you do get it done. That sort of thing.

You'd naturally be capturing the requested host here, ideally along with all other request headers.

And ideally flagging unknown host headers in close to real time, so you can have fun with your visitors next time they say hello :>

5y agoHN ↗

S3 allowing customers to incorporate a "domain" or any part of it as part of the bucket's name is a contributing factor here, in my opinion.

The name of the bucket (which ends up being a part of the bucket's URL) should be entirely outside of the user's control. Make it a random 32 character string (and make it such that old strings can't ever be recycled).

This way people can't "register" buckets for domains they don't own.

5y agoHN ↗

I found a similar issue in a huge, very popular site. Thousands of root domains ripe for the taking. I sent a heads up to the company in question, but never got a reply--

5y agoHN ↗

I think it's the issue with mark monitor and not s3/aws.

What if mark monitor would put all parked domains to bob.com.mys3.com service? Me, as mys3 provider I'm at fault? I doubt.

What if mark monitor would point to an IP address they don't own? Still not their fault?

5y agoHN ↗

You appear to be shadow-banned... which means that almost no one can see what your posts because of HN's algo and moderators (only people with show-dead enabled are able to see your posts).

5y agoHN ↗

While many domains began responding with an S3 404, others began switching from S3 to the parked page. What is interesting is that DNS was not involved — all domains pointed to 93.191.168.52 both before and after the issue.

I don't get this. These domains weren't configured with CNAME DNS entries like one normally does. Does this mean MarkMonitor's parked domain name server was trying to load from S3, then falling back to something else if that failed?

5y agoHN ↗

I suspect what happened (as the author of the post) is that given the IP address is from Akamai, they misconfigured Akamai to naively proxy the traffic towards S3.

S3 will just pick a bucket based on the Host header in the request, and it seems like Akamai just proxied that from the client.

5y agoHN ↗

Ahh, sure, that'll do it.

That's quite a big oops. I'd love to see the panicked messages on their internal slack as they figure out what they did.

5y agoHN ↗

I guess they have an automatic system to do the same thing as your exploit. After they misconfigured Akamai, their system and your system were racing to claim the exposed S3 buckets.

5y agoHN ↗

They should be using cloudfront to serve those pages over https.

5y agoHN ↗

Why so? Expecting to claim the bucket name for each of their domains is clearly not the way to go. With cloudfront they have can have a single bucket and map upto 100 multiple (or request a higher limit) domains per cloudfront distribution backed by that bucket.

5y agoHN ↗

Can someone explain how a subdomain takeover is done in layman terms?

5y agoHN ↗

Subdomain points to a hosting provider. Hosting doesn’t know who the owner is (yet) and waits for someone to sign up/register. Attacker signs up before the real owner does, is lucky that the hosting provider does not verify ownership, and is able to serve whatever they want on the domain, for example a fake website or fake verification files.

5y agoHN ↗

Or often the other way around - a subdomain for some legacy feature points to a host. After the feature gets axed the server gets shut down (as it costs money) but the domain is left dangling. Anyone can claim the target subdomain (for CNAME) or IP (for A records) on the same hosting provider if not in use. Apart from fake websites it can also be used to bypass SOP/CORS protection in some contexts.

5y agoHN ↗

What does the OP's "S3 detection" mean in this context?

I'm not clear how the list of vulnerable domains was collected in this instance - presumably he had to know which domains to create buckets for?

5y agoHN ↗

Presumably he's scanning DNS records looking for subdomains (A or CNAME records) pointing to S3 buckets. Script then checks to see if these buckets exist and outputs a list of subdomains where the bucket doesn't exist.

At least that's my understanding of his process.

5y agoHN ↗

“ MarkMonitor sells themselves as the domain registrar that does not make mistakes. (…) MarkMonitor is not a cheap solution to this problem, but it is widely used (apparently by "more than half of the Fortune 100", per the page)“

And then:

“ MarkMonitor does not have a way of disclosing security issues, which inhibited reporting this to them in a timely manner. They have not responded to any of our communications.”

Should anyone be surprised that a company that claims to not make mistakes, but has no way to report vulnerabilities gets their vulnerability on the front page of HN? (And this is a good case scenario, as opposed to their customers being hacked: which could have happened by an attacker claiming some of these domains)

Let’s hope they make at least the change of putting a bug bounty program in place.

5y agoHN ↗

This issue is not entirely the fault of MarkMonitor. While they need to be careful with handling parked domains, AWS is at fault for not being more stringent with claiming S3 buckets. Google Cloud, for example, has required domain verification for years, rendering this useless.

This sounds like more of an Amazon problem than a MarkMonitor problem to me. And it makes a good case for using other cloud providers over AWS, as with GCP this attack wouldn't have been possible. Merely creating the DNS records pointing to the cloud provider's nameservers shouldn't be enough for anyone to then claim it and start hosting.

5y agoHN ↗

From Amazon's own documentation about this:

"An Amazon S3 bucket name is globally unique, and the namespace is shared by all AWS accounts. This means that after a bucket is created, the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted. You should not depend on specific bucket naming conventions for availability or security verification purposes. For bucket naming guidelines, see Bucket naming rules."

... but if you want to host a static website via S3, they explicitly say you need to create a bucket with the domain name as the bucket :shrug:

5y agoHN ↗

Seems like a lot of people are blaming AWS. S3 buckets have a grace period of 24 hours after a bucket is to be deleted in which it cannot be registered by anyone else. We call it internally, "bucket sniping". AWS does not have control of your DNS records. This falls under the typical cloud shared security model.

It's purely a DNS management issue. This happens with other technologies too, like mail servers. Domains (unique identifiers like S3 bucks) expire, someone registers it, spins up a mailserver, and begins recovering accounts via password resets. Who is to blame here, the application for allowing the password reset (provided the valid email/fetched the reset key), or the person who let the domain expire? I'd say the latter, and in the s3 situation, I believe it's the same.

5y agoHN ↗

Another great post Ian!

If any researcher here needs data or help to do investigations like this please reach out to me chris at securitytrails.com - we're trying to hone the tools to be as useful as possible with as little effort.