Pete's right-on about this: "You've chosen to leave all that information out in the open so you can benefit from the search traffic, and instead try to change the established rules of the web so you can selectively sue anyone you decide is a threat. "

Speaking as someone who's working on leveraging the Facebook API in a commercial product, this leaves me feeling like I'm opening myself to a lot of legal exposure if Facebook subjectively decides that my service poses even a minor threat to them. Given that I'm bootstrapping, there's no way I'd be able to put up any sort of legal fight what so ever against a company as well-funded as Facebook.

I think that Facebook has a right to protect their data, but leaving robots.txt wide open and trying to enforce some random TOS that a developer might not even know about is the wrong way to do it. I think a much better method would be to ban all bots from robots.txt (and crawling) except the major search engines and provide data to others through APIs or other means that are controlled (and the TOS can apply to the API rather than random crawling). Maybe not what a developer wants to hear, but the truth is that the internet is infested with bots that do nefarious things, and I think it's better for the privacy and security of their userbase to control the spread of their data as much as possible.

I am unfamiliar with the details of a robots.txt file. Is it possible to specify "I only want bots x, y and z to crawl my site"?

yes, using the user-agent:

http://en.wikipedia.org/wiki/Robots_exclusion_standard

How difficult is it for a bot to lie?

Though if you've made it clear that only x, y and z can crawl your site, and someone spoofs, say, y, then it would be easy to demonstrate that someone has done something they know they shouldn't.

They could also make the pages NOT indexable and then have a side agreement with Google to give Google API access to index the pages they want.

Sorry, as implied to my comment on the first story about this posted to Hacker News, I'm siding with Facebook on this one.

Of course they tune their page so that search crawlers can best index the information, so does everyone else on the web.

However, they also provide an API, with clearly defined terms of use, which you may use to get information. Basically your complaint boils down to you don't want to use the methods they've set up for you to access their data, and you're complaining about it.

As for comments about fair, true spirit of the internet what have you... I don't think the true spirit of the internet has ever been everyone has to give everything away in every possible imaginable way. And, in the end, it is Facebook's data, they make that clear before you ever start adding data to their servers. So does just about everyone else who allows you to submit data to their servers.

This shouldn't be confused with how difficult it is to remove data and accounts from their system, which is a giant pain in the butt, or the fact that they've made drastic changes to the public nature of the data after keeping it private for so long. That's all just a giant mess.

It's not Facebook's data. It's the users' data.

I am just logically arguing here, not in support for anyone. But hasn't Facebook worked "hard" to setup the infrastructure to "collect" users' data? If you want users' data, go ask the users; not via scraping Facebook.

If I request content without circumventing any access controls and you give that content to me, I've done nothing wrong. I didn't accept any terms of service to do that, and even if I did, what is the legal recourse for someone breaking the terms of service? I always assumed it was termination of service, but that's probably wrong.

Suing people because you're too lazy to only give your content to people who you want to receive it is wrong.

What will happen is that they will send a cease and desist letter to you (like they did to Pete). At that point, you have basically given notice and it becomes much riskier to continue doing what you're doing.

That's not to say their C&D is valid or enforceable. It just enters into another legal bin that's a bit more hairy.

Unfortunately, the legal precedent for all this is murky at best. Past cases have been very specific to the details of those cases. No line has ever been drawn about what is ok and what is not.

In my personal opinion, the Facebook employee's response shows that he doesn't understand the company's high-level business goals. Their developers may think they are building an open web, but it's very clear the company is only interested in a closed web they control.

Based on my interaction with website API developers, most of them honestly believe that they are building an open web. But let's face it, the API is a benefit to them and creates a critical dependency for the user of the API. It's basically vendor lock-in.

This is Bret Taylor, CTO of Facebook.

There are a couple of things I want to clarify. First, we genuinely support data portability: we want users to be able to use their data in other applications without restriction. Our new data policies, which we deployed at f8, clearly reflect this (http://developers.facebook.com/policy/):

    "Users give you their basic account information when they connect with your application. For all other data, you must obtain explicit consent from the user who provided the data to us before using it for any purpose other than displaying it back to the user on your application."

Basically, users have complete control over their data, and as long as user gives an application explicit consent, Facebook doesn't get in the way of the user using their data in your applications beyond basic protections like selling data to ad networks and other sleazy data collectors.

Crawling is a bit of special case. We have a privacy control enabling users to decide whether they want their profile page to show up in search engines. Many of the other "crawlers" don't really meet user expectations. As Blake mentioned in his response on Pete's blog post, some sleazy crawlers simply aggregate user data en masse and then sell it, which we view as a threat to user privacy.

Pete's post did bring up some real issues with the way we were handling things. In particular, I think it was bad for us to stray from Internet standards and conventions by having an robots.txt that was open and a separate agreement with additional restrictions. This was just a lapse of judgment.

We are updating our robots.txt to explicitly allow the crawlers of search engines that we currently allow to index Facebook content and disallow all other crawlers. We will whitelist crawlers when legitimate companies contact us who want to crawl us (presumably search engines). For other purposes, we really want people using our API because it has explicit controls around privacy and has important additional requirements that we feel are important when a company is using users' data from Facebook (e.g., we require that you have a privacy policy and offer users the ability to delete their data from your service).

This robots.txt change should be deployed today. The change will make our robots.txt abide by conventions and standards, which I think is the main legitimate complaint in Pete's post.

Bret wrote: "I think it was bad for us to stray from Internet standards and conventions by having an robots.txt that was open and a separate agreement with additional restrictions."

You don't have an "agreement" at all. An agreement requires that two parties actually, um, agree. You've published a statement where you assert certain rights and imply that you will sue anyone who accesses data on your site in a way you don't like. You may get away with that, regardless of the legal merits of your position, because you have more money for lawyers than most people you're likely to sue. But don't try to dignify what you're doing by calling it an "agreement". It's like an extortionist telling me that we have an agreement that he won't break my windows if I pay him protection money.

It would actually be interesting to have a browser plug-in that shows you if a page you are on is excluded in the robots.txt file or which agents are specifically excluded/allowed to scan the page. Might make for some interesting viewing on sites like Facebook.

I'm tired of hearing people complain about private companies' rules for using their information/platforms. Facebook and Apple can do whatever they want on their own platforms and if people don't like it they can leave/not use the service. That's how a free market and free web works. People complained about Microsoft's dominance years ago and now their ascendancy is ending due to this idea called the "market"...