The article explains a few network protocol reverse-engineering techniques that were used over the course of 12 years to write Samba.

The techniques are:

  1. Use of publicly available documents on the CIFS/SMB protocol
     (not reverse-engineering but the documents are incomplete);

  2. Use a network sniffer to listen in on conversations between Microsoft clients and servers;

  3. Use of a protocol scanner: a program that tries all possible "words"
     in some section of a protocol and uses the response
     to automatically deduce new information about the protocol.

  4. Use the "differential" technique: You write your own implementation
     of the protocol with what you know, then send pairs of identical messages
     to both the real implementation and your implementation and verify that the responses are the same.

Trivia: Andrew Tridgell's techniques here catalyzed the creation of git. He applied them to reverse-engineer parts of the Bitkeeper protocol, a proprietary version control system then hosting the linux kernel. Bitkeeper CEO Larry McVoy revoked linux's license to use BK, Linus went off and wrote a replacement, and the rest is history.

http://en.wikipedia.org/wiki/BitKeeper