Dai uno sguardo qua.
http://www.robotstxt.org/wc/active/html/index.html