prova a dare uno sguardo a questo link:
http://www.codeproject.com/cs/internet/Crawler.asp