http://spiderzilla.mozdev.org/notes.html#c10 How can I get spiderzilla to ignore robots.txt like WinHTTrack can? Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)